Skip to content

Comment on Super-Structured Data: Rethinking the Schemaparent

Comments

Cool, I didn't realize you used sqlite-utils for your performance demo!

It's not particularly designed for speed - it should be fast as far as Python code goes (I use some generator tricks to stream data and avoid having to load everything into memory at once) but I wouldn't expect "sqlite-utils insert" to win any performance competitions with tools written in other languages.

Those benchmarks against sqlite itself are definitely interesting. I'm looking forward to playing with the "native ZNG support for Python" mentioned on https://github.com/brimdata/zed/blob/main/docs/libraries/pyt... when that becomes available.

One trick I sometimes use is to pipe it through pv to get a progress bar:

    % pv conn.json | sqlite-utils insert conn.db conn - --nl
    36.9MiB 0:00:06 [5.81MiB/s] [==>             ]  9% ETA 0:01:00
AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.