danso on 1.1B Taxi Rides Using OmniSciDB and a MacBook Pro

Comment on 1.1B Taxi Rides Using OmniSciDB and a MacBook Pro

    COPY trips
    FROM '/Users/mark/taxi_csv/*.gz'
    WITH (HEADER='false');

> The above managed to complete in 31 minutes and 40 seconds. The resulting import produced 294 GB of data in OmniSciDB's internal format.

I’m really curious how a simple import (no indexes or data typing) into SQLite would compare. But I don’t have 700GB of free SSD space to spare.

tmostakOP6y

Mark did a benchmark of SQLite using its internal file format a few years ago (https://tech.marksblogg.com/billion-nyc-taxi-rides-sqlite-pa...), clocking the import at 5.5 hours. It looks like this was done though on a spinning disk, so given a proper SSD, and a newer version of SQLite, it might be much faster.

momonga 6y

The same blog has other benchmarks[1], one of which is SQLite[2].

[1] https://tech.marksblogg.com/benchmarks.html

[2] https://tech.marksblogg.com/billion-nyc-taxi-rides-sqlite-pa...

Comments