How to process 100gb tsv and XML files?

I am trying to parse a music data file that is close to 100gb. What app or programming language is best for handling a file like this?

Thanks!

FlyingAvatar 2y

It really depends on what you need to do with the data, but in most cases Python could do this pretty easily with csv.reader (with a \t delimiter for TSV) or xml.etree.ElementTree.iterparse (for XML) in streaming fashion such that you're not loading the whole file at once.

pradeepchhetri 2y

You can leverage ClickHouse to process your music data. ClickHouse supports both TSV[0] and XML[1] data formats.

[0] https://clickhouse.com/docs/en/interfaces/formats#tabseparat...

[1] https://clickhouse.com/docs/en/interfaces/formats#xml

mobilio 2y

Great solution is DuckDB: https://duckdb.org/docs/data/csv/overview.html

datadrivenangel 2y

What kind of single music data file is 100gb?

Also how is it structured? If it's actually a tab separated value file, consider using something like polars or DuckDB?

anindhaOP2y

I found this https://klogg.filimonov.dev

abdusco 2y