I use it daily and it never crashed. How long ago was this?
I am a big fan of DuckDB. Plow through hundrets of GB of logs on a 5 year old linux laptop - no problem.
it's not the focus or very performant but you can have it spill to disk if you run out of memory. I wouldn't suggest building a solution based on this approach though; the sweet-spot is memory-constrained.
Comments
I use it daily and it never crashed. How long ago was this? I am a big fan of DuckDB. Plow through hundrets of GB of logs on a 5 year old linux laptop - no problem.
Same here. I have however seen a few out of memory cases in the past when given large input files.
By default, it tries to take 80% of your memory. I've found that you need to set it to something much smaller in ~/.duckdbrc `set max_memory='1GB';`
it's not the focus or very performant but you can have it spill to disk if you run out of memory. I wouldn't suggest building a solution based on this approach though; the sweet-spot is memory-constrained.
Really? How large? I’ve only managed to crash it with hundreds/thousands of files so far, but haven’t so many huge files to deal with.