I don't know about most people, but I deal with such files (usually JSON) on a weekly basis. It's surely one of the things that makes implementing text editors tricky.
Honest question, do you just open the files, or do you change them and write them back out?
I, too, have dealt with huge files. My primary large file use case for vim is as an interactive grep, opening files, then just performing iterative filters reducing the file to something "manageable". If the file is too big, I might resort to raw streaming filters to temp files before going all vim on it (even vim has issues with huge files -- huge files are issues all their own).
But it's more a matter of pragmatism as to how far one wants to take a pet project like this. There's a curve of diminishing return depending on what someone wants from such a thing.
A single RAM buffer is a bad idea, to be sure, but raw horsepower of modern systems and CPUs cover up a lot of bad ideas. Simply the idea that you COULD mmap or even malloc a 2G buffer to suck the file in, and it would WORK, is enough to make ones teeth itch. I remember long a ago an article from someone encountering an early workstation with 1G of RAM, and the worlds it opened up. We have come far from those days.
If the OP is interested in buffer mechanics, then starting with an API backed by a simple buffer goes a long way. If they're interested in screen painting, window management, etc, then the RAM buffer may be all they need. Otherwise, they can work on replacing their back end with all of the assorted structures folks have mentioned, see what they like best, as they all have tradeoffs.
I don't often need to edit the files (I also mostly use for searching and my most common editing action would be pretty printing which I could well manage externally if need be). However, I very commonly accidentally edit the files due to a stray keypress, so if edits are not disabled entirely then not locking up or crashing if edits do occur would definitely be a requirement. Occasionally it useful to be able to edit, and it's nice to have at least an editor that can do this.
More like a whole database table as a JSON file - I split the whole database export into tables using `jq` (an export from Firebase Realtime Database). I generally use sublime text to open them, it's quite happy with anything that will fit in RAM. For very large file it just shows a load bar while it loads it.
It was vital for exploring the shape of our data during a data migration to Postgres as our data was not very regular in shape and firebase's querying capabilities are pretty limited.
Just this morning I needed to do a global find/replace for a string in a 3GB .sql file. I was in VS Code anyway so tried that - but the app choked on it. Ended up going with sed and it worked perfectly.
Comments
I don't know about most people, but I deal with such files (usually JSON) on a weekly basis. It's surely one of the things that makes implementing text editors tricky.
Honest question, do you just open the files, or do you change them and write them back out?
I, too, have dealt with huge files. My primary large file use case for vim is as an interactive grep, opening files, then just performing iterative filters reducing the file to something "manageable". If the file is too big, I might resort to raw streaming filters to temp files before going all vim on it (even vim has issues with huge files -- huge files are issues all their own).
But it's more a matter of pragmatism as to how far one wants to take a pet project like this. There's a curve of diminishing return depending on what someone wants from such a thing.
A single RAM buffer is a bad idea, to be sure, but raw horsepower of modern systems and CPUs cover up a lot of bad ideas. Simply the idea that you COULD mmap or even malloc a 2G buffer to suck the file in, and it would WORK, is enough to make ones teeth itch. I remember long a ago an article from someone encountering an early workstation with 1G of RAM, and the worlds it opened up. We have come far from those days.
If the OP is interested in buffer mechanics, then starting with an API backed by a simple buffer goes a long way. If they're interested in screen painting, window management, etc, then the RAM buffer may be all they need. Otherwise, they can work on replacing their back end with all of the assorted structures folks have mentioned, see what they like best, as they all have tradeoffs.
I don't often need to edit the files (I also mostly use for searching and my most common editing action would be pretty printing which I could well manage externally if need be). However, I very commonly accidentally edit the files due to a stray keypress, so if edits are not disabled entirely then not locking up or crashing if edits do occur would definitely be a requirement. Occasionally it useful to be able to edit, and it's nice to have at least an editor that can do this.
Most people won't, they will just get annoyed, if they accidently open a video and the editor hangs up for some time, trying to process it.
And I regulary work with JSON files the size of some MB, which already puts some editors into a real struggle.
Out of curiosity, why do you have so big files? (A whole DB as a json file?) And what editor do you use for them?
More like a whole database table as a JSON file - I split the whole database export into tables using `jq` (an export from Firebase Realtime Database). I generally use sublime text to open them, it's quite happy with anything that will fit in RAM. For very large file it just shows a load bar while it loads it.
It was vital for exploring the shape of our data during a data migration to Postgres as our data was not very regular in shape and firebase's querying capabilities are pretty limited.
Just this morning I needed to do a global find/replace for a string in a 3GB .sql file. I was in VS Code anyway so tried that - but the app choked on it. Ended up going with sed and it worked perfectly.