Honest question, do you just open the files, or do you change them and write them back out?
I, too, have dealt with huge files. My primary large file use case for vim is as an interactive grep, opening files, then just performing iterative filters reducing the file to something "manageable". If the file is too big, I might resort to raw streaming filters to temp files before going all vim on it (even vim has issues with huge files -- huge files are issues all their own).
But it's more a matter of pragmatism as to how far one wants to take a pet project like this. There's a curve of diminishing return depending on what someone wants from such a thing.
A single RAM buffer is a bad idea, to be sure, but raw horsepower of modern systems and CPUs cover up a lot of bad ideas. Simply the idea that you COULD mmap or even malloc a 2G buffer to suck the file in, and it would WORK, is enough to make ones teeth itch. I remember long a ago an article from someone encountering an early workstation with 1G of RAM, and the worlds it opened up. We have come far from those days.
If the OP is interested in buffer mechanics, then starting with an API backed by a simple buffer goes a long way. If they're interested in screen painting, window management, etc, then the RAM buffer may be all they need. Otherwise, they can work on replacing their back end with all of the assorted structures folks have mentioned, see what they like best, as they all have tradeoffs.
I don't often need to edit the files (I also mostly use for searching and my most common editing action would be pretty printing which I could well manage externally if need be). However, I very commonly accidentally edit the files due to a stray keypress, so if edits are not disabled entirely then not locking up or crashing if edits do occur would definitely be a requirement. Occasionally it useful to be able to edit, and it's nice to have at least an editor that can do this.
Comments
Honest question, do you just open the files, or do you change them and write them back out?
I, too, have dealt with huge files. My primary large file use case for vim is as an interactive grep, opening files, then just performing iterative filters reducing the file to something "manageable". If the file is too big, I might resort to raw streaming filters to temp files before going all vim on it (even vim has issues with huge files -- huge files are issues all their own).
But it's more a matter of pragmatism as to how far one wants to take a pet project like this. There's a curve of diminishing return depending on what someone wants from such a thing.
A single RAM buffer is a bad idea, to be sure, but raw horsepower of modern systems and CPUs cover up a lot of bad ideas. Simply the idea that you COULD mmap or even malloc a 2G buffer to suck the file in, and it would WORK, is enough to make ones teeth itch. I remember long a ago an article from someone encountering an early workstation with 1G of RAM, and the worlds it opened up. We have come far from those days.
If the OP is interested in buffer mechanics, then starting with an API backed by a simple buffer goes a long way. If they're interested in screen painting, window management, etc, then the RAM buffer may be all they need. Otherwise, they can work on replacing their back end with all of the assorted structures folks have mentioned, see what they like best, as they all have tradeoffs.
I don't often need to edit the files (I also mostly use for searching and my most common editing action would be pretty printing which I could well manage externally if need be). However, I very commonly accidentally edit the files due to a stray keypress, so if edits are not disabled entirely then not locking up or crashing if edits do occur would definitely be a requirement. Occasionally it useful to be able to edit, and it's nice to have at least an editor that can do this.