It might be worth storing this data separately, to improve cache usage, but my experience is on any modern machine the amount of space taken by parsed code / ASTs is small enough to not care about, unless you do something like C++ where you "reparse the world" for every tiny file (due to header includes)
Maybe it's only necessary for serious compilers. Let's say 10 tokens / line avg., 40 bytes / token (but a naive token struct could easily be 100s of bytes), then we're in the region of 100s to 1000s of bytes per line. Now let's say compiling 100K lines would not be unheard of, and for benchmarks you want to push the millions. We're getting into regions where tokens alone can fill a computer's memory. If there is an easy to make and effective optimization, I'm all for it :-)
Comments
It might be worth storing this data separately, to improve cache usage, but my experience is on any modern machine the amount of space taken by parsed code / ASTs is small enough to not care about, unless you do something like C++ where you "reparse the world" for every tiny file (due to header includes)
Maybe it's only necessary for serious compilers. Let's say 10 tokens / line avg., 40 bytes / token (but a naive token struct could easily be 100s of bytes), then we're in the region of 100s to 1000s of bytes per line. Now let's say compiling 100K lines would not be unheard of, and for benchmarks you want to push the millions. We're getting into regions where tokens alone can fill a computer's memory. If there is an easy to make and effective optimization, I'm all for it :-)