Relative to the preceding token in the stream, in number of lines and columns from the end of the last token (or beginning, but to me that's conceptually confusing). Take a look at the language server protocol specification for semantic tokens with relative positions as a reference.
Using an absolute line/column/index to indicate the location of a token in a file means you need to retokenize a large amount of the file whenever there's an edit. In a streaming parser that responds to live edits, storing the tokens with relative file locations allows you to only update neighbors on insertion or deletion (and also allows for bulk insert/delete). Absolute file locations can be recovered by scanning the token stream - which is why it's important for the tokens to form the leaves of the tree, it guarantees you can always recover the file locations.
Then you can pull some tricks like the Roslyn compiler does, which is to store things like comments, diagnostics, and white space as "trivia" associated with the tokens and very quickly recover the text that created the token stream/AST nodes. That's invaluable for tooling.
I agree, that's the right way to do it (not required for a batch compiler, though, because there are no edits). Text ropes work that way as well - contents are stored in a tree, and absolute coordinates are computed from relative coordinates using an associative operation (monoid).
Comments
Could you clarify what you mean by "relative" file locations? Relative to containing AST node?
Relative to the preceding token in the stream, in number of lines and columns from the end of the last token (or beginning, but to me that's conceptually confusing). Take a look at the language server protocol specification for semantic tokens with relative positions as a reference.
Using an absolute line/column/index to indicate the location of a token in a file means you need to retokenize a large amount of the file whenever there's an edit. In a streaming parser that responds to live edits, storing the tokens with relative file locations allows you to only update neighbors on insertion or deletion (and also allows for bulk insert/delete). Absolute file locations can be recovered by scanning the token stream - which is why it's important for the tokens to form the leaves of the tree, it guarantees you can always recover the file locations.
Then you can pull some tricks like the Roslyn compiler does, which is to store things like comments, diagnostics, and white space as "trivia" associated with the tokens and very quickly recover the text that created the token stream/AST nodes. That's invaluable for tooling.
I agree, that's the right way to do it (not required for a batch compiler, though, because there are no edits). Text ropes work that way as well - contents are stored in a tree, and absolute coordinates are computed from relative coordinates using an associative operation (monoid).
I have yet to see a batch compiler that wasn't hacked into some context where it's used as anything but a batch compiler!