Apologies for the slow reply, my lack of karma is throttling how often I can comment.
I appreciate you sharing your thoughts, the whole point of a public preview is to find out how things are received, what’s working, what isn’t.
We more or less agree on the vibe-coded website (especially the copywriting). It’ll improve over time, but this is a side-project, time is limited, and I wanted to release something on 7/4.
Tungsten’s grammar and design are mine.
Tungsten’s implementation, is almost entirely LLM (lowering, escape analysis, SSA), with a good bit of prodding, benchmarking, troubleshooting; in order to have the performance land remotely close to where I knew it could.
While my primary motivation was to build the language that I wish existed, if I can make something that others enjoy or find useful, even better.
To address your other point:
Whitespace is almost certainly getting tokenized whether it’s significant to the language or not. And I would expect most folks are indenting their blocks anyway, even when they are delimited.
I think you’ll find the token reduction comes from more than just dropping delimiters.
I won’t try to litigate the point here, I’ll save that for a longer-form write up that is lacking on the website.
Thanks for the in-depth reply, appreciate the elaboration on where you did and didn't use LLMs. Looking forward to reading the blog posts and other future updates!
Comments
Apologies for the slow reply, my lack of karma is throttling how often I can comment.
I appreciate you sharing your thoughts, the whole point of a public preview is to find out how things are received, what’s working, what isn’t.
We more or less agree on the vibe-coded website (especially the copywriting). It’ll improve over time, but this is a side-project, time is limited, and I wanted to release something on 7/4.
Tungsten’s grammar and design are mine.
Tungsten’s implementation, is almost entirely LLM (lowering, escape analysis, SSA), with a good bit of prodding, benchmarking, troubleshooting; in order to have the performance land remotely close to where I knew it could.
While my primary motivation was to build the language that I wish existed, if I can make something that others enjoy or find useful, even better.
To address your other point:
Whitespace is almost certainly getting tokenized whether it’s significant to the language or not. And I would expect most folks are indenting their blocks anyway, even when they are delimited.
I think you’ll find the token reduction comes from more than just dropping delimiters.
I won’t try to litigate the point here, I’ll save that for a longer-form write up that is lacking on the website.
Thanks again for the feedback.
Related (in the repo):
WValue layout https://github.com/tungsten-lang/tungsten/blob/d5f505acca245...
Ruby Lexer (regex-based) https://github.com/tungsten-lang/tungsten/blob/d5f505acca245...
Ruby Lexer (spec) https://github.com/tungsten-lang/tungsten/blob/d5f505acca245...
Thanks for the in-depth reply, appreciate the elaboration on where you did and didn't use LLMs. Looking forward to reading the blog posts and other future updates!