It's an ad. "The Solution: TokensTree". From tokenstree.com
I was expecting a secondary market in tokens, perhaps crypto-powered, but no.
The cost difference for languages roughly correlates with how much text it takes to say something in that language. English is relatively terse. (This is a common annoyance when internationalizing dialog boxes. If sized for English, boxes need to be expanded.) They don't list any of the ideographic languages, which would be interesting.
That would cause the opposite effect of what we’re actually seeing (i.e. “more redundant languages” would be using comparatively fewer tokens).
The real reason is that tokens are probably strictly based on n-gram frequency of the training data, and English is the most common language in the training data.
It has nothing to do with English being terse, a claim that probably isn't even true for code when measuring character or rune or glyph length. It has everything to do with the proportional frequency of a language in the training corpus.
Comments
It's an ad. "The Solution: TokensTree". From tokenstree.com
I was expecting a secondary market in tokens, perhaps crypto-powered, but no.
The cost difference for languages roughly correlates with how much text it takes to say something in that language. English is relatively terse. (This is a common annoyance when internationalizing dialog boxes. If sized for English, boxes need to be expanded.) They don't list any of the ideographic languages, which would be interesting.
That would cause the opposite effect of what we’re actually seeing (i.e. “more redundant languages” would be using comparatively fewer tokens).
The real reason is that tokens are probably strictly based on n-gram frequency of the training data, and English is the most common language in the training data.
It has nothing to do with English being terse, a claim that probably isn't even true for code when measuring character or rune or glyph length. It has everything to do with the proportional frequency of a language in the training corpus.
My impression of dialog box size from least to greatest is CJK (Chinese < Korean < Japanese) < English < everything else
This is not cryto or something else, it’s a platform for tokens reduction. You can try and then post it before do it assumtions. :)