That would cause the opposite effect of what we’re actually seeing (i.e. “more redundant languages” would be using comparatively fewer tokens).
The real reason is that tokens are probably strictly based on n-gram frequency of the training data, and English is the most common language in the training data.
Comments
That would cause the opposite effect of what we’re actually seeing (i.e. “more redundant languages” would be using comparatively fewer tokens).
The real reason is that tokens are probably strictly based on n-gram frequency of the training data, and English is the most common language in the training data.