They are sorted, but in a way such that, roughly speaking, rank is proportional to inverse frequency (like Zipf’s law, but also permitting merging of subwords to be ranked). This is actually extremely important because it makes the otherwise very-high-cardinality categorical feature of target predicted argmax vocab dictionary key index slightly smoother and slightly more predictable for the model