Comment on GigaToken: ~1000x faster Language model tokenizationComments−phplovesong1moIs tokenizing really the bottleneck? If we go from 20ms to 15ms does it really matter?−gnabgib1moYou sound like someone who used to write fastruby−phplovesong1moNever heard of fast ruby. My point is tokenizing is rarely a bottleneck, as 99% of time is spent in inference.So you speed up 1% of the pipeline by some factor, and the end result is unobservable for a human.−antonvs1moTime to first token is observable by a human, and they’re reporting up to 10% reduction there.Plus, inference is not the only place tokenization happens. This can make a big difference during development of ML models.
Comments
Is tokenizing really the bottleneck? If we go from 20ms to 15ms does it really matter?
You sound like someone who used to write fastruby
Never heard of fast ruby. My point is tokenizing is rarely a bottleneck, as 99% of time is spent in inference.
So you speed up 1% of the pipeline by some factor, and the end result is unobservable for a human.
Time to first token is observable by a human, and they’re reporting up to 10% reduction there.
Plus, inference is not the only place tokenization happens. This can make a big difference during development of ML models.