Comment on GigaToken: ~1000x faster Language model tokenizationparentComments−gnabgib1moYou sound like someone who used to write fastruby−phplovesong1moNever heard of fast ruby. My point is tokenizing is rarely a bottleneck, as 99% of time is spent in inference.So you speed up 1% of the pipeline by some factor, and the end result is unobservable for a human.−antonvs1moTime to first token is observable by a human, and they’re reporting up to 10% reduction there.Plus, inference is not the only place tokenization happens. This can make a big difference during development of ML models.
Comments
You sound like someone who used to write fastruby
Never heard of fast ruby. My point is tokenizing is rarely a bottleneck, as 99% of time is spent in inference.
So you speed up 1% of the pipeline by some factor, and the end result is unobservable for a human.
Time to first token is observable by a human, and they’re reporting up to 10% reduction there.
Plus, inference is not the only place tokenization happens. This can make a big difference during development of ML models.