Comment on GigaToken: ~1000x faster Language model tokenizationparentComments−cs7021moThat is my reaction too. It looks like great work!Valuable not only for inference, but for training too (think proprietary datasets).I would add, a single individual did this.One person can make a difference :-)
Comments
That is my reaction too. It looks like great work!
Valuable not only for inference, but for training too (think proprietary datasets).
I would add, a single individual did this.
One person can make a difference :-)