Comment on GigaToken: ~1000x faster Language model tokenizationparentComments−marcelroed1moYes! I will publish a Rust crate soon. If you have thoughts about how to structure the API I would love to hear them.
Comments
Yes! I will publish a Rust crate soon. If you have thoughts about how to structure the API I would love to hear them.