Comment on GigaToken: ~1000x faster Language model tokenizationparentComments−ubedan1moGithub: https://github.com/simdjson/simdjsonIt showcases an especially ingenious scheme for escaping json strings, as well as the other tokens necessary to parse json.Daniel Lemire - One of the authors at QCon 2019: https://www.youtube.com/watch?v=wlvKAT7SZIQAnother Youtube video that explains it: https://www.youtube.com/watch?v=vd9J9PPmAMM
Comments
Github: https://github.com/simdjson/simdjson
It showcases an especially ingenious scheme for escaping json strings, as well as the other tokens necessary to parse json.
Daniel Lemire - One of the authors at QCon 2019: https://www.youtube.com/watch?v=wlvKAT7SZIQ
Another Youtube video that explains it: https://www.youtube.com/watch?v=vd9J9PPmAMM