Comment on Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAGparentComments−nostrebored1yPoorly, just like it does for text.Chunking is easily where all of these problems die beyond PoC scale.I’ve talked to multiple code generation companies in the past week — most are stuck with BM25 and taking in whole files.−potatoman221yWhat do they use BM25 for? RAG?−nostrebored1yCorrect -- finding the correct functions and files to include
Comments
Poorly, just like it does for text.
Chunking is easily where all of these problems die beyond PoC scale.
I’ve talked to multiple code generation companies in the past week — most are stuck with BM25 and taking in whole files.
What do they use BM25 for? RAG?
Correct -- finding the correct functions and files to include