Comment on Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAGComments−samlinnfer1yHow does it work for code? (Chunking code that is)−nostrebored1yPoorly, just like it does for text.Chunking is easily where all of these problems die beyond PoC scale.I’ve talked to multiple code generation companies in the past week — most are stuck with BM25 and taking in whole files.−potatoman221yWhat do they use BM25 for? RAG?−nostrebored1yCorrect -- finding the correct functions and files to include−bhavnicksmOP1yRight now, we haven't worked on adding support for code -- some things like comments (#, //) have punctuations that adversely affect chunking, along with indentation and other issues.But, it's on the roadmap, so please hold on!
Comments
How does it work for code? (Chunking code that is)
Poorly, just like it does for text.
Chunking is easily where all of these problems die beyond PoC scale.
I’ve talked to multiple code generation companies in the past week — most are stuck with BM25 and taking in whole files.
What do they use BM25 for? RAG?
Correct -- finding the correct functions and files to include
Right now, we haven't worked on adding support for code -- some things like comments (#, //) have punctuations that adversely affect chunking, along with indentation and other issues.
But, it's on the roadmap, so please hold on!