Comment on Show HN: Chonkie – A Fast, Lightweight Text Chunking Library for RAGComments−petesergeant1yWhat other chunking strategies would be useful for RAG applications?I’m using o1-preview for chunking, creating summary subdocuments.−bhavnicksmOP1yThat's pretty cool! I believe a research paper called LumberChunker recently evaluated that to be pretty decent as well.Thanks for responding, I'll try to make it easier to use something like that in Chonkie in the future!−petesergeant1yAh, that's an interesting paper, and a slightly different approach to what I'm doing, but possibly a superior one. Thanks!
Comments
I’m using o1-preview for chunking, creating summary subdocuments.
That's pretty cool! I believe a research paper called LumberChunker recently evaluated that to be pretty decent as well.
Thanks for responding, I'll try to make it easier to use something like that in Chonkie in the future!
Ah, that's an interesting paper, and a slightly different approach to what I'm doing, but possibly a superior one. Thanks!