The difference is packaging it as a consumable PyPI package that can easily be used in a project (they even call out for separating this out into a stand alone project but that they lack the time to do so: https://docs.sweep.dev/blogs/chunking-2m-files#future- )
In addition, we expand and fix the implementation, for example it now supports limiting on token count instead of character count, and we fix some white space inconsistencies in parsing/chunk reconstruction.
Comments
Sweep is mentioned as attribution in multiple places a) https://github.com/definitive-io/code-indexer-loop#attributi... b) https://github.com/definitive-io/code-indexer-loop/blob/fd9d...
The difference is packaging it as a consumable PyPI package that can easily be used in a project (they even call out for separating this out into a stand alone project but that they lack the time to do so: https://docs.sweep.dev/blogs/chunking-2m-files#future- )
In addition, we expand and fix the implementation, for example it now supports limiting on token count instead of character count, and we fix some white space inconsistencies in parsing/chunk reconstruction.
Looking forward to diving further in and seeing how you handle white spaces issues as I’ve had massive issues with that already
Please let me know if anything isn’t as expected and I’ll try to look into it