what i really want to see from this article is a curve showing a tradeoff between speed and embedded text quality. there's the preamble that just going by character has quality problems, but i dont think delimiters are necessarily the best either, vs being able to find paragraph or even chapter boundaries.
how much of a problem is it that ~1 sentence per chunk gets corrupted in the by-character solution? what level of sentence corruption is left in by switching to these delimiters? what level of paragraph/idea corruption is left in with each? chapter/argument level?
Comments
what i really want to see from this article is a curve showing a tradeoff between speed and embedded text quality. there's the preamble that just going by character has quality problems, but i dont think delimiters are necessarily the best either, vs being able to find paragraph or even chapter boundaries.
how much of a problem is it that ~1 sentence per chunk gets corrupted in the by-character solution? what level of sentence corruption is left in by switching to these delimiters? what level of paragraph/idea corruption is left in with each? chapter/argument level?