Comment on So, you want to chunk really fast?parentComments−snyyOP8moNo, delimiters can be multiple bytes. They have to be passed as a pattern.// With multi-byte patternlet metaspace = "<japanese_full_stop>".as_bytes();let chunks: Vec<&[u8]> = chunk(text).pattern(metaspace).prefix().collect();
Comments
No, delimiters can be multiple bytes. They have to be passed as a pattern.
// With multi-byte pattern
let metaspace = "<japanese_full_stop>".as_bytes();
let chunks: Vec<&[u8]> = chunk(text).pattern(metaspace).prefix().collect();