Comment on So, you want to chunk really fast?parentComments−ks20488moDo the delimiters have to be single bytes? e.g. Japanese full stop (IDEOGRAPHIC FULL STOP) is 3 bytes in UTF-8.−snyyOP8moNo, delimiters can be multiple bytes. They have to be passed as a pattern.// With multi-byte patternlet metaspace = "<japanese_full_stop>".as_bytes();let chunks: Vec<&[u8]> = chunk(text).pattern(metaspace).prefix().collect();
Comments
Do the delimiters have to be single bytes? e.g. Japanese full stop (IDEOGRAPHIC FULL STOP) is 3 bytes in UTF-8.
No, delimiters can be multiple bytes. They have to be passed as a pattern.
// With multi-byte pattern
let metaspace = "<japanese_full_stop>".as_bytes();
let chunks: Vec<&[u8]> = chunk(text).pattern(metaspace).prefix().collect();