and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.

The hash just has to uniquely identify the contents. I still don't see what stops you from walking the chunk tree by chunks of characters instead of chunks of tokens, then lazily finding the token boundary once you've found the longest common chunk prefix and also (in parallel) tokenized the input.