The tokens are in this case actually the individual characters:

    vocab = sorted(list(set(lines)))