Text models receive tokens(akin to morpheme chunks) and don't have access to the underlying letters
Heh, this gives me the idea of training an image generator not on tokens but on rendered text of the prompt in some 8x8 font, that would be a fun experiment ;)
I'd also be curious how text rendering performance changes if the tokenizer could be made aware of quoted strings and instead tokenizes the contents as characters instead. Surely someone has tried this, (right?) but I haven't seen it in the literature.
Comments
Heh, this gives me the idea of training an image generator not on tokens but on rendered text of the prompt in some 8x8 font, that would be a fun experiment ;)
That would be an interesting experiment!
I'd also be curious how text rendering performance changes if the tokenizer could be made aware of quoted strings and instead tokenizes the contents as characters instead. Surely someone has tried this, (right?) but I haven't seen it in the literature.