So at least some of what LLMs do internally seems to involve a bit more structure than just linear combinations of some overcomplete dictionary.

Well that's good to hear. Would have been rather exasperating to find out we could have somehow gotten Fable or Sol out of sufficiently advanced sparse-coding ages ago.