Oh absolutely, the article gave me quite a bit to think about. It wasn't until I sat down and tried swapping a fourier transform/representation into the conjectures that I was able to think critically on the topic.

I suspect that the pruning operation is useful to consider mathematically. A fourier transform is a universal approximator - but only has useful approximation power when the basis vectors have eigenvalues which are significant for the problem at hand (PCA). If NN's replace that condition with a topological sense of utility. Then that is a major win (if formalized).