I disagree. I see that we do exactly that with neural nets. The distinctions you are trying to come up with sound semantic and arbitrary.

You disagree, and therefore you are wrong. Similar to how if I say the sky is blue and you disagree. The distinction sounds semantic and arbitrary but it is not. Think harder, the failure here is not a semantic difference but a failure in you to process the right abstraction. Your disagreement is irrelevant in the face of reality.

Take the neural net or several neural nets. Decompose those neural nets into modules. Recompose those modules into new neural nets. Can you do this? No. Why?

Because you can't really modularize neural nets. What these analysis techniques are doing is showing you that there is sort of a module like thing here but like brain surgery it doesn't mean you can rip it out and reuse it somewhere else. That's a true lack of understanding of what's going on.

We both agree that the human brain is a black box. We also agree that we know about the existence and location of modules in the human brain. Things like the "emotion" and "locomotion" are known modules. Let's say we meet a paraplegic person who's locomotion part of his brain is damaged. Can't we fix him by doing a transplant? A recently deceased patient who died of unrelated reasons could have his "locomotion" module cut and transplanted into the brain of the person who needs it.

We can't because we actually don't have access to the modules. Just a blurry picture that some sort of module is there. Same with artificial neural nets. The day you can graft a module in a neural net and compose it with another is the day you have fulfilled the definition of what a "module" is. You haven't and therefore you are utterly wrong and therefore you lack knowledge about what is going on inside a neural net. This is definitive logic.

You have made up the "by hand" requirement. The real world does not have it. If a scientist builds a neural network that generates (through machine learning) an algorithm that works, we all say that the scientist has written the algorithm, no one cares if they have done it by hand or through application of a clever meta algorithm.

Yes I have made it up to illustrate that there is more than a semantic difference. Look I'm not making up requirements here and there just to screw with you. I'm making them up so you can see there is actually a huge difference between training a neural network VS. doing meta programming.

A compiler is a meta programmer. You give it a high level language and it programs the CPU in a lower level language. There is a fundamental difference between what's going on here and what's going on when you train a neural net. There is a Functor between the difference in the process of creation to the level of understanding. WE have less control over the creation of weights in a neural net then we do over the assembly code a compiler generates just like how we have less understanding of the overall neural net then we do of the assembled program.

There is a clear gap here. I'm not literally setting a requirement here. My intention is to illustrate a gap in understanding and the gap seems to be permanent and a bulwark in our overall goals of understanding consciousness, not from your "requirement" perspective, but from scholars in the field.

literally a compiler "translates" code and a neural network is "trained." The word translate and train have more then a semantic difference.

If the "overintellectualizing" term is not common enough, let's replace it by "overthinking", it's very close.

There is literally only a semantic difference here. You eat your own words. From my perspective overthinking is not what's going on here. It's "underthinking"... what is an adjective to describe a person who "underthinks?"