I think we need to be honest about the low merit of appealing to how well the average person's brain works.

This is the architecture equivalent of blaming software usability problems on the user. If the average person can't use your product or appreciate your building's design, then the design is flawed - end of story.