I would argue that there is indeed some kind of "color affinity", if we don't want to call it color intelligence. My father is a painter, and at some workshops people often asked him how to get some specific color or another. He told me that it was a problem he didn't understood, because he never encountered it even when he started painting. Colors just come easily to him, in a intuitive way, he never had to read theory about it.

To me, there is some parallelism with the guys who are gifted at math or logic : there are a lot of things that come intuitively to them where others have to struggle. The difference is that we are more prone to call them smart than people with other sort of abilities (musicians, painters, etc.).