Non-Interaction of Color

At art school there are moments where a jury looks at your work. During one of those moments, a member of this jury asked me about my choice of colours for a certain work. I replied plainly that in some way the colours were arbitrary. That colour is merely a way to differentiate and that my particular choice of colours, shades of the primary and secondary variety, was made because I wanted the largest amount of difference between some objects that were otherwise equal in their appearance.
This answer was met with derision and frustration, as if I was being deliberately obtuse. Which surprises me even now, half a decade later.
The differentiation between two similar things is exactly one of the basic functions of human colour perception. It is a signal that this thing is not the same as that thing. Using specific colours can thus establish relationships through association without altering other characteristics.
To give an example, here are nine equal fishes:

 And then there are nine different ones:

 And here there are 7 different fishes, but some are clearly closer related than others:

The differentiating function is one of the most basic characteristics of colour, for better or for worse. It is inescapable, as is evidenced in antithesis by the decidedly unreal appearance of works by Hans op de Beeck and more plainly confirmed by the use of colour in thermography.
So with it's existence and importance so patently clear, why was any doubt ever cast upon this fact?