This is all very fascinating. Neuroscientists have weighted in and think it's because of the brains top down processing method. This is very similar to the Adleson Checkerboard illusion. Basically, your brain perceives signals from the world around based on experiences. I see a white dress with gold, but the white looks like it's in a shadow and the material is slightly reflective. Some people see blue and black, but the blue portion is not reflective at all.
It has nothing to do with anyone's eyes. The eyes don't see, the mind sees, but the eyes are the sense organ for it. I also think it could have directly to do with the environment one is seeing from. I forget exactly what the phenomenon is called but it works like this: Your brain has neurons that do very very specialized things. That said, it has neurons that specifically put together images using specific colors. If you are in an environment where the neurons are worked a lot they actually get tired. Then you see something that works the others. These neurons are in a tug of war with each other when they do opposite things. Due to the fatigue of them being worked, when you see something of a specific opposite then the rested neurons actually are more effective at pulling that rope, so now you see something a different way due to the neurons who are rested now winning that tug of war.
The above is also why you experience a phenomenon called "updrift" after watching something stream downward and you look away. It was probably the most famous sensation from Guitar Hero that was not intended.
Remember, your neurons are always in a tug of war, and this will change your perception of how things appear.
