Why do Fremen distrust off-worlders?

The Fremen distrust anyone not from Dune because of their eyes. The Fremen believe that anyone without the blue-on-blue eyes of a spice-addicted individual sees more than they should.