Why does Adam tell Isra not to wear her hijab when he takes her out for pizza in the novel, A Woman Is No Man?

From the text:

“You think you're doing the right thing by wearing this hijab, but that’s not what Americans will see when they look at you. They won’t see your modesty of your goodness. All they’ll see is an outcast, someone who doesn’t belong” (56).