What stops the narrator from responding to Joe's voice?

The narrator hears annoyance in Joe's voice, and this gives her pause. She was willing to believe that she and Joe could communicate, and that she was able to accept a kind of captivity with him, but his tone stops her from approaching him.