Why does the hunter assume Phoenix is making a trip to town in the story, A Worn Path?

When he learns that she intends to walk to town, the hunter assumes Phoenix is not able to make the long journey and he tells her to go home; he has no qualms about issuing the order. But when she persists, he relents, assuming that the only reason "old colored people" would embark on such a long trail would be to see Santa Claus.