Why does Skiffington go to Mildred's house looking for Moses.

Skiffington is told by one of the slaves that Moses has a bad sense of direction. Skiffington realizes that Moses might have gone South, taking him in the direction of Mildred's house.