What major misinterpretation makes David leave town?

One night, however, he overhears the man who gave him the bread talking with some of the other people in the town about him. Fearing that their conversation means they are going to contact "THEM" (his captors) to come find and recapture him, David flees to his cave, where he collects his belongings and weeps briefly over the fact that he has to leave (see "Quotes," p. 50). He realizes, however, that he is still free, remembers everything he has learned, and resolves to keep living and keep being free. In the middle of the night, he leaves the once-welcoming town behind and continues his journey north towards Denmark.