In Chapter 2 of the book Great Expectations, why didn't Pip think of the household as belonging to Joe also?

In his mind, Joe separated the home he'd grown up in from Joe's forge. The house was their, the adjoining forge beloned to Joe.

Joe’s forge adjoined our house, which was a wooden house, as many of the dwellings in our country were,—most of them, at that time.