Why does Masao avoid visiting his daughter in the novel, What You Are Looking For Is in the Library?

Chie has her own apartment and works at the local bookshop. Although Masao's wife Yoriko often visits Chie at Meishin Books, Masao is reluctant to do so, afraid he would be intruding. However, after he and Yoriko pay Chie a visit at work at the start of Chapter 5, Masao realizes that he wants to rekindle his connection with his daughter. Their subsequent conversations about books, society, and dreams inspire and hearten Masao.