Why do Saul and Sofia not see one another much after they sleep together?

After Sofia and Saul sleep together, Saul can barely meet her eyes at the next Sunday dinner. What they have done seems like a wall between them. Then it is Christmas, New Year's, and the first weeks of 1942, which are very busy for Joey and his team. Saul keeps strange hours, and Sofia goes on solitary walks. Therefore, they rarely see one another.