How does Jimmy fall out with Hugh?

Hugh leaves purportedly to go abroad to work on his novel, and although he invites Jimmy and Alison along for the trip, Jimmy ends up refusing and ending their friendship.