Who was responsible for Milton's death in the novel, We Begin at the End?

When Brandon decided to try to make peace with Milton by taking him out fishing, Brandon could not think of anything except how Milton scratched Brandon’s car during their trip. As a result, Brandon decided to play a joke on Milton by pushing him from the boat and making him swim back to the dock. Brandon was unaware that Milton did not know how to swim.