Who is the antagonist in A Passage to India by E. M. Forster?

While many consider Adela and Aziz to be antagonists to each other I am inclined to add colonialism/British imperialism as the main antagonist. The major characters and relationships are directly impacted by the British Empire and the cultural implications of it.