Who is the antagonist in the novel, James?

The institution of slavery is the main antagonist. The Duke, the King, and Mr. Wiley also serve as antagonistic characters.