Who is the antagonist in Ender's Game by Orson Scott Card?

Although the buggers are seen as the overall antagonist, Ender comes to see the adults and his brother Peter as the real antagonists. Adults like Colonel Graff control and manipulate children while Peter is just a general sociopath.