Why is Beaver annoyed at Polly's party in the novel, A Handful of Dust?

Beaver is annoyed when Brenda leaves him alone because no one actually likes him, and he is thus in an awkward position at the party. At the same time he wants to make his way into the circle, and at Brenda's side, he's accepted, so he puts up with her behavior.