Why was Chillingworth the one who sinned greatly?

Chillingworth is clearly the major antagonist in the story. Really, Chillingworth didn't even want Hester back. He merely came to punish and torture the man who loved her. Chillingworth becomes consumed by his need to destroy Dimmesdale. Ironically Chillingworth's thirst for vengeance destroys him too.