Who is the antagonist in Up at the Villa by W. Somerset Maugham?

Matthew Panton, Mary's deceased husband, although he never physically appears in the novel would be considered the antagonist. All of the stress and grief in her life are a direct result of Matthew's thoughtless and life changing actions.