What or who does Mr. Ellenwood blame for his unhappy life in the short-story, The Wedding-Knell?

Mr. Ellenwood says that much of his life’s unhappiness was caused by his love for Dabney and the fact that she married other men.