Why do the people think the Blackwood House is haunted in the novel, We Have Always Lived in the Castle?

The Blackwood House is not in the middle of the village, but it is on the way to the highway from town. It is one of the nicest estates in the area, but ever since the murders, people think it is haunted.