Why is M.C. forbidden to play with Ben in the novel, M.C. Higgins, the Great?

M.C.'s father has forbidden M.C. to play with Ben because his family, with their six-fingered hands, are thought to be "witchy."