Who refuses to help Elijah free the slaves at East Lee stable? Why?

The people of the small village where Mr. Highgate was shot refuse to help Elijah free the chained slaves because they know that if they help they will be killed or forced into slavery themselves. The men do not think saving a few is worth the risk of the others being harmed.