Why does Hightower remember his parents as nothing more than ghosts in his life in the novel, Light in August?

Hightower thinks of his parents as ghosts because his mother was an invalid for most of his childhood, and his father was emotionally absent.