What does the author claim to be the only way you can be destroyed in "My Dungeon Shook"?

Baldwin writes, both referencing his father and other African Americans, "You can only be destroyed by believing that you really are what the white world calls a nigger." He says the same of his father, that his destruction came about because he believes what white men said of him.