From a philosophical standpoint one can be extremely wealthy and feel unfulfilled in life, and conversely someone can be extremely poor by material measures and be completely at peace with themselves. Given the overall context of the poem, however, and era in which it was written, he seems to be making more of a point about those people who find it very easy to sit around pointing out the problems in the world, but don't want to lift a finger to be the change they want to see in the world.
