What does The Giver say is the worst part of holding the memories?

The Giver says that the worst part of holding the memories isn't the pain of them, but rather the loneliness that comes with holding the memories and being unable to share them.