Prospero was more into his studies and neglected his duties as duke. Antonio had no choice but to consolidate Prospero's power. He was a "poor man" because he neglected his people, towns people, and prefered to read his books instead. His library was his "dukedom" (what he ruled over) and for that, he lost his kingship.