Why couldn't the mages defend their own Towers?

The mages, although being skilled, couldn't defend their own tower against a hoard of people, as every time they utilize a spell it is lost from their memory until they re-read their spell book. Not only do the mages forget the spells, but it costs them physically to cast magic, as their body is sapped of some of its life force for each utilization of magic.