Agility measures your character's flexibility and balance. It adds to Acrobatics and Stealth skill checks, and is the defence primarily targeted by physical attacks.
The Duelist and the Warrior classes both rely on Agility.