The mirror itself is held to the arm with a screw, if you're looking to remove the mirror to replace the glass part, just remove the screw located at the bottom of the arm holding the swivel ball.
If your arm is to be replaced as well, as gr8vfr and mattz280 said, just pull down on it. It's held to the mirror mount via a spring clip that fits into the mount. To replace just insert the clip and push in.
If the mount is to be replaced, if I recall there are two or three screws hidden underneath the arm.