Actually that is exactly when it is allowed. That said, if done, you are likely going to be in the same place as Zimmerman and have to justify that thought and subsequent action.
No, it is not exactly when it is allowed. As you seem to acknowledge in the follow up sentence, that is a part of it, but one needs to have a RATIONAL fear that their life is in danger.