Let's construct an example here.
Spain, nervous about a rapidly expanding France, invites England and Austria into an alliance. Spain is the alliance leader, only they can invite new members.
France, in response, creates an alliance with Scotland, Portugal, and Denmark (TAN: this must be a multiplayer game, such logic would NEVER occur to the AI) So France leads this alliance.
Now, England declares war on Scotland, and everyone honors the alliances.
For a French alliance member, the diplo screen would read At War with England, Spain and Austria. the Spanish members would have At War with Scotland, France, Portugal and Denmark.
Even though Spain and France are the (to use Castelion's terms) peacetime leaders, in this war, it is England and Scotland who have the right to sign a treaty for their entire alliance.
Hope this helps.