There has been renewed interest in deriving tight bounds on the error performance of specific codes and ensembles, based on their distance spectrum. We discuss many reported upper bounds on the maximum-likelihood (ML) decoding error probability and demonstrate the underlying connections that exist between them. In addressing the Gallager bounds and their variations, we focus on the Duman and Salehi (see IEEE Trans. Commun., vol.46, p.717-723, 1998)variation, which originates from the standard Gallager bound. A large class of efficient bounds (or their Chernoff versions) is demonstrated to be a special case of the generalized second version of the Duman and Salehi bounds. Implications and applications of these observations are pointed out, including the fully interleaved fading channel, resorting to either matched or mismatched decoding. The proposed approach can be generalized to geometrically uniform nonbinary codes, finite-state channels, bit interleaved coded modulation systems, and it can be also used for the derivation of upper bounds on the conditional decoding error probability.