Multiple-Input-Multiple-Output (MIMO) wireless communication systems have attracted an enormous interest in both academy and industry due to the potential to provide remarkable spectral efficiency by taking advantage of multipath fading instead of combating it. Analysis of classic channel models under basic assumptions has demonstrated possibility to scale up linearly the information rate by deploying multiple transmitting and receiving antennas within the same frequency bandwidth. Recent works show that the achievable performance of practical MIMO systems depends heavily on the underlying fading distribution and system configuration. The spectral efficiency may severely degrade due to the high correlation between multipath components, channel rank deficiency or power imbalance.

The objective of this thesis is to study the capacity of correlated, rank-deficient and full-rank MIMO channels. While in many cases the exact capacity expressions are complicated and do not allow for significant insight, an asymptotic approximation for a large number of antennas is used to obtain simpler and well tractable results for a broad class of MIMO channels. Starting from the single keyhole channel as a basic and simple model of a rank-one MIMO channel, the analysis is extended to a family of higher-rank channels (including also the canonical full-rank Rayleigh-fading one) via a transition model which includes a number of statistically independent keyholes (multi-keyhole channels). It is shown that under certain mild conditions on correlation, the outage capacity distribution of the single keyhole and multi-keyhole channels is asymptotically Gaussian. The general conditions and propagation-based implications of the convergence are studied. In some cases, the asymptotic outage capacity distribution follows closely the exact one for a reasonably small number of transmit and receive antennas.

A number of applications of the asymptotic theory are discussed. (i) A new scalar measure of correlation and power imbalance is introduced to quantify the impact of correlation on the capacity and evaluate effective degrees of freedom in MIMO channels. The measure is simple, well tractable and full-ordering (any two channels can be compared). (ii) Finite SNR size-asymptotic diversity-multiplexing trade-off (DMT) is analyzed for the multi-keyhole channels. Unlike the SNR-asymptotic DMT of Zheng and Tse, the size-asymptotic one accurately represents the effect of correlation and power imbalance on the capacity. (iii) Telatar’s conjecture is proven for multi-keyhole channels with a large number of antennas. (iv) The best multipath angular density, which maximizes the asymptotic capacity of a broad class of MIMO channels with linear uniform antenna arrays, is derived. The density is non-uniform, which implies that the popular Clarke’s (Jakes) model does not represent the best case scenario.

Using the rigorous methods of hypothesis testing, it is demonstrated that the outage capacity distribution of some measured 5.2GHz indoor MIMO channels is statistically Gaussian with a reasonable significance level already for two antennas at each end. The latter can serve as an empirical validation of the obtained theoretical results and implies that the asymptotic analysis with respect to the number of antennas not only offers a significant insight and simplification, but also can be applied to realistic systems of a moderate size.