A Design of Spectrally-Efficient Low-Complexity QAM-FBMC Systems With Mismatched Prototype Filters

In this article, we propose a low-complexity quadrature amplitude modulation (QAM) filter-bank multicarrier (FBMC) system with high spectral efficiency and low self-interference. To circumvent the limitations imposed by the Balian-Low theorem on the interference suppression capability of QAM-FBMC systems, the proposed system relaxes time-frequency (TF) product at the transmitter (TX) and time-domain localization at the receiver (RX). The PHYDYAS filter is employed as a TX prototype filter, and a filter mismatched to the TX filter is employed as an RX prototype filter. Since the PHYDYAS filter has excellent spectral confinement, its use well compensates for the loss in TF-product and leads to higher spectral efficiency than the conventional systems in multi-user uplink. The RX has an extended observation window and a separate equalizer followed by a fixed pre-computed filter. To find the RX prototype filter that minimizes the self-interference, an optimization problem is formulated based on a bi-orthogonality condition, subject to the frequency-domain sparsity constraint in terms of the $\ell _{0}$ -norm. To efficiently search for the solution, a modified relaxed LASSO procedure is proposed, where the size of the problem is reduced to make the problem numerically tractable. Numerical results and discussions show that the proposed QAM-FBMC system strikes an excellent balance between performance and complexity.