Eliminating intrinsic interference in GFDM systems: Design of a complex-valued prototype filter

Prototype filter design is a critical challenge in generalized frequency division multiplexing (GFDM) systems. Poorly designed filters can result in inherent and out-of-band (OOB) interferences, severely impacting system performance. This paper proposes a novel solution to this problem by introducing an optimal prototype filter that mitigates the adverse effects of intrinsic interferences in GFDM systems. Our approach involves using a complex-valued prototype filter, which resembles a single-sideband (SSB) modulation scheme and significantly enhances bandwidth efficiency compared to conventional GFDM. To design the optimal filter, we formulate an optimization problem that eliminates inband and adjacent subcarriers intrinsic interferences. We also provide analytical expressions for evaluating the system's bit error rate (BER) and demonstrate the superiority of our optimized prototype filter over its current counterparts. To validate the design of the proposed prototype filter, we assess the noise enhancement factor (NEF). Our results indicate that our proposed filter improves system performance and reduces BER in GFDM systems.