A Dual-branch Convolutional Network Architecture Processing on both Frequency and Time Domain for Single-channel Speech Enhancement

Single-channel speech enhancement aims to remove the interfering noise and reverberation in real environments by a single microphone, which is a very challenging task in the speech signal processing field. Over the past years, deep learning has shown great potential for speech enhancement. In this paper, we propose a novel real-time framework, called DBCN, which is a dual-branch architecture. One branch takes waveform as its input for time-domain modeling and the other one takes shift real spectrum as input for frequency-domain modeling. The two branches have the same network structure, which is the representative convolutional recurrent network. To exchange information sufficiently, a bridge module is added between the two branches. Furthermore, we propose a novel feature normalization approach that enables each band to complete the normalization independently by counting the root mean square of each band and obtaining the inter-frame relationship for each band. The proposed approach allows the network to ignore the magnitude during processing, reducing learning difficulty and improving performance. Systematical evaluation and comparison are conducted. Experimental results show that the proposed system substantially outperforms related algorithms for causal and non-causal speech enhancement under very challenging environments.