Selective feature connection mechanism: Concatenating multi-layer CNN features with a feature selector

•We propose a novel method termed SFCM to fuse CNNs features of different layers.•Our SFCM can be implemented by direct connection and residual connection.•Our SFCM can be used in many existing frameworks to achieve better performance.•We evaluate the SFCM on many different challenging computer vision tasks.