Judgment of benign and early malignant colorectal tumors from ultrasound images with deep multi-View fusion

•A computer-aided diagnosis system is proposed based on deep neural network for judging benign and early malignant colorectal tumors.•The computer-aided diagnosis system proposed is the first one based on endorectal ultrasound images, and the first to adopt a multi-view fusion mechanism. The first multi-view endorectal ultrasound image dataset is constructed.•A new form of center loss was designed to optimize the model and achieves good results.•The performance of the proposed algorithm surpasses both expert diagnosis results and comparison methods.