Task-Incremental Medical Image Classification with Task-Specific Batch Normalization

Currently, multiple intelligent diagnosis systems are developed independently partly due to the difficulty in collecting training data for all tasks of disease diagnosis. It would be better if an intelligent diagnosis system can incrementally learn more diagnosis tasks as what general practitioners have done. Existing approaches to such a task-incremental learning problem still more or less suffer from the catastrophic forgetting issue, i.e., system performance on old tasks is often gradually decreased when the system incrementally learns new tasks. Here a simple but effective approach is proposed to solve the catastrophic forgetting of old task knowledge for task-incremental learning. Specifically, when training a convolutional neural network (CNN) classifier to incrementally learn more tasks, the kernels in the task-shared CNN feature extractor are initially learned or fine-tuned from the first task and then fixed through subsequent learning tasks, and only the task-specific batch normalization parameters in the CNN feature extractor and the new task-specific classifier head are learned and stored for each new task. Empirical evaluation on medical image datasets supports that the task-specific batch normalization provides an effective solution for an intelligent system to incrementally learn multiple tasks of disease diagnosis. The source code will be released publicly.