Justifying Diagnosis Decisions by Deep Neural Networks.

•A holistic approach to justification for neural networks outperforms saliency maps.•Visualizing the nearest alternative diagnosis is a powerful, novel approach.•Creating a continuous textual representation is useful to bridge modalities.•Holistic, multi-tasking approach achieves excellent results in individual tasks.