“I do not know! but why?” — Local model-agnostic example-based explanations of reject

Machine learning based decision making systems in safety critical areas place high demands on the accuracy and generalization ability of the underlying model. A common strategy to deal with uncertainties and possible mistakes is offered by learning with reject option, i.e. a model can refrain from prediction in ambiguous cases and leave the decision to a human expert. Yet, as for the models themselves, human decision-making is hampered by the fact that reject options are often implemented as black-box rules: Experts cannot readily understand the reasons for rejection.