Data, depth, and design: Learning reliable models for skin lesion analysis

•Optimizing and evaluating models on the same dataset lead to overoptimistic results.•We evaluate the impact of 9 factors from deep-learning for skin lesion analysis.•We show the importance of ensembles and transfer learning to provide good results.•We achieve competitive AUC in all 3 classification tasks of the ISIC Challenge 2017.•We make all our code, data, and procedures available to the community.