Simple to Complex Cross-modal Learning to Rank.

•We learn a more optimal multi-modal embedding space gradually from easy to more complex rankings.•We employ non-linear mapping functions to establish the multi-modal embedding space for more sophisticated cross-modal correspondence.•An efficient alternative algorithm is exploited to solve the proposed challenging problem with a fast convergence in practice.