Integrating expert and novice evaluations for augmenting ordinal regression models

•A strategy of incorporating rankings in ordinal regression models is proposed.•Improvement in performance of the strategy is proved.•Results of real-world study were consistent with those of synthetic studies.