Informing multi-objective optimisation benchmark construction through Instance Space Analysis

The role of carefully constructed benchmark suites in algorithm design and testing is critical. Within the continuous multiobjective optimization domain, existing suites include the general purpose ZDT, DTLZ, and WFG suites, and more recent ones specifically designed to explore the impacts of a particular problem characteristic. However, the relationship between existing suites is not clear, and the field would benefit from a “stock-take” assessment. This article investigates the coverage of current continuous multiobjective suites using the instance space analysis (ISA) methodology. Exploratory landscape analysis is used to measure critical features of each problem suite. Thereafter, we generate a 2-D visualization of the existing problem instances by locating them in the instance space, assessing their diversity, and identifying whether there are sparse areas of value to fill with new problem instances. Our findings show that the current suites are restricted in diversity when representing the entire problem instance space. We propose and evaluate three problem construction methods: 1) problem tuning; 2) toolkit hybridization; and 3) new function injection. Problem tuning is shown to generate problems surrounding existing instances, while hybridization creates problems falling between existing suites. Furthermore, utilizing the insights afforded by ISA, we show how problem features can be identified to inform the creation of new functions which fill gaps toward the boundaries of the instance space.