2CEE, A TWENTY FIRST CENTURY EFFORT ESTIMATION METHODOLOGY

There exists an extensive academic literature on software cost estimation that explores techniques such as boot strapping, assorted analogy methods such as nearest neighbor, and even highly non-linear 'models' such as decision trees. However, industry "best practice" virtually ignores the academic literature and continues to rely upon standard regression-based algorithms and most often local calibration. Local calibration only calibrates or tunes the main intercept and slope in a log-linear regression. Over the past three years our research has been investigating the behavior and performance of these various models and calibration/tuning techniques using machine learning methods. A summary of our preliminary findings was presented in 2006 at the 28th Annual Conference of the International Society of Parametric Analysts. While all of the analysis has been performed on NASA software project COCOMO data, the results should easily extend to systems and size estimation models. Our work cautions that current approaches to model specification and calibration can often produce sub- optimal models, which are likely to be a significant contributor to the cost growth exhibited by most software projects. This paper will provide an overview of the systemic cost estimation issues that have been identified, and a description of the best performing tuning techniques. While we have found that COCOMO is a very robust model, our results also indicate that local calibration using boot strapping over standard regression, combined with variable reduction (column pruning) and stratification (row pruning using nearest neighbor) is in the vast majority of experiments the most efficient and effective tuning method. Our research findings are captured in what we call the 21 st Century Effort Estimation Methodology (2cee). 2cee has been encoded in a Windows based tool that can be used to both generate an estimate and allow the model developer to calibrate and develop models using these techniques.