Validating high level simulation results against experimental data and low level simulation: a case study

Simulation can be considered a necessary evil in the validation of systems, especially when the system under consideration is being prototyped and therefore does not presently exist. This is compounded by the use of high level simulators; on the one hand, high level simulation is efficient, in that it abstracts away many details of the system which are deemed to be not important. This allows for a simpler and faster running simulator, which allows the user to obtain results faster and/or perform more experiments. On the other hand, some of the details abstracted away might turn out to be important, introducing inaccuracies. This paper outlines a framework for the statistical understanding and attribution of the errors produced by a high level simulator when compared against real experiments by means of a low level simulator. This allows the user of a simulator to determine whether or not the inaccuracies are significant, and whether or not the high level simulator requires refinements in its accuracy for the results to be valid. These techniques are illustrated via a case study.