Analysis Of Benchmark Development Times In The Transaction Processing Performance Council And Ideas On How To Reduce It With A Domain Independent Benchmark Evolution Model

The Transaction Processing Performance Council (TPC) has a very successful history of disseminating objective and verifiable performance data to the industry. However, it lacks the ability to create new benchmarks in a timely fashion. In its initial years, the TPC defined benchmarks in about two years on average while recently this number increased to about eight years.Although TPC benchmarks measure fundamental performance characteristics of transaction systems, which do not change quickly over time, new feature development, e.g. SQL language cannot be ignored.This paper analyses the reasons for this increased development time and proposes ideas on how to improve benchmark development time by using a domain independent benchmark evolution model.