Automated data verification in a format-free environment

Data collection and interpretation are vital for innumerable purposes: both commercial and academic. Sifting through vast mountains of data to separate correct information from incorrect can be expensive both in terms of money and of time. Automation of as much of this process as possible is the key to collecting useful information in an efficient and timely manner. This paper discusses a system designed to automate the comparison of raw collected data to store of previously verified data. This comparison can be used both to estimate the accuracy and the value of the collected data. In addition, it is possible to gauge the efficacy of various collection methods. In this system special attention was paid to accepting a wide range of document formats and to properly handling data sets whose attribute types might be differently organized than those in the reference data.