Mixing Synthetic and Real Data to Build AI Vision Models

Searching millions of overhead images in order to identify objects that are of interest to national security missions presents a beneficial use case for AI models to assist human analysts. However, training AI models for target recognition typically requires large amounts of data with thousands of labeled examples. Labeling is expensive and, more importantly, in some cases sufficient examples do not exist to create an AI detection model, suggesting a need of synthetic data. We investigated multiple configurations for model training, including using various mixes of real and synthetic data, domain adaptation, and fine tuning of models. Creation of the best synthetic data via physics based simulation methods proved to be time consuming and still left a domain gap between synthetic and real data. Attempts to bridge this gap with domain adaptation suffered from model induced artifacts and still required fine-tuning with some real data to yield an improvement. While AI generated data provides less realism, it can be effective for creating a closed loop system between data generation and model development. Our results show that it is usually possible to use synthetic data to improve the performance of AI models compared to those trained solely on real data. However, the performance improvement for adding additional real data is significantly higher than for adding a similar number of synthetically generated samples.