DFlow: Learning to Synthesize Better Optical Flow Datasets via a Differentiable Pipeline

Comprehensive studies of synthetic optical flow datasets have attempted to reveal what properties lead to accuracy improvement. However, manually identifying and verifying all such necessary properties are intractable mainly due to the requirement of large-scale trial-and-error experiments with iteratively generating whole synthetic datasets. To tackle this challenge, we propose a differentiable optical flow data generation pipeline and a loss function to drive the pipeline, called DFlow. These enable automatic and efficient synthesis of a dataset effective to a target domain, given a snippet of target data. This distinctiveness is achieved by proposing an efficient data comparison method, where we approximately encode reference sets of data into neural networks and compare the proxy networks instead of explicitly comparing datasets in a sample-wise way. Our experiments show the competitive performance of our DFlow against the prior arts in pre-training. Moreover, the RAFT model pre-trained with DFlow achieves state-of-the-art performance on the Sintel public benchmark in fine-tuning.