ACRoBat: Optimizing Auto-batching of Dynamic Deep Learning at Compile Time

Dynamic control flow is an important technique often used to designexpressive and efficient deep learning computations for applications such astext parsing, machine translation, exiting early out of deep models and so on.The control flow divergence resulting from dynamic control flow makes batching,an important optimization enabling high throughput and hardware utilization,difficult to perform manually. In this paper, we present ACRoBat, a frameworkthat enables efficient automatic batching for dynamic deep learningcomputations by performing hybrid static+dynamic compiler optimizations andend-to-end tensor code generation. ACRoBat performs up to 8.5X better thanDyNet, a state-of-the-art framework for automatic batching, on an NvidiaGeForce GPU.