Continuous Integration service at Fermilab

This paper describes the current architecture of Continuous Integration (CI) service developed at Fermilab, encountered successes and difficulties, as well as future development plans. The current experiment code has hundreds of contributors that provide new features, bug fixes, and other improvements. Version control systems help developers to collaborate in contributing software for their experiments, while the CI system helps developers keep their code healthy. The Fermilab CI service allows experiments and projects to test and validate their offline production and analysis code on the supported platforms. It is built on top of Jenkins, designed to be set up from a configuration file that provides implementation for each phase of the CI workflow, and able to validate experiment code through grid jobs. This CI service provides a dashboard for easy access to logs and statistical graphs. Since the CI service has been adopted by Fermilab experiments/projects, it proved to be very useful to intercept issues in their code early on and get them fixed before running it in production. Currently the CI service is in use by the ArgoNeuT, DUNE, g-2, LArIAT, MINERvA, mu2e, NOvA, SBND and uBooNE experiments and by the following projects: ART and LArSoft software suites, GENIE, and GlideinWMS. The CI service is under active development and planning to support code profiling.