Technical Note: 3D localization of lung tumors on cone beam CT projections via a convolutional recurrent neural network.

Purpose To design a convolutional recurrent neural network (CRNN) that calculates three-dimensional (3D) positions of lung tumors from continuously acquired cone beam computed tomography (CBCT) projections, and facilitates the sorting and reconstruction of 4D-CBCT images. Method Under an IRB-approved clinical lung protocol, kilovoltage (kV) projections of the setup CBCT were collected in free-breathing. Concurrently, an electromagnetic signal-guided system recorded motion traces of three transponders implanted in or near the tumor. Convolutional recurrent neural network was designed to utilize a convolutional neural network (CNN) for extracting relevant features of the kV projections around the tumor, followed by a recurrent neural network for analyzing the temporal patterns of the moving features. Convolutional recurrent neural network was trained on the simultaneously collected kV projections and motion traces, subsequently utilized to calculate motion traces solely based on the continuous feed of kV projections. To enhance performance, CRNN was also facilitated by frequent calibrations (e.g., at 10 degrees gantry rotation intervals) derived from cross-correlation-based registrations between kV projections and templates created from the planning 4DCT. Convolutional recurrent neural network was validated on a leave-one-out strategy using data from 11 lung patients, including 5500 kV images. The root-mean-square error between the CRNN and motion traces was calculated to evaluate the localization accuracy. Result Three-dimensional displacement around the simulation position shown in the Calypso traces was 3.4 +/- 1.7 mm. Using motion traces as ground truth, the 3D localization error of CRNN with calibrations was 1.3 +/- 1.4 mm. CRNN had a success rate of 86 +/- 8% in determining whether the motion was within a 3D displacement window of 2 mm. The latency was 20 ms when CRNN ran on a high-performance computer cluster. Conclusions CRNN is able to provide accurate localization of lung tumors with aid from frequent recalibrations using the conventional cross-correlation-based registration approach, and has the potential to remove reliance on the implanted fiducials.