Analysis of Cataract Surgery Instrument Identification Performance of Convolutional and Recurrent Neural Network Ensembles Leveraging BigCat

Purpose: To develop a method for accurate automated real-time identification of instruments in cataract surgery videos. Methods: Cataract surgery videos were collected at University of Michigan's Kellogg Eye Center between 2020 and 2021. Videos were annotated for the presence of instruments to aid in the development, validation, and testing of machine learning (ML) models for multiclass, multilabel instrument identification. Results: A new cataract surgery database, BigCat, was assembled, containing 190 videos with over 3.9 million annotated frames, the largest reported cataract surgery annotation database to date. Using a dense convolutional neural network (CNN) and a recursive averaging method, we were able to achieve a test F1 score of 0.9528 and test area under the receiver operator characteristic curve of 0.9985 for surgical instrument identification. These prove to be state-of-the-art results compared to previous works, while also only using a fraction of the model parameters of the previous architectures. Conclusions: Accurate automated surgical instrument identification is possible with lightweight CNNs and large datasets. Increasingly complex model architecture is not necessary to retain a well-performing model. Recurrent neural network architectures add additional complexity to a model and are unnecessary to attain state-of-the-art performance. Translational Relevance: Instrument identification in the operative field can be used for further applications such as evaluating surgical trainee skill level and developing early warning detection systems for use during surgery.