Automatic knowledge-based recognition of low-level tasks in ophthalmological procedures

Purpose Surgical process models (SPMs) have recently been created for situation-aware computer-assisted systems in the operating room. One important challenge in this area is the automatic acquisition of SPMs. The purpose of this study is to present a new method for the automatic detection of low-level surgical tasks, that is, the sequence of activities in a surgical procedure, from microscope video images only. The level of granularity that we addressed in this work is symbolized by activities formalized by triplets < action, surgical tool, anatomical structure > . Methods Using the results of our latest work on the recognition of surgical phases in cataract surgeries, and based on the hypothesis that most activities occur in one or two phases only, we created a light-weight ontology, formalized as a hierarchical decomposition into phases and activities. Information concerning the surgical tools, the areas where tools are used and three other visual cues were detected through an image-based approach and combined with the information of the current surgical phase within a knowledge-based recognition system. Knowing the surgical phases before the activity, recognition allows supervised classification to be adapted to the phase. Multiclass Support Vector Machines were chosen as a classification algorithm. Results Using a dataset of 20 cataract surgeries, and identifying 25 possible pairs of activities, a frame-by-frame recognition rate of 64.5 % was achieved with the proposed system. Conclusions The addition of human knowledge to traditional bottom-up approaches based on image analysis appears to be promising for low-level task detection. The results of this work could be used for the automatic indexation of post-operative videos.