Semantic segmentation for tracking moving anatomies in robotic surgery

Robotics and Artificial Intelligence (AI) can bring in- valuable benefits to clinicians in the surgical context. For instance, decision support systems and advanced autonomous systems for execution of standard repetitive sub-tasks will help reduce surgeon’s fatigue, optimiz- ing the outcome for the patient and minimizing usage of resources . With the recent advances of machine learning and particularly deep learning, AI systems can achieve outstanding scene understanding , providing the surgeon with intra-operative guidance , e.g., via reality augmentation . In the wide area of surgical robotic perception, one fundamental direction of research deals with the accu- rate tracking of laparoscopic tools for monitoring, skill assessment and extraction of expert gestures for developing dexterous autonomous robotic systems. However, several problems need still to be addressed. First, most deep learning-based systems require large datasets for robust training, as evidenced from the research boost to produce benchmarking datasets . Moreover, instrument tracking is not enough to achieve proper situation assess- ment in surgery, but also accurate anatomical recognition is needed. One prominent data-efficient solution for this task is semantic segmentation . However, existing applications consider static or quasi-static scene, hence not accounting for moving anatomies due, e.g., to surgeon’s manipulation and natural body motion. In this paper, we address the problem of intra-operative anatomy tracking, using semantic segmentation trained on few RGB images of a realistic phantom for partial nephrectomy. We employ the da Vinci Research Kit (dVRK) in our experiments. We show that our methodology achieves comparable accuracy to the state of the art, over a wide motion range in the field of view of the RGB camera