The authors of the SARAS-ESAD 2020 dataset have compiled a collection of data, referred to as the ESAD (Expert Surgical Actions Detection) dataset, to address the challenges involved in Minimally Invasive Surgery (MIS). The success of MIS depends on the proficiency of human surgeons and the efficiency of their coordination. To introduce more automation in MIS, the authors’ SARAS consortium is working on the Smart Autonomous Robotic Assistant Surgeon (SARAS) project, aiming to replace the assistant surgeon with two assistive robotic arms. For this purpose, an artificial intelligence-based system is needed to understand the surgical scene, detect the actions performed by the main surgeon, and provide cues for the autonomous assistant surgeon. To create such a system, the SARAS endoscopic vision challenge for surgeon action detection was held in 2020.

The ESAD dataset is a benchmark for action detection in the surgical domain, comprising four sessions of complete prostatectomy procedures performed by expert surgeons on real patients with prostate cancer. The dataset is unique as it involves recordings from the da Vinci Xi robotic system, integrated with a binocular endoscope, which provides detailed visual information during different stages of the operation. The videos used in the dataset are monocular.

The following guidelines were enforced for the annotation:

• Each bounding box should contain both the organ and tool acting for consideration, as each action class is highly dependent on the organ under operation.
• To balance the presence of tools and organs or tissue in a bounding box, bounding boxes are restricted to containing 30%-70% of either tools or organs.
• An action label is only assigned when a tool is close enough to the appropriate organ, as informed by the medical expert. Similarly, an action stops as soon as the tool starts to move away from the organ.
• Each video frame can have two actions, whose bounding boxes are allowed to overlap.

The ESAD dataset is divided into three sets: train, validation, and test. The training data contains 22,601 annotated frames with 28,055 action instances, while the validation data consists of 4,574 frames with 7,133 action instances. The test data, yet to be released, will contain 6,223 annotated frames with 11,565 action instances.

For the creation of the dataset, the authors collected complete prostatectomy procedure videos with the consent of patients and the hospital. Each video is approximately 2 hours and 20 minutes long, and annotation is performed at 1 FPS to maintain scene variation. The dataset includes 21 different action classes, each representing specific surgical actions performed during the procedure. The classes were carefully selected in consultation with multiple surgeons and medical professionals to strike a balance between simplicity and complexity. The dataset faces class imbalance, reflecting the nature of surgical procedures, with common actions having more samples than rarer actions.

Detailed guidelines were provided to annotators to standardize the annotation process, ensuring that each bounding box contains both the tool acting and the organ under operation, as these factors significantly influence action detection. The ESAD dataset is a valuable resource for advancing the development of an online surgeon action detection system and introducing partial/full autonomy in surgical robotics.