DisseminationTERRINet SUCCESS STORY: Robotics Surgical Tasks Data Acquisition on the da Vinci Research Kit Platform | TERRINet

29 March 2021

Irene Rivas Blanco is an Assistant Lecturer in the Department of Systems Engineering and Automation at the University of Malaga (Malaga, Spain). She applied to the TERRINet project to build a dataset of robotic surgical tasks using the Da Vinci Research Kit platform (dVRK), offered by the TERRINet partner The BioRobotics Institute – SSSA (Pontedera, Italy).

Da Vinci Research Kit (Figure 1) is the leading reference in the field of surgical robotics. The platform consists of a surgeon’s console to tele-operate the surgery and a patient side system where the surgery takes place. The platform enables the execution of complex motions due to additional degrees of freedom of the instruments. The users can perform the motions that are not possible on other platforms.

Figure 1: Da Vinci Research Kit (dVRK) offered by The BioRobotics Institute – SSSA (Pontedera, Italy).

Designing collaborative strategies for robots to aid surgeons in performing certain actions with autonomy can lead to increased precision and accuracy of robot-assisted operations, improved consistency in treatments, and higher dexterity and access to tissues.

Building a Dataset of Surgical Tasks for Autonomous Auxiliary Assistance

Surgical datasets containing surgical videos of real interventions are no novelty to the public. So far, they were mostly used for the recognition of surgical actions.

The objective of this experiment was to build a dataset of surgical tasks with different levels of complexity combining camera image and robotic system sensory data, which would serve for further exploration of deep learning techniques for autonomous auxiliary tasks in laparoscopic procedures.

Surgical Tasks Addressing Motions and Skills Necessary for Laparoscopic Surgery

The experiment implementation was divided into two access periods – the preparation period (July 2019) and the execution period (February 2020).

The preparation period was devoted to getting trained for the platform usage, software preparation and definition of the surgical tasks performed by the users. The surgical tasks were selected based on the SAGES manual skills in laparoscopy proposition (Choy, 2012).

In the execution period, 16 users performed the surgical tasks to obtain the required data (Figure 2). 10 users were male and 6 females ranging between 21 and 42 years old. Each user performed 6 trials of each exercise – 3 trials using the right hand and 3 using the left hand. 


Figure 2: Experimental setup conducted in the execution period. On the left picture: the master console of dVRK, where the operator tele operatively controls the instruments while watching the operational site in a 3D viewer. On the right picture: the slave arms are reproducing the operator movements.

The surgical tasks were the following:

  • Post and Sleeve: The goal was to move the coloured sleeves from side to side, whereas the user was allowed to take a sleeve with one hand, pass it to the other, and place it over a peg on the other side.
  • Pea on a Peg: The goal was to take beads from the cup and place them on top of the pegs using only a single hand.
  • Wire Chaser: The goal was to move a ring from one side to the other using only one hand.
Figure 3: The experimental tasks: (a) Post and Sleeve, (b) Pea on a Peg, and (c) Wire Chaser.

The surgical tasks addressed motions and skills necessary for laparoscopic surgery, such as hand-eye coordination, bimanual dexterity, depth perception, and interaction between the dominant and non-dominant hand. The teleoperation of the tools was performed with haptic guidance, using the dVRK console. The control of the teleoperation was carried out with a ROS software provided by the host institution.

Dataset of Surgical Tasks Based On 282 Recordings

The final dataset contained 282 recordings in total – 92 of post and sleeve task, 95 of pea on a peg, and 95 of wire chaser. The parameters were recorded on the master tool manipulators (44 recordings), patient side manipulators (80), and the pedal (40).

The mean scores of the surgical tasks (Figure 4) were 138.72 for post and sleeve, 152.25 for pea on a peg, and 42.94 for wire chaser. The mean score of the latter was lower due to a shorter duration of the task.

Figure 4: The distribution of the total score of the 282 trials divided by exercise using boxplots.

The dataset included the following data:

  • Images (each exercise was recorded using the mp4 format),
  • Kinematic data (kinematic data of the two master tool manipulators and patient’s side manipulators),
  • Skills evaluation (the user performance evaluation based on time and task efficiency),
  • Questionnaire (about the personal data, experience with teleoperated systems and visuo-motor skills).

“The TERRINet project was a unique opportunity to work with da Vinci Research Kit that I can’t access at my home institution. It was an amazing experience to get to know the platform, work on it, and collect data, which enabled further research on the topic”, Irene described her TERRINet experience.

She suggests applying to students and researchers who would like to implement their research experiment on a platform not available at their home institution and would like to expand the knowledge from a different research group.

Lear more about Irene’s project. Click on the video below:


  • Choy I, Okrainec A. Fundamentals of Laparoscopic Surgery-FLS. In: The SAGES Manual of Quality, Outcomes and Patient Safety. Boston, MA: Springer US; 2012. p. 461–71.


Coronavirus pandemic

Dear TERRINet applicants and future users,

we sincerely hope that you are staying safe in these challenging times that we are facing at the moment. The current emergency due to the COVID-19 pandemic has deeply changed our lives. The recent and progressive lock-down forced us to reorganize our working routines in smart working modalities and to shut down the TERRINet infrastructures. Therefore, we regret to inform you that Trans-National Access has been suspended until further notice.

However, we would like to reassure you that we will keep evaluating the proposals submitted for Trans-National Access regularly. Indeed, despite access cannot be allowed for a while, the selected users will be queued and access will take over gradually as soon as the situation will allow it. We also inform you that a special application mechanism has been settled in TERRINet to mitigate the delays that will occur due to the COVID-19.

The Open Call no. 4 is currently open: we are accepting proposals under the new mechanism called "first come, first served". Proposals are directly forwarded to the evaluation panel as soon as they are submitted. Proposals submitted closer to the deadline will be dispatched after that. The deadline for the Open Call n.4 will be on May 31, 2020.

Do not hesitate! Get this unique opportunity to be granted with fully-covered access (including travel and accommodation costs) to 15 leading European robotics labs and 100+ best robotic platforms and to get in touch with leading robotic experts. 

For any additional questions, please, contact us at info@terrinet.eu

 Let's take this time of the world calming down to plan our future. 

Stay safe.  

TERRINet team