Automated Methods of Technical Skill Assessment in Surgery: A Systematic Review.

The goal of the current study is to systematically review the literature addressing the use of automated methods to evaluate technical skills in surgery. The classic apprenticeship model of surgical training includes subjective assessments of technical skill. However, automated methods to evaluate surgical technical skill have been recently studied. These automated methods are a more objective, versatile, and analytical way to evaluate a surgical trainee's technical skill. A literature search of the Ovid Medline, Web of Science, and EMBASE Classic databases was performed. Articles evaluating automated methods for surgical technical skill assessment were abstracted. The quality of all included studies was assessed using the Medical Education Research Study Quality Instrument. A total of 1715 articles were identified, 76 of which were selected for final analysis. An automated methods pathway was defined that included kinetics and computer vision data extraction methods. Automated methods included tool motion tracking, hand motion tracking, eye motion tracking, and muscle contraction analysis. Finally, machine learning, deep learning, and performance classification were used to analyse these methods. These methods of surgical skill assessment were used in the operating room and simulated environments. The average Medical Education Research Study Quality Instrument score across all studies was 10.86 (maximum score of 18). Automated methods for technical skill assessment is a growing field in surgical education. We found quality studies evaluating these techniques across many environments and surgeries. More research must be done to ensure these techniques are further verified and implemented in surgical curricula.

Levin M ，McKechnie T ，Khalid S ，Grantcharov TP ，Goldenberg M ... -  《-》

Basic Surgical Skill Retention: Can Patriot Motion Tracking System Provide an Objective Measurement for it?

Knot tying is a fundamental skill that surgical trainees have to learn early on in their training. The aim of this study was to establish the predictive and concurrent validity of the Patriot as an assessment tool and determine the skill retention in first-year surgical trainees after 5 months of training. First-year surgical trainees were recruited in their first month of the training program. Experts were invited to set the proficiency level. The subjects performed hand knot tying on a bench model. The skill was assessed at baseline in the first month of training and at 5 months. The assessment tools were the Patriot electromagnetic tracking system and Objective Structured Assessment of Technical Skills (OSATS). The trainees' scores were compared to the proficiency score. The data were analyzed using paired t-test and Pearson correlation analysis. A total of 14 first-year trainees participated in this study. The time taken to complete the task and the path length (PL) were significantly shorter (p = 0.007 and p = 0.0085, respectively) at 5 months. OSATS scoring showed a significant improvement (p = 0.0004). There was a significant correlation between PL and OSATS at baseline (r = -0.873) and at Month 5 (r = -0.774). In all, 50% of trainees reached the proficiency PL at baseline and at Month 5. Among them, 3 trainees improved their PL to reach proficiency and the other 3 trainees failed to reach proficiency. The parameters from the Patriot motion tracker demonstrated a significant correlation with the classical observational assessment tool and were capable of highlighting the skill retention in surgical trainees. Therefore, the automated scoring system has a significant role in the surgical training curriculum as an adjunct to the available assessment tool.

Shaharan S ，Nugent E ，Ryan DM ，Traynor O ，Neary P ，Buckley D ... -  《-》

Eye tracking for skills assessment and training: a systematic review.

The development of quantitative objective tools is critical to the assessment of surgeon skill. Eye tracking is a novel tool, which has been proposed may provide suitable metrics for this task. The aim of this study was to review current evidence for the use of eye tracking in training and assessment. A systematic literature review was conducted in line with PRISMA guidelines. A search of EMBASE, OVID MEDLINE, Maternity and Infant Care, PsycINFO, and Transport databases was conducted, till March 2013. Studies describing the use of eye tracking in the execution, training or assessment of a task, or for skill acquisition were included in the review. Initial search results returned 12,051 results. Twenty-four studies were included in the final qualitative synthesis. Sixteen studies were based on eye tracking in assessment and eight studies were on eye tacking in training. These demonstrated feasibility and validity in the use of eye tracking metrics and gaze tracking to differentiate between subjects of varying skill levels. Several training methods using gaze training and pattern recognition were also described. Current literature demonstrates the ability of eye tracking to provide reliable quantitative data as an objective assessment tool, with potential applications to surgical training to improve performance. Eye tracking remains a promising area of research with the possibility of future implementation into surgical skill assessment.

Tien T ，Pucher PH ，Sodergren MH ，Sriskandarajah K ，Yang GZ ，Darzi A ... -  《-》

Artificial Intelligence in Medical Education: Best Practices Using Machine Learning to Assess Surgical Expertise in Virtual Reality Simulation.

Virtual reality simulators track all movements and forces of simulated instruments, generating enormous datasets which can be further analyzed with machine learning algorithms. These advancements may increase the understanding, assessment and training of psychomotor performance. Consequently, the application of machine learning techniques to evaluate performance on virtual reality simulators has led to an increase in the volume and complexity of publications which bridge the fields of computer science, medicine, and education. Although all disciplines stand to gain from research in this field, important differences in reporting exist, limiting interdisciplinary communication and knowledge transfer. Thus, our objective was to develop a checklist to provide a general framework when reporting or analyzing studies involving virtual reality surgical simulation and machine learning algorithms. By including a total score as well as clear subsections of the checklist, authors and reviewers can both easily assess the overall quality and specific deficiencies of a manuscript. The Machine Learning to Assess Surgical Expertise (MLASE) checklist was developed to help computer science, medicine, and education researchers ensure quality when producing and reviewing virtual reality manuscripts involving machine learning to assess surgical expertise. This study was carried out at the McGill Neurosurgical Simulation and Artificial Intelligence Learning Centre. The authors applied the checklist to 12 articles using machine learning to assess surgical expertise in virtual reality simulation, obtained through a systematic literature review. Important differences in reporting were found between medical and computer science journals. The medical journals proved stronger in discussion quality and weaker in areas related to study design. The opposite trends were observed in computer science journals. This checklist will aid in narrowing the knowledge divide between computer science, medicine, and education: helping facilitate the burgeoning field of machine learning assisted surgical education.

Winkler-Schwartz A ，Bissonnette V ，Mirchi N ，Ponnudurai N ，Yilmaz R ，Ledwos N ，Siyar S ，Azarnoush H ，Karlik B ，Del Maestro RF ... -  《-》

Objective assessment of technical skill.

Objective assessment of technical skill is an important component of skills training: trainees require that deficiencies are clearly and objectively identified if a model of deliberate practice with feedback on skill acquisition is to be employed. There are several types of reliable and valid assessments for technical skill currently available.

MacRae HM 《SURGEON-JOURNAL OF THE ROYAL COLLEGES OF SURGEONS OF EDINBURGH AND IRELAND》