Development and Reliability of a Competency Assessment Tool for Magnetic Sphincter Augmentation of the Esophagus (MSA CAT) as Part of a Video-based Surgical Quality Assurance Program for the GOLF Trial

Objective: The development and reliability testing of a video-based competency assessment tool for magnetic sphincter augmentation of the esophagus (MSA CAT). Summary background data: Surgical quality assurance (SQA) is the assessment of intraoperative performance by measuring adherence to standardized steps, occurrence of technical errors, and overall technical proficiency. SQA is part of the GOLF trial (ISRCTN: 1353317), an international, multicenter randomized controlled trial (RCT) comparing 2 antireflux procedures, magnetic sphincter augmentation (MSA) and fundoplication. GOLF is the first RCT in which a video-based SQA program was developed with a plan for validation and implementation. Methods: Ten international experts in antireflux surgery participated in a 3-round virtual Delphi consensus process to develop the MSA CAT. To establish interassessor reliability, 6 surgeons reviewed 2 MSA videos. After 3 months, they reviewed the same MSA videos to test intra-assessor reliability using Cohen's kappa coefficient. Results: Operative steps and key errors were drafted based on the literature. During the 3-round virtual Delphi Consensus amendments to each version were undertaken. Finally, an operative manual and CAT were developed with mandatory, optional, and prohibited steps and quality scores ranging from 1 to 4. The MSA CAT demonstrated good inter- and intra-assessor reliability, particularly when categorizing pairs of scores into "concerning" versus "acceptable." For interassessor reliability, the mean Cohen's kappa was 0.785 (0.582-0.892) and 0.827 (0.591-0.912) for each video. For intra-assessor reliability, it was 0.745 (0.717-0.773) and 0.740 (0.510-1.00) for each video. Discussion: This work describes the development and reliability of the novel MSA CAT. This will be further validated by prospective and retrospective video analysis.