June 1999 Bulletin

OSATS valid for basic procedures


Carol R. Hutchinson, MD, Med., right, watches resident perform knee procedure at the Objective Structured Assessment of Technical Skills pilot study in the Orthopaedic Learning Center.

Surgical skills courses are undeniably popular, but how effective are they? That's the $122,000 question.

The Objective Structured Assessment of Technical Skills (OSATS) research study, which is being funded by $70,000 from the Academy's Council on Education and a $52,000 grant from the Orthopaedic Research and Education Foundation, is designed to measure the effectiveness of surgical education activities. The study's first phase, which took place in January in the Orthopaedic Learning Center, was intended to measure OSATS' validity and reliability in measuring orthopaedic skills; results from that study arrived in mid-April. The second phase, to take place some time in the year 2000, will determine whether the OSATS form of evaluation is valid and practical for assessment of OLC courses

On the day of the pilot study, which was organized and monitored by the Task Force on Educational Effectiveness, 24 orthopaedic residents from Chicago, St. Louis, Boston and Toronto area hospitals underwent an intensive day of training and testing in the OCL.

"It was a long, tedious process," says Shawn Hennigan, a fifth year orthopaedic resident at Rush-Presbyterian-St. Luke's Medical Center in Chicago. "But I think the testing procedure was very worthwhile. This type of hands-on education is so important."

Each resident was asked to perform a pre-test, which included several of the steps involved in a total knee arthroplasty (TKA) procedure, and in an anterior cruciate ligament (ACL) reconstruction, within a given time frame. The steps were performed using cadaveric knee specimens and simulated models that were prepared in advance to make the testing uniform for each participant.

Residents then were divided into two groups to receive instruction on either the TKA or ACL procedure. Following the instruction period, the entire group reconvened in the lab and was given a post-test on both procedures. The group that received ACL instruction served as the control group for the TKA training, and vice versa.

Twenty-four orthopaedic surgeons were recruited to complete the examining functions of the research. The evaluation system components included: the OSATS examiner checklist, oral questions, global ratings and end-product analysis.

So what were the results? Upon initial examination of the data, the following conclusions were drawn:

Study variables:

The next step for the task force will be to further analyze the data, followed by publication of the "phase one" data.

In phase two of the study, the task force will design a research effort to be conducted in a CME course. That will probably occur sometime in the year 2000.

The OSATS testing process, which is intended to provide controlled, scientific investigation that documents the acquisition and retention of surgical skills, was designed by Carol R. Hutchison, MD, MEd, orthopaedic surgeon and director of the University of Toronto Surgical Skills Center at the Mount Sinai Hospital. She is also the principal investigator for the Academy's study.


Home Previous Page