AAOS Bulletin - February, 2005

Overcoming bias in technology assessment

Point of View

James S. Breivis, MD

The road from research and development to marketing and acceptance of any technology is paved with detours, steep hills and many dead ends. One of the prominent obstacles impeding medical product development is the creeping inclusion of bias. The dictionary defines bias as “a preference or an inclination that inhibits impartial judgment.”

Medical device and pharmaceutical companies are eager to attain a competitive advantage in order to sell their products.

Although most conduct their business in an ethical and otherwise reputable manner, unfortunately there are also those that overlook the lack of supportive scientific evidence in favor of profitability.

Physicians and other caregivers are also subject to bias. Sometimes we are too busy to question marketing claims or have never been taught how to evaluate evidence. The stakes are often high when physicians indulge in educated (and uneducated) guessing about technologies, and the subsequent risks are borne by our patients, our budgets and our reputations.

Similarly, physicians witness bias on the part of our patients: those who approach us with information obtained on the Internet or from direct-to-consumer advertising regarding emerging technologies that are “perfect” for them. Religion and politics also creep into our practices when the technology under consideration involves controversial topics such as stem cell research or pregnancy termination. In all of these scenarios, science and evidence have been minimized and usurped by proponents with bias.

Lessons from Vioxx

Assessing the accumulated evidence on a topic can generate surprising outcomes. Just ask the executives of Merck & Co., manufacturers of Vioxx® (rofecoxib), a drug approved by the Food and Drug Administration in May 1999, and subsequently taken by more than 80 million people.1 According to a recent editorial perspective in the New England Journal of Medicine, Merck’s stock price collapsed on Sept. 30, 2004, the day of its announcement to withdraw Vioxx®, wiping out nearly 25 percent of the company’s market value—$28 billion.2 The company withdrew the drug because of an excess risk of myocardial infarctions and strokes.3 Although the incidence of cardiovascular events was low, there was an increased risk beginning after 18 months of treatment.4 Merck notes that its decision was based on new data from a three-year clinical study.5 However, over the course of five and a half years, many epidemiological studies confirmed and amplified the concern about the risk of myocardial infarctions and serious cardiovascular events associated with rofecoxib.6

One would think that obtaining accurate and sustainable evidence about the safety and efficacy of the drugs we prescribe, the procedures we perform, and the pathways we choose would be natural, easily obtainable and facile. The truth is actually quite to the contrary. The Vioxx product withdrawal serves as a timely example of the importance for surgeons to obtain unbiased scientific evidence. Orthopaedic surgeons are continually exposed to new and emerging technologies, often accompanied by the frustrating pre-treatment authorizations and reimbursement scenarios that seem to stifle innovative use of these technologies. This situation is unlikely to change until our profession relies less on case series and more on rigorous levels of evidence. It is important for practicing surgeons to ask, “How can I choose the technology for my patient that is most clinically appropriate, particularly when the technology is new and the accumulated evidence is scant?”

The final verdict on a new technology—be it good, bad or neutral—is decided by an unbiased assessment of its evidence. There exists a paradigm known as “levels of evidence.” In January 2003, the Journal of Bone and Joint Surgery introduced the practice of declaring the level of evidence for original articles. Many other journals are planning to adopt this practice. The discipline known as Health Technology Assessment (HTA) serves patients, physicians, payors and the public by collecting and systematically analyzing the accumulated scientific evidence on technologies for their risks, benefits and costs. Those performing the assessment should be independent and free from conflicts of interest.

The Academy’s role

Only in recent years has the AAOS appreciated the need for its members to improve their skills in the critical review of evidence and to promote fellows’ exposure to the application of evidence in clinical decision-making. In 2003, an AAOS Task Force was formed to consider the prospect of implementing a program to provide the Academy fellowship with an assessment of the evidence surrounding specific new technologies used in the practice of orthopaedic surgery. The goals of the program were to develop a simple rating scale that conveys the technology’s safety and efficacy, and to provide the existing level of evidence supporting its use in defined indications.

The task force initially solicited proposals from two very reputable HTA firms. The proposals were highly detailed, complicated and, in general, indicated that the comprehensive assessments of five separate technologies per year, updated annually for five years, would range in cost from $215,000 to $325,000.

The task force theorized that CME training from experts in the field of HTA would provide a valuable resource and service to the fellowship by encouraging adoption of evidence analysis techniques. The Academy’s Evidence-Based Practice Committee simultaneously conceived the potential for incorporating the use of evidence into Academy courses and products by declaring the levels of evidence at all Annual Meeting presentations and exhibits, AAOS publications and CME courses.

This brings me to the most important issue encountered by the task force on HTA: “You can lead a horse to water, but you can’t make him drink.”

Orthopaedic surgeons have long had access to evidence-based practice guidelines and performance measures. These products represent the integration of “best research” evidence with clinical expertise and patient values, and are the product of thousands of work hours. In addition, they are provided free to all fellows. In reality, however, only a very small percentage of Academy fellows have incorporated such evidence into their clinical decision-making processes. The reasons for such apathy could alone be the topic of another article, but suffice it to say that the lack of evidence adoption among fellows, along with the costs of implementing an HTA program, have hindered any further forward movement by your Academy.

In its summary report, the task force stated, “The concurrent issues surrounding technology assessment affect the clinical practice of medicine through concerns of efficacy, decision-making, reimbursement and informed consent, and are present and real concerns of the fellowship. We believe that a technology assessment program, in theory, would be of benefit to the fellowship. However, this venture would be highly expensive to implement, and its value to the membership cannot be determined without a needs assessment to the members…”

HTA is an important and integral consideration of health systems in countries with universal or national health care systems. With no central oversight for this in the United States, should an organization such as the AAOS assist the orthopaedic community with such technology assessment information? Responses to the Academy’s recent membership needs assessment survey will provide AAOS leadership with fellowship opinion on the matter.

James S. Breivis, MD is a retired orthopaedic surgeon in San Francisco and an active member of the AAOS Evidence-Based Practice Committee. He can be reached at jbreivis@earthlink.net


  1. Topol E: “Failing the Public Health – Rofecoxib, Merck, and the FDA,” New Engl J Med 351; 17, p 1707.
  2. Oberholzer-Gee F and Inamdar N: “Merck’s Recall of Rofecoxib – A Strategic Perspective,” New Engl J Med 351;21, p 2147.
  3. Op cit. Topol, p 1707
  4. “Important Information for Patients Taking Vioxx® (rofecoxib).” Memo from Raymond V. Gilmartin, Chairman, President & CEO of Merck, dated Sept. 30, 2004.
  5. Ibid.
  6. Op cit. Topol, p 1708

Close Archives | Previous Page