The recent ruminations of Drs. Katsufrakis and Chaudhry in the form of an invited commentary in Academic Medicine, entitled “Improving Residency Selection Requires Close Study and Better Understanding of Stakeholder Needs,” has garnered a significant amount of attention on Twitter. Drs. Katsufrakis and Chaudhry’s commentary was in response to a well-written and well-reasoned article by a group of medical students published in the same journal recommending the USMLE Step 1 transition from a numeric score to pass/fail.
In their objective opinion,
“If students reduce time and effort devoted to preparing for Step 1, they may indeed devote attention to other activities that will prepare them to be good physicians. This would arguably be an ideal outcome of such a change. However, if students were to devote more time to activities that make them less prepared to provide quality care, such as binge-watching the most recent Netflix series or compulsively updating their Instagram account, this could negatively impact residency performance and ultimately patient safety. We know that assessment drives learning, so another concern resulting from a shift to pass/fail scoring may be a less knowledgeable physician population.”
Oh dear. I’ll address my Netflix obsessions at the end, but first…
Understandably, Drs. Katsufrakis and Chaudhry, the CEOs of the National Board of Medical Examiners and Federation of State Medical Boards, respectively, are concerned that such a transition in Step 1 scoring could result in significant patient harm secondary to, in their words, a “less knowledgeable physician population.”
Political, economic, racial, gender, and sexual orientation disparities and widespread inequity of health care access could never hold a candle to the potential apocalypse if medical students do not receive a three-digit numeric score on (mostly) highly irrelevant knowledge to clinical practice. Who would memorize the common biochemical pathways in Figure 1 below, learned in the preclinical years of medical school and tested on Step 1, if it weren’t for the numerical score? No knowledgeable physician should be without this information on the wards or in the emergency department.
Figure 1. Guide to life-saving fluid management in the inpatient setting. Image source: ExPASy’s ENZYME Database.
Of course, there is no high-quality evidence to support the aforementioned concerns of these two esteemed physicians. And the evidence that is cited to challenge the critiques of Step 1 are either a misunderstanding of study design and generalizability of results or intentional obscurantism.
For example, they state that,
“Research demonstrates some differences in USMLE scores attributable to race and ethnicity, with self-identified Black, Asian, and Hispanic examinees showing score differences when compared with self-identified White examinees (Rubright et al 2018). Some cite this as evidence in support of eliminating Step 1 scores, at least for residency selection. However, the majority of these observed differences disappear when controlling for undergraduate grade point average and Medical College Admission Test scores.”
This was an incorrect interpretation of the results of Rubright et al.’s 2018 study. Rubright et al.’s results suggest that when MCAT and GPA are added to demographic factors in their hierarchical linear model, the effect size of self-identified race and ethnicity as predictor variables of Step 1 numeric score are attenuated, but they still remain significant. Interestingly, they further ignored multicollinearity of the predictor variables since “the variables likely to be correlated are used as control variables and not variables of interest,” with no sensitivity analysis provided to see how multicollinearity affects the outcome variable (predicted Step 1 score) and influence of the predictor variables that are most likely highly correlated (such as MCAT and GPA…). Rubright et al. further excluded all students taking USMLE exams who identified as more than one race or ethnicity, though we are not given data on how many students were excluded in this analysis. Sadly, predictor variables like socioeconomic status, first generation college student (yes/no), and a host of other significant predictor variables were not included in this analysis. Oh, and Rubright and the other two co-authors conveniently work for the NBME. The de-identified, ‘exempt’ data is not provided by the authors for reproducibility analysis, which I would love to do, to be honest.
But that is not why I am writing this piece. Many on Twitter much smarter and more eloquent than I — such as Dr. Shannon McNamara and Dr. Bryan Carmody — have already poignantly critiqued many of the statements made by these two figureheads of profit-oriented companies. What follows, however, are some thoughts by an actual stakeholder, a medical student.
Let’s start with the obvious. The burden of proof that USMLE Step 1 numeric scoring is important for patient care — i.e., meaningful, hard outcomes and not surrogate outcomes such as in-training exam scores — rests on the NBME and FSMB. I hope both of these organizations fund a high-quality study that is designed and evaluated by an independent third-party to assess their alternative hypothesis.
What about Netflix binges and obsessions with Instagram is so concerning to Dr. Katsufrakis and Dr. Chaudhry? According to the NRMP’s own “Charting Outcomes in the Match,” not only do current medical students continue to improve in their Step scores, they do so while showing incredible distinction outside the wards with numerous research publications and leadership experiences. The evidence seems to suggest that even if Step 1 became pass/fail, medical students of this generation will only become more productive as it relates to medicine. Such concern only serves as legerdemain.
Before I conclude, I have one more point of contention. Academic Medicine appears content with Drs. Katsufrakis and Chaudhry’s “Other Disclosure” that they are “the CEOs of the two organizations that sponsor the USMLE program.” Of all the conflicts of interest we read about in the medical literature, one would think this has to be one of the most relevant financial conflicts of interest, given that it affects up to 20,000 medical students a year. As Dr. Vinay Prasad of Oregon Health Sciences University wrote in Hastings Center Report, “Closed Financial Loops: When They Happen in Government, They’re Called Corruption; in Medicine, They’re Just a Footnote.” While it is important to hear the voices of all stakeholders in this Step 1 debate, an overview of the pros and cons of transitioning Step 1 to pass/fail requires a more objective rebuttal from individuals who do not personally benefit from years of students taking licensing exams. Why not have experts who have devoted a career to researching medical licensing exams, medical student burnout and risk factors for depression and anxiety in training, or honestly anything related to this topic that does not involve pushing forward conflicted commentaries?
Alright, let’s get back to Netflix. Who could have predicted that Netflix and Instagram would lead to the demise of patient care in the United States?
Honestly, I would have bet on Snapchat or Facebook. But now I digressed.
While I have more to say on this topic, I am currently seven hours into binge-watching Property Brothers on HGTV, my preferred TV network.
Medical Student Guest Writer
Cleveland Clinic Lerner College of Medicine
Joshua D. Niforatos, MTS is in his last year of medical school in Cleveland, Ohio. He aspires to be an emergency medicine physician actively involved in clinical research and all things social emergency medicine and advocacy. Joshua also watches a lot of HGTV.
