Introduction
For medical students navigating the competitive residency application process, resources like the Student Doctor Network (SDN) – a public online forum dedicated to medical education discussions – are frequently consulted. In recent years, specialty-specific forums within SDN have gained traction, allowing residency applicants to anonymously share their application data. This peer-reported information is intended to help prospective applicants gauge their competitiveness and strategize their application process. However, the reliability of this self-reported data as a true reflection of the broader applicant pool remains uncertain. This study aims to assess the accuracy of SDN as an informational resource for emergency medicine (EM) applicants by comparing self-reported data from SDN forums with objective data from the National Resident Matching Program (NRMP).
Methods
This research involved a retrospective analysis of self-reported data from Doctor of Osteopathic Medicine (DO) and Doctor of Medicine (MD) candidates on SDN’s emergency medicine forums. The data spanned three residency application cycles: 2014, 2016, and 2018. These specific years were chosen to align with publicly available NRMP charting outcomes data for comparative analysis. We extracted self-reported USMLE (United States Medical Licensing Examination) Step 1 and Step 2 scores, contiguous residency program ranks, and Alpha Omega Alpha (AOA) Honor Medical Society membership status from the SDN forums for each cycle. This self-reported SDN data was then directly compared against the corresponding NRMP charting outcomes reports for each of the analyzed years to determine any discrepancies.
Results
Across the three application cycles, a total of 360 Emergency Medicine Student Doctor applicants contributed self-reported data on the SDN forums. Notably, a significant majority (79%) of these postings occurred during the 2018 application cycle, coinciding with the increased use of Google Docs spreadsheets for data collection within the SDN forums. In the initial two years examined (2014 and 2016), the mean USMLE Step scores reported on SDN were generally consistent with the aggregate data presented in the NRMP charting outcomes. However, in the most recent cycle studied (2018), a noticeable trend emerged. Applicants self-reporting on SDN demonstrated statistically higher mean USMLE Step 1 scores (234, 95% Confidence Interval [CI], 233-236) and Step 2 scores (250, 95% CI, 248-251) compared to the objective NRMP data (Step 1: 231; Step 2: 241). Conversely, self-reported contiguous residency program ranks on SDN remained comparable to NRMP data across all analyzed years. Similarly, the proportion of applicants indicating Alpha Omega Alpha Honor Medical Society membership on SDN was consistent with NRMP figures only for the 2018 application cycle, suggesting potential variations in the accuracy of reporting for this metric across different years.
Conclusion
Our analysis indicates that self-reporting on the Student Doctor Network for emergency medicine residency applications exhibited a slight upward bias in USMLE Step scores, particularly in the most recent application cycle studied when compared to objective NRMP data. While self-reporting activity on SDN has increased, especially with the adoption of more organized data collection methods, the impact of this increase on the overall accuracy and representativeness of the information available to emergency medicine student doctor applicants remains unclear. Given the inherent limitations of self-reported data and the observed bias, it is crucial for emergency medicine residency applicants to interpret and utilize information gleaned from SDN forums with caution and consider corroborating data from multiple sources to form a comprehensive understanding of the residency application landscape.
