Emergency Medicine · Medical Literature
Identification of diagnostic discrepancies as a quality assurance measure in emergency medicine - a validation study.
Abstract
Diagnostic errors are a major care health concern but remain difficult to study because their identification often requires resource-intensive chart reviews. We aimed to validate a previously proposed automated method for detecting discrepancies between an initial and a later, more definitive diagnosis as a screening tool for potential diagnostic errors in a large, prospective cohort of emergency department (ED) patients. This secondary analysis included 1,204 patients enrolled in the DDxBRO randomized trial, which evaluated the effect of a diagnostic decision support tool on diagnostic quality in four Swiss emergency departments. For each patient, the ED diagnosis was extracted from the ED discharge letter, and the follow-up diagnosis at 14 days was obtained from hospital discharge letters, or general practitioner notes. All diagnoses were coded using ICD-10 and manually classified for discrepancies by two blinded ED physicians according to a predefined scheme. The automated method calculated the "similarity" between ICD-10 codes for ED and follow-up diagnoses. Discriminative performance of this method to distinguish between cases with and without diagnostic error was evaluated using receiver operating characteristic (ROC) curves, and sensitivity, specificity, and predictive values were assessed across multiple cutoffs. The automated method showed high and consistent discriminative performance across all algorithms tested, with areas under the ROC curve (AUCs) ranging from 0.94 to 0.95. Using the most sensitive cutoff in the simplest algorithm, all true discrepancies were detected, but 162 cases (15%) were incorrectly flagged as discrepant. The automated method demonstrated high accuracy and shows promise as a practical screening tool to prioritize cases for resource-intensive chart review. NCT05346523.
Clinical relevance
Can an automated ICD-10 code similarity algorithm accurately screen for diagnostic discrepancies between ED discharge diagnoses and 14-day follow-up diagnoses, compared with expert manual physician classification, in a large prospective ED cohort? __SNAPSHOT__ {"studyDesign":"Diagnostic test accuracy validation study; pre-specified secondary analysis of a prospective, multicenter randomized controlled trial (DDxBRO trial)","setting":"Four Swiss emergency departments: one tertiary care centre (Inselspital Bern), one large urban hospital (Bürgerspital Solothurn), and two community hospitals (Spital Tiefenau and Spital Münsigen)","sampleSize":"n=1,204 (591 female, 613 male); all enrolled in the DDxBRO RCT intention-to-treat population","population":"Adults (≥16 years) presenting to the ED with abdominal pain, fever of unknown origin, syncope, or other non-specific symptoms (e.g., generalized weakness, generally degraded condition); all provided written informed consent","intervention":"Automated ICD-10 hierarchical code similarity algorithm calculating 'distance' between ED discharge diagnosis and 14-day follow-up diagnosis, using four distinct algorithmic variants (including a step-count algorithm), implemented via freely available software (https://right-icd.ch/)","comparator":"Manual classification of diagnostic discrepancies by two blinded, trained ED physicians using a predefined classification scheme, serving as the reference standard (gold standard)","primaryOutcome":"Discriminative performance of the automated method to distinguish cases with versus without diagnostic discrepancy, assessed by area under the receiver operating characteristic (ROC) curve (AUC), sensitivity, specificity, and predictive values across multiple cutoffs","followUp":"14 days (follow-up diagnoses obtained from hospital discharge letters or general practitioner notes)","fundingSource":"Government/non-profit: Swiss National Science Foundation (Project number 187284) and intramural funding from Bern University Hospital; no industry funding declared"}
Key findings
The automated ICD-10 similarity method demonstrated consistently high discriminative accuracy across all four algorithmic variants tested, with AUCs ranging from 0.94 to 0.95. For the simplest step-count algorithm, at the most sensitive cutoff (≥2 ICD-10 hierarchical steps defined as discrepant), sensitivity was 100% (all manually classified discrepant cases detected), but the false-positive rate was 15% (162 non-discrepant cases incorrectly flagged). At the cutoff that optimized both sensitivity and specificity (≥5 hierarchical steps, per Youden's Index), both sensitivity and specificity were approximately 90%, representing a clinically actionable trade-off. Inter-rater agreement for the manual reference standard was moderate (Cohen's κ=0.4 assessed on 100 cases), notably higher than prior published literature in which κ ranged from 0.00 to 0.04 for similar tasks. NNT/NNH are not applicable to this diagnostic accuracy study. No patient-level adverse event data were reported, as this was a methodological validation study.
Practice implications
This study established that an automated, ICD-10 hierarchy-based algorithm can serve as a high-accuracy screening tool to flag potential diagnostic discrepancies in emergency medicine, achieving AUC ≈ 0.95 against expert physician manual review. The tool is explicitly not intended to replace expert chart review or to adjudicate patient harm directly; its intended role is to triage which cases warrant targeted, resource-intensive manual review, thereby improving the scalability of quality assurance programs. The study did NOT establish that the method can directly identify diagnostic errors causing harm, classify severity of discrepancy, or replace a clinician judgment process. Findings apply to adult ED patients presenting with diagnostically challenging, non-specific symptoms (abdominal pain, fever, syncope, generalized weakness) in a Swiss, predominantly academic hospital system; generalizability to EDs with different case-mix, documentation practices, or ICD coding conventions (e.g., ICD-10-CM in the U.S.) has not been established. The moderate inter-rater agreement of the manual reference standard (κ=0.4) is itself a limitation, meaning the 'gold standard' carries meaningful uncertainty. Because only the published abstract and open-access full text were used, all reported values are directly drawn from the source document. __LIMITATIONS__ 1. Secondary analysis of a trial population enriched for diagnostically challenging, non-specific presentations (abdominal pain, fever, syncope, generalized weakness); results may not generalize to unselected ED populations or to presentations with high diagnostic certainty. 2. The 14-day follow-up window for defining discrepancy was acknowledged by the authors as arbitrary; shorter windows may reduce sensitivity, longer windows may reduce specificity. 3. The manual reference standard itself carries substantial uncertainty: inter-rater agreement was only moderate (Cohen's κ=0.4 on 100 cases), and remaining cases were classified individually without duplicate review. 4. The study was conducted entirely in Switzerland; ICD-10 coding practices differ internationally (e.g., ICD-10-CM vs. ICD-10-WHO), potentially limiting direct transferability. 5. The automated method detects ICD-code discrepancy as a proxy for diagnostic error, but diagnosis changes may reflect natural disease evolution, deliberate diagnostic refinement, or documentation variation rather than true error. 6. Harm classification, downstream patient outcomes, and clinical significance of flagged discrepancies were not assessed. 7. No confidence intervals are reported for the sensitivity and specificity point estimates at specific cutoffs. 8. Blinding of the raters to the automated output during manual classification was maintained; however, both raters were from the same institution and trained together, which may limit independence. 9. Single-country, predominantly academic setting limits generalizability to community or resource-limited ED environments. __BOTTOM_LINE_CATEGORY__ potentially_practice_informing __BOTTOM_LINE__ An automated ICD-10 code similarity algorithm achieved AUC ≈ 0.95 for detecting diagnostic discrepancies compared with expert physician review across 1,204 Swiss ED patients; at maximum sensitivity, it correctly identified all discrepant cases with a 15% false-positive rate, while a balanced cutoff achieved approximately 90% sensitivity and specificity. This tool shows strong potential as a scalable, low-resource first-pass screening mechanism to prioritize cases for targeted manual chart review in ED quality assurance programs, but it does not replace expert adjudication and has not been validated in non-Swiss or unselected ED populations. Adoption as a quality improvement screening instrument is reasonable to consider, with the understanding that flagged cases still require clinical expert review before any error classification or attribution of patient harm.