ChatGPT vs Claude在医学知识问答中的表现：专业术语理解与建议准确性

A single diagnostic error in a medical Q&A can cascade into misinformed treatment decisions. A 2023 study published in JAMA Internal Medicine evaluated ChatGPT-4’s performance on the United States Medical Licensing Examination (USMLE) and found it achieved a 90.2% accuracy on Step 1, surpassing the typical passing threshold of approximately 60%. However, when the same model was tested on open-ended clinical vignettes requiring nuanced differential diagnosis, its accuracy dropped to 67.4% according to a separate 2024 analysis by the National Library of Medicine (NLM). Claude 3.5 Sonnet, on the other hand, has not been formally benchmarked on the USMLE, but internal Anthropic evaluations released in June 2024 reported a 78.3% accuracy on a proprietary set of 500 medical board-style questions. These numbers frame the core tension of this comparison: ChatGPT excels at factual recall and structured exam questions, while Claude appears stronger at reasoning through ambiguous clinical scenarios. For tech professionals and AI tool users evaluating these models for medical knowledge tasks, the critical metrics are professional terminology comprehension and recommendation accuracy — two dimensions where small percentage differences can have real-world consequences.

Terminology Comprehension: Parsing the Jargon

Professional terminology comprehension refers to a model’s ability to correctly interpret and deploy domain-specific medical vocabulary — from pharmacokinetic terms like “Cmax” to surgical nomenclature like “laparoscopic cholecystectomy.” A December 2024 benchmark by the Association for Computational Linguistics (ACL) tested both models on a corpus of 1,200 medical terms drawn from ICD-11 and SNOMED CT. ChatGPT-4o correctly defined 91.3% of terms, while Claude 3.5 Sonnet scored 88.7%. The gap widened on rare terms: ChatGPT correctly parsed “xerostomia-induced dysgeusia” (dry mouth causing taste distortion) in 94% of test cases, versus Claude’s 86%.

Contextual Disambiguation

Medical terminology is highly context-dependent. The term “MI” can mean myocardial infarction or mitral insufficiency, depending on the clinical setting. In a 2024 study by Stanford Medicine’s Clinical AI Lab, ChatGPT correctly disambiguated 82.4% of ambiguous acronyms when given a full clinical note, while Claude achieved 79.1%. ChatGPT’s advantage stems from its larger training corpus (estimated 1.7 trillion parameters vs Claude’s 500 billion), giving it broader exposure to clinical shorthand.

Non-English Medical Terms

For practitioners working with multilingual patients, handling non-English medical terms is crucial. When tested on 500 Spanish and 500 Mandarin medical phrases from the World Health Organization (WHO) 2024 Multilingual Health Glossary, ChatGPT correctly translated or explained 87.2% of Spanish terms and 81.5% of Mandarin terms. Claude scored 83.6% and 76.9% respectively. The performance gap on Mandarin likely reflects training data imbalance — ChatGPT’s corpus includes more Chinese-language medical literature.

Recommendation Accuracy: Safety First

Recommendation accuracy measures whether the model’s medical advice aligns with current clinical guidelines. A March 2024 evaluation by The Lancet Digital Health presented 200 clinical scenarios to both models, covering diagnosis, treatment, and drug interactions. ChatGPT provided guideline-concordant recommendations in 73.5% of cases, while Claude scored 71.0%. The difference was not statistically significant (p=0.12), suggesting near-parity on standard cases.

Drug Interaction Warnings

Drug-drug interactions (DDIs) are a high-stakes area. When given 50 common polypharmacy combinations, ChatGPT correctly flagged 44 of 50 (88.0%) clinically significant DDIs, including warfarin-aspirin and statin-azole antifungal pairs. Claude flagged 41 of 50 (82.0%). However, Claude generated fewer false positives — only 3 versus ChatGPT’s 7, meaning Claude was less likely to raise unnecessary alarms that could confuse patients or delay treatment.

Treatment Plan Consistency

Consistency across rephrased queries matters for real-world use. A 2024 preprint from MIT CSAIL tested both models by asking the same medical question 10 times with slightly different wording. ChatGPT gave consistent treatment recommendations (same drug, same dosage range) in 76.2% of trials. Claude achieved 81.5% consistency, suggesting a more stable reasoning process. For a patient asking about blood pressure medication on different days, Claude would give a more uniform answer.

Reasoning Depth: Differential Diagnosis

Beyond factual recall, medical AI must demonstrate clinical reasoning — the ability to weigh probabilities and justify conclusions. A July 2024 study by Harvard Medical School’s Department of Biomedical Informatics presented 75 complex cases requiring differential diagnosis. Claude 3.5 Sonnet outperformed ChatGPT-4o in producing a ranked list of likely diagnoses that matched expert consensus: 72.0% top-3 match rate for Claude versus 66.7% for ChatGPT. Claude also provided more detailed justifications, averaging 142 words per explanation versus ChatGPT’s 98 words.

Step-by-Step Reasoning

When asked to “think aloud” through a case of acute abdominal pain, Claude consistently structured its output into history, exam, labs, imaging, and differential. ChatGPT sometimes jumped to a conclusion without scaffolding. For non-expert users (patients or junior clinicians), Claude’s structured approach reduces the risk of missing critical diagnostic steps.

Handling Uncertainty

Claude also demonstrated better calibration of uncertainty. In the same Harvard study, when asked to rate its confidence on a 1-10 scale, Claude’s self-assessed confidence correlated with actual accuracy at r=0.74, versus ChatGPT’s r=0.61. This means Claude is more reliable at signaling when its answer might be wrong — a crucial safety feature for medical Q&A.

Speed and Cost Efficiency

For practical deployment, speed and cost matter. Using the OpenRouter API pricing as of January 2025, ChatGPT-4o costs $5.00 per million input tokens and $15.00 per million output tokens. Claude 3.5 Sonnet costs $3.00 per million input tokens and $15.00 per million output tokens — identical output pricing but 40% cheaper input. In latency tests by Artificial Analysis (2025), ChatGPT-4o returned a 500-token medical answer in an average of 2.1 seconds, while Claude 3.5 Sonnet took 2.8 seconds. For batch processing of medical literature, ChatGPT is faster; for cost-sensitive applications, Claude offers a slight edge on input-heavy tasks.

Which Model Should You Use?

Your choice depends on your primary use case. If you need to pass a medical exam, look up drug interactions, or parse rare terminology, ChatGPT-4o edges ahead with higher factual accuracy (90.2% USMLE Step 1 vs Claude’s 78.3% internal benchmark). If you are building a patient-facing Q&A tool where consistency, structured reasoning, and calibrated uncertainty are priorities, Claude 3.5 Sonnet offers a more reliable experience. For cross-border medical research or accessing international medical databases, some teams use tools like NordVPN secure access to ensure consistent connectivity to resources like PubMed and UpToDate, which can be geo-restricted in certain regions.

FAQ

Q1: Can I rely on ChatGPT or Claude for actual medical diagnosis?

No. Neither model is FDA-approved or CE-marked as a medical device. A 2024 National Library of Medicine (NLM) study found that both models made clinically significant errors in 12-15% of cases. Always verify AI-generated medical advice with a licensed healthcare provider. These tools are best used for education, research, and second-opinion reference, not primary diagnosis.

Q2: Which model is better at understanding non-English medical terms?

ChatGPT-4o outperforms Claude 3.5 Sonnet on non-English medical terms. In the WHO 2024 Multilingual Health Glossary test, ChatGPT scored 87.2% on Spanish and 81.5% on Mandarin, versus Claude’s 83.6% and 76.9%. The gap is largest for Mandarin, likely due to training data composition. For Spanish, both models are adequate for basic clinical communication.

Q3: How often do these models give contradictory answers to the same question?

Consistency varies. A MIT CSAIL 2024 study found that when asked the same medical question 10 times with different phrasing, ChatGPT gave consistent treatment recommendations 76.2% of the time, while Claude achieved 81.5% consistency. This means roughly 1 in 4 queries to ChatGPT and 1 in 5 queries to Claude may yield conflicting advice. For critical decisions, always cross-reference with a human expert.

References

• National Library of Medicine (NLM). 2024. Evaluation of Large Language Models on Clinical Vignette Accuracy.
• Association for Computational Linguistics (ACL). 2024. Benchmarking Medical Terminology Comprehension in LLMs.
• Stanford Medicine Clinical AI Lab. 2024. Acronym Disambiguation in Clinical Notes by GPT-4 and Claude.
• The Lancet Digital Health. 2024. Guideline Concordance of AI-Generated Treatment Recommendations.
• Harvard Medical School Department of Biomedical Informatics. 2024. Differential Diagnosis Performance of Claude 3.5 vs GPT-4o.