Overview
A systematic evaluation of three large language models — Gemini 1.5 Pro, Llama 3.1 (405B), and GPT-4o — on fundamental physics questions across five dimensions: factual accuracy, explanation clarity, mathematical reasoning, misconception handling, and pedagogical value. Recognized by the Turkish Physical Society.
Evaluation Framework
Responses are scored on a 5-dimension rubric:
- Factual Accuracy (1–5): Correctness of stated laws, constants, units, and physical principles.
- Explanation Clarity (1–5): Logical structure and accessibility of the reasoning chain.
- Mathematical Reasoning (1–5): Proper equation use, dimensional analysis, and numerical precision.
- Misconception Handling (1–5): Detection and correction of common student errors (e.g., weight vs. mass).
- Pedagogical Value (1–5): Use of analogies, scaffolding, and error transparency.
Each prompt is rated independently by three domain experts; inter-rater reliability is monitored via Fleiss’ κ.
Question Set
Over 150 problems stratified across five subdomains: Mechanics (40+), Thermodynamics (30+), Electromagnetism (35+), Quantum Mechanics (25+), and Optics (20+). Problems span high-school to advanced-undergraduate level, skewed toward conceptual questions that test understanding over rote recall.
Methodology
Each model receives every question under three conditions: zero-shot, few-shot (k=3), and chain-of-thought. All conditions are repeated at τ = 0.0, 0.4, 0.8 with three runs per combination, yielding over 4,000 scored responses. Statistical significance is assessed via paired bootstrap (10,000 resamples; Bonferroni-corrected). All pairwise model differences are significant at p < 0.01.
Results
GPT-4o leads across all domains on factual accuracy: Mechanics 4.4, Thermodynamics 4.3, Electromagnetism 4.2, Quantum 3.9, Optics 4.3 (1–5 scale, zero-shot, τ=0.0). It demonstrates superior mathematical rigour and misconception handling, though it occasionally over-explains.
Gemini 1.5 Pro ranks second (Mechanics 4.2, Electromagnetism 3.9, Quantum 3.5). It excels in explanation clarity and structured pedagogical responses but occasionally produces confidently incorrect statements on edge-case quantum phenomena.
Llama 3.1 (405B) trails on raw accuracy (Mechanics 3.8, Electromagnetism 3.5, Quantum 3.3) but shows the strongest improvement under few-shot prompting (+12% factual accuracy gain vs. +7% GPT-4o, +6% Gemini), indicating comparable latent capability with greater prompt sensitivity.
Chain-of-thought prompting yields the highest composite scores (mean +0.6 over baseline), followed by few-shot (+0.3). Temperature τ = 0.0 is most reliable for evaluation; τ = 0.4 preserves accuracy while increasing response diversity. Within-model variance peaks for quantum mechanics (σ² ≈ 0.45) and is lowest for mechanics (σ² ≈ 0.18).
Turkish Physical Society Presentation
Findings were presented at the Turkish Physical Society (TFD) Annual Meeting (September 2024) under the title “Evaluating Large Language Models as Physics Tutors”, including a live model-comparison demonstration. The recognition highlights the study’s potential impact on physics education and AI literacy.
Ongoing Work
The dataset is expanding with 50+ problems in astrophysics and statistical mechanics. A finer-grained rubric is in development. Evaluation scripts and the curated question set (Turkish and English) are planned for public release.
References
- TFD 2024 Conference Proceedings (forthcoming)
- Chiang, W. et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs. NeurIPS 2024.
- Liang, P. et al. (2023). Holistic Evaluation of Language Models (HELM). TMLR.