Dietary Supplement Opinion Lab.

Large language models are increasingly used as informal sources of nutrition and health information, yet their systematic tendencies (over-agreement, framing sensitivity, audience-dependent bias) are invisible to end users. For my master's project at Northeastern Illinois University, I measured these tendencies across ten frontier LLMs and 400 evidence-grounded statements. The follow-on question: how do you make those measurements inspectable by anyone, and keep them current as models change?

The study curated 400 Likert-scale statements about dietary supplements, each paired with an evidence source, primarily NIH Office of Dietary Supplements fact sheets, with a small number from NCCIH, PubMed Central, and other public health sources. Statements were balanced across audience (consumer vs. health professional) and framing (favoring vs. questioning supplements), 100 per group. A knowledgeable consumer and a registered dietitian provided reference ratings; each LLM rated every statement zero-shot with a shared prompt template, producing 4,000 ratings.

The platform turns that dataset into a standing instrument: visitors read a statement, check its source, rate their own agreement, and only then see how each model rated it. The statement bank and every evaluation run are versioned public data, so the record extends over time instead of being rewritten.

Every number in the thesis was recomputed independently from the raw collection. The audit confirmed the dataset's integrity (400 statements, four groups of exactly 100, 4,000 complete ratings) and all headline metrics, and it caught a real defect: an unanchored regular expression in the analysis notebook had silently mixed statements across groups in one chart, and a reported bias-gap value ("1.32 vs. 1.02") did not survive recomputation from clean groups (correct: 1.28 vs. 1.11). All published subgroup values were corrected, the paper updated with tracked changes, and the verification report ships in the repository.

Mean Absolute Difference (MAD) against the human reference ratings, and Pearson correlation of rating patterns, across all 400 statements. These metrics measure agreement with human raters, not factual or medical correctness.

Two findings stand out. Numerical closeness and directional alignment are different properties: GPT-3.5 Turbo posts a moderate MAD (0.82) but the weakest correlation (0.28), meaning its scores look close while its reasoning pattern diverges. And most models agree more readily with positively framed consumer statements than with skeptical ones, an optimism bias with real implications for health-adjacent use of LLMs.

App

React, TypeScript, Vite, Tailwind

Data

Versioned JSON statement bank and immutable evaluation runs (CC BY 4.0)

Analysis

Python scripts that regenerate every published metric

Hosting

Static deployment from git

Source & data↗

The dataset is archived for citation (DOI: https://doi.org/10.5281/zenodo.23357963) and the app is live at opinionlab.hamdam.me. Research and education only. This project does not provide medical or nutritional advice.