hallucination-leaderboard
The Hallucination Leaderboard is an open-source benchmarking tool that ranks LLM performance on factual accuracy when summarizing documents, using Vectara's HHEM-2.3 evaluation model. It provides comparative metrics across 100+ models and exposes which LLMs tend to introduce false information during summarization tasks.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Repository | vectara/hallucination-leaderboard |
| Owner | vectara |
| Primary language | Python |
| License | Apache-2.0 — OSI-approved |
| Stars | 3.3k |
| Forks | 106 |
| Open issues | 19 |
| Latest release | Unknown |
| Last updated | 2026-05-11 |
| Source | https://github.com/vectara/hallucination-leaderboard |
What hallucination-leaderboard is
A Python-based leaderboard repository that applies HHEM-2.3 (a commercial hallucination detection model) to evaluate LLM summarization outputs against a curated 7700+ document dataset spanning news, tech, science, and other domains. Rankings expose hallucination rates, factual consistency scores, answer rates, and summary length patterns across model families.
Get the hallucination-leaderboard source
Clone the repository and explore it locally.
git clone https://github.com/vectara/hallucination-leaderboard.gitcd hallucination-leaderboard# follow the project's README for install & configurationNeed it deployed, integrated, or customized instead? DEV.co ships production installs.
Best use cases
Implementation considerations
- Dataset is not publicly released; benchmarks are computed and published by Vectara. To reproduce or extend, HHEM-2.1-Open (open-source variant) is available on HuggingFace but may have different performance characteristics than HHEM-2.3.
- Evaluation covers 7700+ diverse articles with variable length (50–24K words) and complexity; ensure test data distribution aligns with your use case domain and document lengths.
- Leaderboard is updated periodically (last update May 2026). Monitor release branch for new model additions; model rankings may shift as upstream models are patched or new versions released.
- Metrics are summarization-specific (hallucination rate, factual consistency, answer rate, summary length). Do not generalize findings to other NLP tasks or use cases.
- HHEM-2.3 is a commercial model; access and licensing terms are not disclosed in this data. Requires engagement with Vectara for production use.
When to avoid it — and what to weigh
- General-purpose Model Evaluation — Leaderboard measures only hallucination in document summarization. Does not evaluate coding, reasoning, multilingual, or non-summarization tasks. Not suitable for broad model comparison.
- Proprietary or Custom Model Evaluation — Requires integration with HHEM-2.3 (commercial, closed-source). Cannot easily extend to evaluate internal or fine-tuned models without access to the evaluation model API or running HHEM-2.1-Open.
- Real-time Production Monitoring — Leaderboard is a static benchmark snapshot. Does not provide infrastructure for continuous evaluation of live model outputs or production hallucination detection in streaming scenarios.
- Compliance or Formal Model Auditing — Data set is proprietary and not publicly disclosed to avoid overfitting; limits reproducibility and third-party validation. Does not provide attestation, certification, or formal audit trail.
License & commercial use
Repository is licensed under Apache License 2.0 (Apache-2.0), a permissive OSI license. Source code, benchmark methodology, and leaderboard data in the GitHub repository are available for use, modification, and redistribution under Apache 2.0 terms.
Apache-2.0 permits commercial use of the repository code and benchmark framework. However, the underlying evaluation model (HHEM-2.3) is proprietary and owned by Vectara. License status, pricing, and permitted uses for HHEM-2.3 in production are Unknown. Requires explicit review with Vectara before commercial deployment of evaluation pipelines.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Adequate |
| License clarity | Needs review |
| Deployment complexity | Low |
| DEV.co fit | Good |
| Assessment confidence | High |
No security vulnerabilities or threat model detailed in available data. Dataset is proprietary (not publicly released), reducing attack surface for data poisoning. HHEM-2.3 is a closed-source model; no audits or threat assessments disclosed. If integrating via API or running locally, apply standard ML model security practices: validate inputs, monitor for prompt injection, and isolate evaluation workloads. No security incidents or CVEs noted.
Alternatives to consider
SUMMAC (NLI-based Inconsistency Detection)
Open-source academic benchmark for factual consistency in summarization. More reproducible and no commercial licensing dependency, but narrower model coverage and potentially less mature evaluation.
TrueTeacher / TRUE Framework
LLM-based and statistical approaches to factual consistency evaluation. Published research with reproducible methods; however, not maintained as a live leaderboard and requires manual integration.
Internal Custom Evaluation Pipeline
Build bespoke hallucination detection using domain-specific reference data and fine-tuned models. Trades off benchmark standardization for full control and customization to your document types and use cases.
Build on hallucination-leaderboard with DEV.co software developers
Use the Hallucination Leaderboard to compare models and identify the best performer for your document summarization task. Start with published benchmarks, validate on your data, and deploy with confidence.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
hallucination-leaderboard FAQ
Can I use the leaderboard results to select a model for my summarization task?
Can I evaluate my own LLM or fine-tuned model on this leaderboard?
How often is the leaderboard updated?
Is the dataset used for this leaderboard public?
Software developers & web developers for hire
From first prototype to production, DEV.co delivers software development services around tools like hallucination-leaderboard. Our software development agency staffs experienced software developers and web developers for custom software development, web development, integrations, and ongoing support across ai frameworks and beyond.
Optimize Your LLM Summarization Pipeline
Use the Hallucination Leaderboard to compare models and identify the best performer for your document summarization task. Start with published benchmarks, validate on your data, and deploy with confidence.