evalplus
EvalPlus is a Python framework for rigorously evaluating code generated by large language models using extended test suites (HumanEval+ with 80x more tests, MBPP+ with 35x more tests) and performance benchmarking via EvalPerf. It supports multiple LLM backends and provides safe code execution within Docker.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Repository | evalplus/evalplus |
| Owner | evalplus |
| Primary language | Python |
| License | Apache-2.0 — OSI-approved |
| Stars | 1.8k |
| Forks | 201 |
| Open issues | 65 |
| Latest release | v0.3.1 (2024-10-20) |
| Last updated | 2025-10-02 |
| Source | https://github.com/evalplus/evalplus |
What evalplus is
EvalPlus provides HumanEval+ and MBPP+ datasets with significantly expanded test coverage, plus EvalPerf for performance profiling of generated code. The framework integrates with Hugging Face, vLLM, and other inference backends, executes code safely in containerized environments, and supports efficiency evaluation via Linux perf tools.
Get the evalplus source
Clone the repository and explore it locally.
git clone https://github.com/evalplus/evalplus.gitcd evalplus# follow the project's README for install & configurationNeed it deployed, integrated, or customized instead? DEV.co ships production installs.
Best use cases
Implementation considerations
- Code execution runs within Docker by default for safety; ensure Docker daemon availability and proper volume mounting for result persistence.
- EvalPerf requires Linux kernel permission: `echo 0 > /proc/sys/kernel/perf_event_paranoid` to enable hardware performance counter access.
- Multiple inference backend options (HuggingFace transformers, vLLM, Google Gemini, Anthropic); choose based on model format and latency requirements.
- Memory footprint varies with model size and batch settings; vLLM backend recommended for efficiency at scale.
- Chat vs. base model detection is automatic via `tokenizer.chat_template` for HF/vLLM; use `--force-base-prompt` flag to override when needed.
When to avoid it — and what to weigh
- Need Windows support for EvalPerf — EvalPerf efficiency evaluation is *nix only and requires perf subsystem access; Windows users cannot run performance profiling.
- Evaluating non-Python code — EvalPlus datasets focus on Python; no built-in support for evaluating code generation in other languages.
- Require offline operation — Default inference backends require internet for model downloads; Docker execution helps but still needs model pre-caching.
- Custom domain-specific benchmarks — EvalPlus is fixed to HumanEval+/MBPP+/EvalPerf datasets; limited extensibility for proprietary or domain-specific evaluation needs.
License & commercial use
Apache License 2.0 (Apache-2.0). Permissive OSI license allowing commercial use, modification, and distribution with attribution and license notice retention.
Apache-2.0 is a permissive open-source license that explicitly permits commercial use. You may use EvalPlus in proprietary products, charge for services incorporating it, and modify the code. Ensure you retain the license header and provide a copy of the Apache-2.0 license to end users. No patent indemnification is granted.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Adequate |
| License clarity | Clear |
| Deployment complexity | Moderate |
| DEV.co fit | Good |
| Assessment confidence | High |
Code execution risk mitigated by default Docker containerization with image isolation. Timeout settings are configurable and conservative by default. User-supplied model code runs within the container; untrusted model outputs should not be executed outside isolated environments. Kernel perf access (EvalPerf) requires elevated privileges; use at host level only in trusted contexts.
Alternatives to consider
HumanEval (original)
Simpler but significantly weaker test coverage (80x fewer tests); used for quick model screening but insufficient for rigorous production evaluation.
bigcode-evaluation-harness
Broader language support and cross-framework benchmarking; can consume EvalPlus datasets but less specialized for Python code rigor and efficiency metrics.
CodeXGLUE
Broader code understanding tasks beyond generation (clone detection, code search); less focused on evaluating correctness of synthesized code.
Build on evalplus with DEV.co software developers
Use EvalPlus to benchmark LLM code quality beyond simple pass rates. Run correctness and efficiency evaluations today.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
evalplus FAQ
Can I use EvalPlus to evaluate models on my custom test cases?
Does EvalPlus work on Windows?
What inference backends are supported?
How do I interpret the pass@k metric differences before and after EvalPlus tests?
Work with a software development agency
From first prototype to production, DEV.co delivers software development services around tools like evalplus. Our software development agency staffs experienced software developers and web developers for custom software development, web development, integrations, and ongoing support across open-source testing and beyond.
Evaluate Your Code Generation Models with Rigor
Use EvalPlus to benchmark LLM code quality beyond simple pass rates. Run correctness and efficiency evaluations today.