AI Coding Agents · LiveCodeBench

LiveCodeBench

LiveCodeBench is an open-source evaluation framework that continuously collects competitive programming problems from LeetCode, AtCoder, and CodeForces to benchmark LLM coding capabilities. It measures multiple dimensions—code generation, self-repair, code execution, and test prediction—with fresh problems regularly added to prevent training data contamination.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/LiveCodeBench/LiveCodeBench

904

GitHub stars

193

Forks

Python

Primary language

MIT

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	LiveCodeBench/LiveCodeBench
Owner	LiveCodeBench
Primary language	Python
License	MIT — OSI-approved
Stars	904
Forks	193
Open issues	35
Latest release	Unknown
Last updated	2025-07-16
Source	https://github.com/LiveCodeBench/LiveCodeBench

What LiveCodeBench is

A Python-based benchmarking suite using vllm for inference and pass@k metrics for evaluation. Supports both open-source models (via vllm with multi-GPU parallelization) and closed API models (via multiprocessing). Features modular scenario runners, custom evaluators, and time-windowed scoring to isolate pre- and post-training-cutoff performance.

Quickstart

Get the LiveCodeBench source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/LiveCodeBench/LiveCodeBench.git2cd LiveCodeBench3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

Continuous LLM Code Capability Tracking

Evaluate proprietary or open models against a regularly updated benchmark that resists data contamination. Ideal for tracking model improvements across releases without concern that models have memorized test cases.

Multi-Scenario Code Evaluation

Assess models beyond generation—code repair, test prediction, and execution prediction provide broader picture of coding competence than single-task benchmarks. Useful for production readiness assessment.

Research on Code LLM Generalization

Support research into how LLMs handle competitive programming and real-world coding tasks with granular time-windowed filtering, custom evaluators, and multiple dataset versions for reproducibility.

Implementation considerations

Requires Python 3.11 and uv for dependency management; setup is straightforward but adds tooling dependency beyond pip.
GPU or cloud compute needed for vllm-based inference on open models; API quota/cost required for closed models with multiprocessing.
Time limits in evaluation can cause <0.5% variance; larger variation suggests tuning needed on --num_process_evaluate and --timeout flags.
Dataset versions (v1–v6, up to 1055 problems) grow over time; must explicitly specify --release_version or accept rolling updates via release_latest.
Custom evaluators supported for external model outputs; JSON format must match expected schema in custom_evaluator.py.

When to avoid it — and what to weigh

You need pre-built leaderboard results immediately — README notes that leaderboard scores are being updated following a pruning of test cases in the lite version. Results may be in flux; expect ongoing changes.
You require inference for very large closed models without API quotas — Open models use vllm (requires GPU), closed models require multiprocess API calls. No built-in batching optimizer for cost or rate-limit management.
You need evaluation beyond code correctness — Metrics are primarily pass@1 and pass@5 based on test execution. No measurement of code quality, readability, performance, or style conformance.
You need long-term stability guarantees — Project is actively modified (last push July 2025); dataset versions are frequently updated. Breaking changes between versions possible; not packaged as stable releases.

License & commercial use

MIT License (permissive OSI license). Allows commercial use, modification, and distribution with attribution. No restrictions on derivative works or commercial applications.

MIT is a permissive license that explicitly permits commercial use. Benchmark data (problems from public competitive platforms) should be treated as derivative/fair use for evaluation purposes, but legal review is advised if republishing or commercializing benchmark data directly. No licensing concerns with using the evaluation framework itself commercially.

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Adequate
License clarity	Clear
Deployment complexity	Moderate
DEV.co fit	Good
Assessment confidence	High

Security considerations

Executes untrusted model-generated code as part of evaluation (via test execution). Sandboxing or resource limits for code execution not mentioned; consider isolation (containers, VMs) if evaluating code from untrusted sources. API keys for closed models must be managed securely. No explicit discussion of data privacy for benchmark problems sourced from public platforms.

Alternatives to consider

HumanEval / MBPP

Smaller, static benchmarks with no contamination concerns but lack continuous updates and multi-scenario evaluation. Simpler to use for quick spot checks.

CodeXGLUE

Broader code understanding tasks beyond generation, but less focused on competitive programming and less actively maintained than LiveCodeBench.

EvalPlus (HumanEval+)

Extended test suite for HumanEval with more edge-case coverage, but still static and smaller scale. Better for fine-grained evaluation of single-task performance.

Software development agency

Build on LiveCodeBench with DEV.co software developers

Use LiveCodeBench to evaluate your model's coding capabilities across generation, repair, execution, and test prediction—with continuously updated, competition-sourced problems that resist data contamination.

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

opencompass llm-sandbox Qwen3-Coder-Next

Related on DEV.co

Explore the category and the services that help you build with it.

AI Coding Agents AI Application Development API Development Custom Software Development Talk to DEV.co

LiveCodeBench FAQ

How do I avoid contamination concerns when evaluating a model trained on code from the internet?

Use time-windowed evaluation with --start_date and --end_date flags to isolate problems published after your model's training cutoff. LiveCodeBench was designed for this; see compute_scores.py.

Can I use LiveCodeBench with my proprietary closed model?

Yes, via custom_evaluator.py. Generate outputs from your model externally, format as JSON per the spec, and run evaluation without exposing the model itself.

What is the difference between release_v1 and release_v6?

Dataset size grows from 400 to 1055 problems; v6 includes more recent problems (up to Apr 2025). Use --release_version to specify; defaults to release_latest.

Why is the lite version used by default?

README notes it offers 'similar performance estimation much faster' by pruning large test suites. Leaderboard results are being updated to this standard; use --not_fast to revert to full test set.

Custom software development services

DEV.co helps companies turn open-source tools like LiveCodeBench into production software. Our software development services cover the full lifecycle — architecture, web development, integration, and maintenance — delivered by software developers and web developers who ship. Engage our software development agency to implement or customize it for your ai coding agents stack.

Benchmark Your Code LLM Today

Talk to DEV.co Schedule a Consultation