RAG Frameworks · modelscope

evalscope

EvalScope is an open-source Python framework for evaluating large language models (LLMs), vision-language models (VLMs), and other AI systems across multiple benchmarks. It provides built-in datasets, performance testing, visualization dashboards, and agent evaluation capabilities with minimal setup overhead.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/modelscope/evalscope

GitHub stars

417

Forks

Python

Primary language

Apache-2.0

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	modelscope/evalscope
Owner	modelscope
Primary language	Python
License	Apache-2.0 — OSI-approved
Stars	3k
Forks	417
Open issues	30
Latest release	v1.9.0 (2026-07-07)
Last updated	2026-07-08
Source	https://github.com/modelscope/evalscope

What evalscope is

Apache-2.0 licensed Python framework supporting LLM/VLM/AIGC evaluation via multiple backends (OpenCompass, VLMEvalKit, RAGEval). Features multi-turn agent loop evaluation with pluggable strategies (function-calling, ReAct), stress testing (TTFT/TPOT metrics), and interactive WebUI dashboards for result analysis. Extensible architecture allows custom datasets, metrics, and evaluators.

Quickstart

Get the evalscope source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/modelscope/evalscope.git2cd evalscope3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

Comparative LLM Benchmarking

Evaluate and rank multiple LLM/VLM candidates (e.g., proprietary vs. open-source) across standardized benchmarks (MMLU, C-Eval, GSM8K) with built-in multi-dimensional comparison dashboards and pairwise battle modes.

Agent & Agentic Workflow Evaluation

Test multi-turn agent capabilities (ReAct, function-calling, code execution) with sandbox isolation, per-sample trace recording, and visualization. Supports SWE-bench, GAIA, and custom agent-driven benchmarks.

Model Service Performance Testing

Conduct inference stress tests with metrics (time-to-first-token, tokens-per-output-token, throughput) and validate third-party API deployments against official model behavior via Vendor Verifier benchmarks.

Implementation considerations

Python ≥3.10 required; install via pip (evalscope package on PyPI). One-command setup via CLI, but agent/sandbox modes require Docker runtime and external API credentials (OpenAI, proprietary model endpoints).
Evaluation backends (OpenCompass, VLMEvalKit, RAGEval) are pluggable. Ensure compatibility of chosen backend with target model type (LLM vs. VLM vs. embedding) before deployment.
Agent evaluation with multi-turn traces requires model support for function-calling or ReAct-style outputs. Tool definitions (bash, python, submit) must align with model's instruction-following capability.
Performance testing mode generates high request volume; throttle API rate limits and monitor quota consumption when stress-testing against production endpoints.
Custom dataset and metric registration via @register decorators; ensure dataset schema and scorer implementation match framework's Pydantic models to avoid runtime errors.

When to avoid it — and what to weigh

Lightweight Single-Benchmark Use Case — If you need only basic evaluation of one dataset without visualization or multi-model comparison, the framework's breadth may introduce unnecessary complexity.
No Docker/Sandbox Support Available — Agent evaluation modes and many benchmarks (SWE-bench, GAIA) require Docker runtime and network access. Non-sandboxed or air-gapped environments will be limited.
Real-time Streaming Evaluation Required — Framework focuses on batch evaluation and post-hoc analysis. Real-time online evaluation or streaming response analysis capabilities are not documented.
Proprietary Data / Closed Benchmark Integration — Extensibility is high for custom datasets, but no integration of proprietary benchmarks or closed-source evaluation harnesses is guaranteed without custom development.

License & commercial use

Licensed under Apache License 2.0 (Apache-2.0), a permissive OSI-approved license allowing commercial use, modification, and distribution with no warranty. Attribution required.

Apache-2.0 permits commercial use without restriction. No commercial licensing, support contracts, or warranty are provided by the license itself. Use in production or as part of a commercial product is permitted; however, ensure any integrated models, datasets, or third-party backends comply with their own licensing terms (e.g., OpenAI API terms, proprietary benchmark licenses).

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Strong
License clarity	Clear
Deployment complexity	Moderate
DEV.co fit	Strong
Assessment confidence	High

Security considerations

Framework executes arbitrary Python and bash code in agent evaluation modes via local or Docker sandbox isolation. Ensure input datasets are trusted; malicious prompts or tool definitions could lead to code execution. Docker sandbox mitigates local code execution risk if properly configured. API credentials (OpenAI key, model endpoints) are handled via environment variables; secure credential storage and access control are the user's responsibility. No explicit security audit or vulnerability disclosure policy documented.

Alternatives to consider

OpenCompass

Standalone LLM evaluation framework; EvalScope integrates it as a backend. Choose OpenCompass if you prefer a simpler, model-evaluation-only tool without agent modes or stress testing.

LM Evaluation Harness (EleutherAI)

Lightweight, widely-adopted Python harness for few-shot evaluation. Lacks agent modes, visualization, and stress testing; better for minimal, reproducible benchmarking of language-only models.

PromptEvaluator / Prompt Tools

Simpler prompt-level evaluation without multi-model comparison or structured benchmarking. Choose if evaluation scope is limited to single-shot quality assessment rather than standardized benchmarks.

Software development agency

Build on evalscope with DEV.co software developers

Start with a single pip install and one command. Evaluate across 50+ benchmarks, compare models visually, and profile agent workflows—all in one framework.

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

ragas trulens opencompass

Related on DEV.co

Explore the category and the services that help you build with it.

RAG Frameworks AI Application Development API Development DevOps Talk to DEV.co

evalscope FAQ

Can I evaluate my custom model or a closed-source API?

Yes. EvalScope supports OpenAI-compatible APIs and custom model services. Pass --api-url and --api-key. Integration with proprietary APIs depends on exposing a chat/completion endpoint compatible with the framework's adapter interface.

What models does agent evaluation support?

Agent modes require models capable of function-calling or ReAct-style output (e.g., GPT-4, Claude, Llama with tool-use instruction-tuning). Verify instruction-following quality before deployment; poor reasoning will fail multi-turn tasks.

Is commercial use allowed?

Yes, Apache-2.0 permits commercial use and distribution. However, ensure compliance with the licenses and terms of integrated models (OpenAI API, proprietary benchmarks) and any datasets used.

Do I need Docker for all evaluations?

No. Basic benchmark evaluation (GSM8K, MMLU) requires only Python. Docker is required only for agent evaluation modes (GAIA, SWE-bench), sandbox tool execution (bash, python), and some stress testing scenarios.

Software development & web development with DEV.co

From first prototype to production, DEV.co delivers software development services around tools like evalscope. Our software development agency staffs experienced software developers and web developers for custom software development, web development, integrations, and ongoing support across rag frameworks and beyond.

Ready to Benchmark Your Models?

Start with a single pip install and one command. Evaluate across 50+ benchmarks, compare models visually, and profile agent workflows—all in one framework.

Talk to DEV.co Schedule a Consultation