AI Frameworks · Giskard-AI

giskard-oss

Giskard is an open-source Python library for testing and evaluating AI agents and LLM-based systems. It provides tools for creating evals (tests that validate non-deterministic outputs), red-teaming agents for vulnerabilities, and generating synthetic evaluation datasets—all with minimal dependencies and async-first architecture.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/Giskard-AI/giskard-oss

5.5k

GitHub stars

482

Forks

Python

Primary language

Apache-2.0

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	Giskard-AI/giskard-oss
Owner	Giskard-AI
Primary language	Python
License	Apache-2.0 — OSI-approved
Stars	5.5k
Forks	482
Open issues	73
Latest release	v2.19.2 (2026-07-06)
Last updated	2026-07-07
Source	https://github.com/Giskard-AI/giskard-oss

What giskard-oss is

Giskard v3 is a modular, lightweight Python library built on giskard-core that wraps LLMs, black-box agents, and multi-step pipelines. It offers scenario-based testing with built-in checks (Groundedness, Conformity, LLMJudge), an automated vulnerability scanner (Scan), and planned RAG evaluation capabilities. Python 3.12+ required; includes optional telemetry.

Quickstart

Get the giskard-oss source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/Giskard-AI/giskard-oss.git2cd giskard-oss3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

Regression Testing for LLM Applications

Use Scenario-based checks to verify that RAG systems, chatbots, and agentic pipelines continue to produce valid, grounded outputs after code or model changes. Built-in Groundedness and Conformity checks validate output quality without requiring deterministic answers.

Red-Teaming and Vulnerability Scanning

Run Giskard Scan to automatically generate adversarial test suites covering prompt injection, data leakage, harmful content, and OWASP LLM Top-10 threats. Extend with custom scenario generators for domain-specific attack vectors.

Multi-Turn Agent Evaluation

Test full conversation flows and agentic decision trees with async-first scenario API. Validate both single-turn and multi-step agent behavior, including tool use and state management across interactions.

Implementation considerations

Giskard v3 is in Beta (Checks and Scan). RAG evaluation is still in v2 and planned for v3; confirm feature availability before committing to evaluation pipelines that depend on RAGET.
LLMJudge checks require LLM API access (OpenAI, etc.). Budget for inference costs and latency in evaluation runs; async-first architecture helps parallelize multi-scenario tests.
Telemetry is optional but enabled by default. Configure opt-out in giskard-core if data residency or air-gapped requirements exist.
Scenario API is async-first (uses asyncio). Wrap with asyncio.run() in scripts or integrate into async event loops in production frameworks.
Custom checks and scenario generators require Python knowledge. Simple string/regex checks are straightforward; LLM-as-judge checks need prompt engineering and model selection.

When to avoid it — and what to weigh

Deterministic Model Testing — Giskard is designed for non-deterministic systems. If you need strict unit testing for classical ML or rule-based systems, use pytest or unittest instead.
Real-Time Production Monitoring — Giskard is a testing and evaluation library, not a production monitoring framework. Use it in CI/CD pipelines or batch evaluation workflows, not for inline request validation.
Offline-Only Environments — Giskard includes optional telemetry and may require external API calls (for LLM-as-judge checks). If you need fully air-gapped operation, review telemetry settings and LLMJudge dependencies.
Low Python Version Support — Requires Python 3.12+. Do not use if your infrastructure is locked to Python 3.10 or earlier.

License & commercial use

Licensed under Apache License 2.0, a permissive OSI-approved license. Allows commercial use, modification, and distribution with minimal restrictions.

Apache 2.0 permits commercial use without royalties. You may use Giskard in commercial products, modify the source, and distribute derivatives. Retain license notices. No explicit guarantee of support or warranties from Giskard-AI; use in production assessment and support contracts require direct negotiation with Giskard-AI or community contribution.

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Adequate
License clarity	Clear
Deployment complexity	Low
DEV.co fit	Strong
Assessment confidence	High

Security considerations

Giskard itself is a testing tool, not a security product. It helps identify vulnerabilities (e.g., prompt injection, data leakage) in your agents through red-teaming. No security audit details provided in source data. LLMJudge checks depend on external LLM services; ensure your prompts and test data do not leak sensitive information through API calls. Optional telemetry should be reviewed per your compliance requirements.

Alternatives to consider

Promptfoo

Also tests LLM chains and agents with scenario-based checks. Lighter weight, but lacks built-in red-teaming (Scan) and focused on prompt optimization rather than vulnerability detection.

LangSmith (LangChain)

Provides evaluation, tracing, and monitoring for LLM applications. Tighter integration with LangChain; requires vendor lock-in. Better for production observability; weaker at automated red-teaming.

Arize AI (ModelOps)

Enterprise-grade ML monitoring with LLM-specific features. Heavier, requires infrastructure. Better for production monitoring; Giskard is better for pre-deployment testing and red-teaming.

Software development agency

Build on giskard-oss with DEV.co software developers

Use Giskard to automate regression testing, red-team for vulnerabilities, and validate multi-turn agent behavior. Start with pip install giskard-checks—no vendor lock-in.

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

trulens evalscope osmedeus

Related on DEV.co

Explore the category and the services that help you build with it.

AI Frameworks AI Development DevOps & CI/CD Talk to DEV.co

giskard-oss FAQ

Do I have to use OpenAI for Checks or Scan?

No. Checks work with any callable (your own model, local LLM, API). LLMJudge checks need an LLM API (OpenAI by default, but you can pass other providers). Scan needs LLM access for test generation; specify your provider in the call.

Is Giskard v3 production-ready?

Giskard Checks and Scan are in Beta. RAG evaluation (giskard-rag) is planned. Use in pre-deployment testing and CI/CD; not yet recommended as sole evaluation framework for live systems. v2 is stable but no longer maintained.

How much does telemetry collect?

Per the README, telemetry is optional, aggregated, and does not include prompts, model outputs, or scenario text. See giskard-core/README.md for opt-out details.

Can I use Giskard offline?

Partially. String/regex checks work offline. LLMJudge checks and Scan require external LLM APIs. If you run local LLMs and opt out of telemetry, you can be mostly offline.

Custom software development services

Need help beyond evaluating giskard-oss? DEV.co is a software development agency offering software development services and web development for teams of every size. Our software developers and web developers build custom software, web applications, APIs, and ai frameworks integrations — and maintain them long-term.

Strengthen Your AI Agent Quality

Use Giskard to automate regression testing, red-team for vulnerabilities, and validate multi-turn agent behavior. Start with pip install giskard-checks—no vendor lock-in.

Talk to DEV.co Schedule a Consultation