Open-Source Testing · antiwork

shortest

Shortest is a TypeScript framework that lets you write end-to-end tests in plain English, using AI (Claude API) to interpret and execute them against your application. It integrates Playwright for browser automation and supports GitHub 2FA, email validation, and test chaining.

Talk to DEV.co Discuss Implementation

Source: GitHub — github.com/antiwork/shortest

5.6k

GitHub stars

334

Forks

TypeScript

Primary language

MIT

License (OSI-approved)

Key facts

Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.

Field	Value
Repository	antiwork/shortest
Owner	antiwork
Primary language	TypeScript
License	MIT — OSI-approved
Stars	5.6k
Forks	334
Open issues	2
Latest release	v0.4.9 (2025-04-01)
Last updated	2026-05-21
Source	https://github.com/antiwork/shortest

What shortest is

Built on Playwright, Shortest leverages Anthropic's Claude API to parse natural language test descriptions and execute them as E2E test flows. It includes lifecycle hooks, callback assertions, API testing support, and CI/CD integration via headless mode. Configuration is declarative (TypeScript config files), and tests run through a custom CLI runner.

Quickstart

Get the shortest source

Clone the repository and explore it locally.

terminalbash

1git clone https://github.com/antiwork/shortest.git2cd shortest3# follow the project's README for install & configuration

Need it deployed, integrated, or customized instead? DEV.co ships production installs.

Best use cases

Rapid E2E test coverage for product teams

Teams wanting to write tests faster by describing user journeys in English rather than imperative Selenium/Playwright code. Reduces friction for non-automation-expert team members.

GitHub authentication and multi-step user flows

Built-in 2FA support and Mailosaur integration make it well-suited for testing complex login flows, invite workflows, and email-based verification processes.

Continuous integration testing in CI/CD pipelines

Headless mode and GitHub Actions example workflow show it's designed for automated testing in CI; natural language specs double as living documentation.

Implementation considerations

Anthropic API key must be provisioned and stored securely in CI/CD; costs should be tracked and budgeted per sprint.
Test files follow a simple `*.test.ts` pattern; minimal ramp-up for teams familiar with Jest/Mocha conventions.
Lifecycle hooks (`beforeAll`, `beforeEach`, `afterEach`, `afterAll`) and callback assertions enable flexible test composition; plan for shared setup (auth, DB state).
CLI supports line-number-based test execution and selective file runs; integrate with CI workflows similar to Playwright or Jest.
Browser context options (e.g., `ignoreHTTPSErrors`) are configurable; environment-specific overrides (local vs. staging) should be parameterized.

When to avoid it — and what to weigh

Strict cost control on AI API usage — Every test run invokes the Anthropic Claude API. High test volume or frequent reruns will incur per-token costs; no pricing/budget controls mentioned in README.
Deterministic, reproducible test behavior required — AI interpretation of English descriptions may vary between API versions or model updates. Tests relying on exact execution sequences or timing may become flaky.
Legacy browser support or non-Playwright stacks — Locked to Playwright; no mention of Safari, Internet Explorer, or mobile browser support. If you need those, you'll need a different framework.
Minimal external dependencies or offline-first testing — Requires real-time API calls to Anthropic; no option for local/offline test execution mentioned. Network outages or API rate limits will block test runs.

License & commercial use

MIT License. Permissive OSI-approved license allowing free commercial use, modification, and distribution with attribution required.

MIT is a permissive open-source license suitable for commercial use. No restrictions on proprietary applications using Shortest as a dependency. However, verify that your use of the Anthropic Claude API is compliant with Anthropic's terms of service (separate from the software license).

DEV.co evaluation signals

Editorial assessment — not user reviews. Directional, with an explicit confidence level.

Signal	Assessment
Maintenance	Active
Documentation	Strong
License clarity	Clear
Deployment complexity	Low
DEV.co fit	Good
Assessment confidence	High

Security considerations

Anthropic API keys must be treated as secrets and never committed to version control. The `.env.local` is added to `.gitignore` by default. Test files may contain hardcoded credentials (e.g., test usernames/passwords); use environment variables instead. AI model receives test descriptions and page content; review what sensitive data is sent to Anthropic. Playwright browser context options allow disabling HTTPS validation (`ignoreHTTPSErrors`); avoid in production testing. No explicit mention of input sanitization or injection protection in AI prompts.

Alternatives to consider

Playwright + custom assertion helpers

Direct Playwright scripting avoids AI API costs and non-determinism, but requires more boilerplate and higher bar for non-engineers to contribute tests.

Cypress or WebdriverIO

Established E2E frameworks with larger ecosystems, better IDE support, and synchronous syntax. No AI overhead, but also no natural language abstraction.

OpenTest or Testim (commercial AI-driven tools)

Proprietary platforms offering codeless/low-code AI E2E testing with visual recorders and enterprise dashboards. Higher cost but less engineering overhead.

Software development agency

Build on shortest with DEV.co software developers

Shortest transforms test writing from imperative code to plain English. Start with `npx @antiwork/shortest init` and evaluate whether natural language tests fit your team's workflow and Anthropic API costs.

Talk to DEV.co

Related open-source tools

Surfaced by semantic similarity across the DEV.co open-source index.

testsprite-cli playwright-skill ts-jest

Related on DEV.co

Explore the category and the services that help you build with it.

Open-Source Testing AI Development Web App Development API Development Talk to DEV.co

shortest FAQ

What happens if the AI misinterprets a test description?

Depends on test complexity and clarity. Simple, concrete English descriptions (e.g., 'click button with text Login') are more reliable. Ambiguous or vague descriptions may fail unpredictably. Callback assertions and logging help debug failures, but iterative refinement of test prose is expected.

Can I run Shortest without paying for Anthropic's API?

No. Shortest requires a valid Anthropic API key and will incur usage costs per test run. Anthropic offers a free tier with limited tokens; review pricing and budget constraints before committing to the framework.

Does Shortest support mobile or cross-browser testing?

Playwright supports Chrome, Firefox, and WebKit; Shortest inherits this. Mobile device emulation is possible via Playwright's device profiles (not explicitly shown in README). Safari on macOS and mobile browsers require custom device config.

How does Shortest compare to GitHub's native test automation?

Shortest is a testing framework; GitHub Actions is a CI/CD platform. Shortest tests run *in* GitHub Actions (as shown in the example workflow), so they are complementary, not competitive.

Work with a software development agency

From first prototype to production, DEV.co delivers software development services around tools like shortest. Our software development agency staffs experienced software developers and web developers for custom software development, web development, integrations, and ongoing support across open-source testing and beyond.

Ready to simplify your E2E testing?

Talk to DEV.co Schedule a Consultation