midscene
Midscene.js is an open-source, MIT-licensed framework for vision-driven UI automation and testing that works across web browsers, Android, iOS, and desktop applications. It uses multimodal AI models to interpret screenshots and perform actions described in natural language, eliminating the need for brittle DOM selectors or accessibility tree dependencies.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Repository | web-infra-dev/midscene |
| Owner | web-infra-dev |
| Primary language | TypeScript |
| License | MIT — OSI-approved |
| Stars | 14k |
| Forks | 1.1k |
| Open issues | 41 |
| Latest release | v1.10.3 (2026-07-07) |
| Last updated | 2026-07-08 |
| Source | https://github.com/web-infra-dev/midscene |
What midscene is
Built in TypeScript, Midscene.js leverages pure vision-based UI localization via multimodal models (Qwen, GLM-4V, Gemini, UI-TARS) to detect and interact with UI elements from screenshots alone. It integrates with Playwright/Vitest for test suites, supports autonomous operation via AI agents through Skills framework, and provides JavaScript SDK methods (aiAct, aiQuery, aiAssert) for cross-platform automation.
Get the midscene source
Clone the repository and explore it locally.
git clone https://github.com/web-infra-dev/midscene.gitcd midscene# follow the project's README for install & configurationNeed it deployed, integrated, or customized instead? DEV.co ships production installs.
Best use cases
Implementation considerations
- Model selection critical: framework supports multiple multimodal models (Qwen, Doubao, GLM-4V, Gemini, UI-TARS); evaluate cost, latency, and accuracy for your domain before committing.
- Screenshot infrastructure required: ensure reliable screenshot capture for your target platform (web via Playwright/Puppeteer; Android/iOS via platform tools); verify framebuffer or device access.
- Inference cost budgeting: vision model API calls incur per-request charges; large test suites and autonomous agents can accumulate significant costs; consider self-hosted open-source models (UI-TARS) for cost control.
- Natural language prompting skill: test quality depends on how effectively you describe steps in natural language; poorly written instructions lead to unreliable automation.
- Fallback and error handling: vision-based detection can fail on unusual UI or edge cases; design test logic with retry logic and explicit failure modes.
When to avoid it — and what to weigh
- High-Volume, Latency-Critical Testing — Vision-based inference adds latency compared to selector-based automation. Not suitable for millisecond-scale performance testing or batch jobs requiring sub-second response times.
- Headless-Only or Non-Visual Environments — Requires ability to capture screenshots. Environments without display capability (pure headless CI without framebuffer) will require workarounds or alternative approaches.
- Strongly Standardized, Static UI Structures — If your application uses stable, well-structured HTML with consistent semantic markup and no frequent visual changes, traditional selector-based frameworks may be simpler and faster.
- Complex Data Extraction at Scale — For large-scale data scraping or high-frequency extraction from many pages, vision-based inference cost and latency may be prohibitive compared to DOM/API-based extraction.
License & commercial use
MIT License (OSI-compliant, permissive). Allows unrestricted use, modification, and distribution for commercial and proprietary purposes, provided the license notice is included.
MIT license explicitly permits commercial use without restrictions. No license-based barriers to building proprietary automation tools or SaaS offerings on top of Midscene.js. However, cost is primarily in inference (multimodal model API calls), not licensing.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Strong |
| License clarity | Clear |
| Deployment complexity | Moderate |
| DEV.co fit | Strong |
| Assessment confidence | High |
Vision-based automation shares general security practices with browser/mobile automation frameworks: screenshots may contain sensitive data (PII, credentials); ensure secure handling of captured images and API calls to inference endpoints. Model inference via third-party APIs (OpenAI, Alibaba, Baidu) requires trust in those providers and compliance with data residency policies. No publicly disclosed vulnerabilities noted in provided data; refer to GitHub security advisories for latest guidance.
Alternatives to consider
Playwright / Cypress
Traditional DOM/selector-based testing. Lower latency and cost, more stable on well-structured apps; higher maintenance burden on UI refactors. No vision or cross-platform mobile support natively.
Appium
Established mobile automation framework (iOS/Android). Mature ecosystem and documentation; requires element locators (selectors, accessibility IDs); no built-in vision or AI-driven actions. Separate from web testing.
Other Vision-Based Tools (e.g., Claude Sonnet with CV, or commercial AI testing platforms)
Competitive offerings exist in the AI testing space; Midscene.js is open-source and MIT-licensed but may lack some proprietary features or support guarantees offered by closed alternatives.
Build on midscene with DEV.co software developers
Explore Midscene.js for your next automation or testing project. Start with web using Playwright integration, then expand to mobile. Review model strategy and cost implications before full deployment.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
midscene FAQ
Does Midscene.js require changes to my application code?
What are the inference costs, and can I run models locally?
How does Midscene.js compare to traditional Playwright selectors in speed?
Is there a commercial support option or SLA?
Software development & web development with DEV.co
DEV.co is a software development agency delivering custom software development services to companies building on open source. Our software developers and web developers design, integrate, and ship production systems — spanning web development, APIs, AI, data, and cloud. If midscene is part of your open-source testing roadmap, our team can implement, customize, migrate, and maintain it.
Ready to Eliminate Brittle Selectors?
Explore Midscene.js for your next automation or testing project. Start with web using Playwright integration, then expand to mobile. Review model strategy and cost implications before full deployment.