AI ERP with agents inside the systems you run
AI phone and voice agents for real calls
LLM.coPrivate, self-hosted LLM deployments
Legal AI infrastructure for firms
AI RFP discovery and response drafting
Automatic.coBusiness process automation
CyberAttack.aiThreat detection and defense
Secure AI virtual data roomswebclaw
webclaw is a Rust-based web scraper and crawler that extracts clean markdown, JSON, and LLM-ready content from websites. It runs locally via CLI, MCP server, or self-hosted REST API, with optional cloud features for JavaScript-heavy sites and advanced workflows.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Repository | 0xMassi/webclaw |
| Owner | 0xMassi |
| Primary language | Rust |
| License | AGPL-3.0 — OSI-approved |
| Stars | 1.7k |
| Forks | 181 |
| Open issues | 0 |
| Latest release | v0.6.14 (2026-06-27) |
| Last updated | 2026-07-02 |
| Source | https://github.com/0xMassi/webclaw |
What webclaw is
Rust extraction engine with local-first architecture supporting multiple output formats (markdown, JSON, LLM-optimized text, HTML). Integrates via CLI, MCP protocol for AI agents, REST API, and SDKs (TypeScript, Python, Go). Optional cloud backend for bot-protected sites and async jobs.
Get the webclaw source
Clone the repository and explore it locally.
git clone https://github.com/0xMassi/webclaw.gitcd webclaw# follow the project's README for install & configurationNeed it deployed, integrated, or customized instead? DEV.co ships production installs.
Best use cases
Implementation considerations
- AGPL-3.0 license requires legal review before use in proprietary commercial products; source-code disclosure obligations apply if modified and distributed.
- Local extraction works without API key for static HTML; set WEBCLAW_API_KEY for JavaScript rendering, bot-protected sites, and advanced search/research workflows.
- Build from source requires platform-specific dependencies (libssl-dev, cmake, clang); use Homebrew, prebuilt binaries, or Docker to avoid compilation overhead.
- MCP integration works with Claude Desktop, Cursor, Windsurf, and other compatible agents; use `npx create-webclaw` for automatic configuration.
- Extraction quality depends on site structure and bot-detection triggers; test against target sites before production deployment.
When to avoid it — and what to weigh
- Heavy JavaScript Rendering Required (Without API Key) — Local CLI and MCP server cannot render JavaScript-heavy SPAs without cloud API. Complex modern web apps may return incomplete content unless WEBCLAW_API_KEY is set for hosted rendering.
- Permissive Commercial License Needed — Licensed under AGPL-3.0, which requires source-code disclosure if modified and distributed. Not suitable for proprietary closed-source commercial products without careful legal review.
- Minimal or No Operational Overhead Desired — Self-hosting requires Rust build dependencies (pkg-config, libssl-dev, cmake, clang, build-essential) and infrastructure. Managed service at webclaw.io eliminates this but introduces vendor dependency.
- Guaranteed Crawl Success Against All Sites — Bot detection, Cloudflare, and JavaScript-protected content require hosted API. Local extraction is fast but may encounter blocks; no guarantee of universal site coverage.
License & commercial use
Licensed under AGPL-3.0 (GNU Affero General Public License v3.0). Requires source-code disclosure if modified and distributed as a service or in derivative works. Network use triggers copyleft obligations.
AGPL-3.0 is a strong copyleft license. Using the unmodified OSS version for internal operations (via CLI or self-hosted server) is likely permissible, but distributing modifications, offering as a managed service, or incorporating into proprietary products requires legal review and potential source disclosure. Hosted webclaw.io service may have separate commercial terms; requires review.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Strong |
| License clarity | Clear |
| Deployment complexity | Low |
| DEV.co fit | Strong |
| Assessment confidence | High |
Web scraping inherently interacts with untrusted remote content; input validation, output sanitization, and bot-detection evasion (TLS fingerprinting mentioned in topics) carry risk. AGPL-3.0 source is auditable. No security audit, CVE history, or threat model provided. Self-hosted deployments inherit infrastructure security burden; managed service shifts risk to vendor.
Alternatives to consider
Firecrawl
Purpose-built web scraping API with built-in JavaScript rendering, bot handling, and LLM optimization. Proprietary SaaS; mention of 'Firecrawl-alternative' in webclaw topics suggests feature overlap but different licensing model.
Cheerio (Node.js) / BeautifulSoup (Python)
Lightweight, permissive open-source libraries for HTML parsing and extraction. Require manual orchestration; no built-in crawling, JavaScript rendering, or LLM-optimized output, but simpler licensing (MIT/BSD).
Crawl4AI (Python)
Python-native crawler with Markdown/JSON output, LLM integration, and MCP support. Unknown license and maintenance status; different language ecosystem than webclaw's Rust core.
Build on webclaw with DEV.co software developers
webclaw works locally via CLI or MCP for zero-friction integration with Claude, Cursor, and custom apps. Review AGPL-3.0 licensing before production use. Start with the Homebrew or Docker install, or evaluate the hosted API for JavaScript-heavy sites.
Talk to DEV.coRelated on DEV.co
Explore the category and the services that help you build with it.
webclaw FAQ
Can I use webclaw locally without an API key?
What are the AGPL-3.0 license implications for my business?
How do I integrate webclaw with my AI agent?
What if a site uses JavaScript heavily?
From evaluation to production software
Open-source adoption creates integration work. DEV.co supplies the software development services that close the gap between webclaw and a running rag frameworks system your team can operate.
Ready to add web scraping to your AI agent or application?
webclaw works locally via CLI or MCP for zero-friction integration with Claude, Cursor, and custom apps. Review AGPL-3.0 licensing before production use. Start with the Homebrew or Docker install, or evaluate the hosted API for JavaScript-heavy sites.