trafilatura
Trafilatura is a Python tool for discovering, downloading, and extracting text and metadata from web pages. It handles crawling, HTML parsing, and outputs structured data in multiple formats (JSON, CSV, Markdown, HTML, XML), making it useful for building text corpora and content pipelines.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Repository | adbar/trafilatura |
| Owner | adbar |
| Primary language | Python |
| License | Apache-2.0 — OSI-approved |
| Stars | 6.3k |
| Forks | 393 |
| Open issues | 74 |
| Latest release | v2.1.0 (2026-06-07) |
| Last updated | 2026-07-01 |
| Source | https://github.com/adbar/trafilatura |
What trafilatura is
Python package providing web crawling (sitemap/feed support), robust HTML-to-text extraction using pattern matching and algorithms like jusText and readability, parallel processing, and configurable metadata extraction (title, author, date, categories). Outputs to TXT, Markdown, CSV, JSON, HTML, XML-TEI with optional language detection.
Get the trafilatura source
Clone the repository and explore it locally.
git clone https://github.com/adbar/trafilatura.gitcd trafilatura# follow the project's README for install & configurationNeed it deployed, integrated, or customized instead? DEV.co ships production installs.
Best use cases
Implementation considerations
- Extraction quality varies by site structure; configure settings (include comments, optional elements) and test on target domains.
- No built-in JavaScript rendering; pair with Selenium/Playwright for JS-heavy sites or use pre-rendered HTML.
- Language detection is optional and requires additional dependencies; evaluate performance on multilingual corpora.
- Memory usage grows with batch size; use parallel processing and pagination for large crawl jobs.
- Output formats are flexible but TEI and XML variants require familiarity with those schemas if used downstream.
When to avoid it — and what to weigh
- Heavy JavaScript-rendered content — Trafilatura works with static HTML; JavaScript-heavy sites require headless browser integration (not built-in).
- Real-time, low-latency extraction — Designed for batch and crawl workflows; single-page extraction is reasonably fast but not optimized for microsecond latencies.
- Complex table or image extraction — While tables and images are supported, extraction is basic; specialized layout-aware tools may be needed for dense structured data.
- Legal/compliance scraping without review — Tool does not include robots.txt compliance or rate-limiting enforcement; user responsible for respecting site policies and laws.
License & commercial use
Apache 2.0 license (current, v2.1.0). Versions prior to v1.8.0 were GPLv3+. Apache 2.0 is a permissive OSI-approved license allowing commercial use, modification, and distribution with liability disclaimers and patent protection.
Apache 2.0 permits commercial use, including in proprietary products. No license restrictions on distribution or modification. No warranty or support obligation from maintainer. For mission-critical deployments, evaluate maintenance cadence and consider contributing upstream or sponsoring the author.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Active |
| Documentation | Strong |
| License clarity | Clear |
| Deployment complexity | Low |
| DEV.co fit | Strong |
| Assessment confidence | High |
Tool fetches and parses untrusted HTML; potential DoS from malformed or recursive structures. No built-in request validation or rate-limiting. User responsible for: respecting robots.txt, rate-limiting, SSL verification, storing sensitive extracted data securely. Consider sandboxing if extracting from untrusted sources. No security audit data provided.
Alternatives to consider
Readability (Mozilla/Arc90)
Simpler, JavaScript-based article extraction; less control over metadata and output formats; primarily DOM-focused.
Newspaper3k / Newspaper
Python article extractor with NLP; smaller feature set for crawling and format output; less active maintenance.
Scrapy
Full-featured web scraping framework; steeper learning curve; more suitable for custom multi-page extraction pipelines than standalone content extraction.
Build on trafilatura with DEV.co software developers
Trafilatura handles crawling, parsing, and structured output. Pair it with our AI and API development services to scale content pipelines and integrate with LLMs.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
trafilatura FAQ
Does Trafilatura respect robots.txt?
Can it extract from JavaScript-rendered pages?
What Python versions are supported?
Is there a Docker image or deployment package?
Custom software development services
DEV.co is a software development agency delivering custom software development services to companies building on open source. Our software developers and web developers design, integrate, and ship production systems — spanning web development, APIs, AI, data, and cloud. If trafilatura is part of your ai frameworks roadmap, our team can implement, customize, migrate, and maintain it.
Ready to build a web extraction pipeline?
Trafilatura handles crawling, parsing, and structured output. Pair it with our AI and API development services to scale content pipelines and integrate with LLMs.