I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.

In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.

I made it open source for availability, anyone can use it without a gate. Now I’m building the platform for affordability, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.

• LLM ready output, smart Markdown with headings, tables, code, citation hints
• Fast in practice, async browser pool, caching, minimal hops
• Full control, sessions, proxies, cookies, user scripts, hooks
• Adaptive intelligence, learns site patterns, explores only what matters
• Deploy anywhere, zero keys, CLI and Docker, cloud friendly

• 🧹 Clean Markdown: Generates clean, structured Markdown with accurate formatting.
• 🎯 Fit Markdown: Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing.
• 🔗 Citations and References: Converts page links into a numbered reference list with clean citations.
• 🛠️ Custom Strategies: Users can create their own Markdown generation strategies tailored to specific needs.
• 📚 BM25 Algorithm: Employs BM25-based filtering for extracting core information and removing irrelevant content.

• 🤖 LLM-Driven Extraction: Supports all LLMs (open-source and proprietary) for structured data extraction.
• 🧱 Chunking Strategies: Implements chunking (topic-based, regex, sentence-level) for targeted content processing.
• 🌌 Cosine Similarity: Find relevant content chunks based on user queries for semantic extraction.
• 🔎 CSS-Based Extraction: Fast schema-based data extraction using XPath and CSS selectors.
• 🔧 Schema Definition: Define custom schemas for extracting structured JSON from repetitive patterns.

• 🖥️ Managed Browser: Use user-owned browsers with full control, avoiding bot detection.
• 🔄 Remote Browser Control: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.
• 👤 Browser Profiler: Create and manage persistent profiles with saved authentication states, cookies, and settings.
• 🔒 Session Management: Preserve browser states and reuse them for multi-step crawling.
• 🧩 Proxy Support: Seamlessly connect to proxies with authentication for secure access.
• ⚙️ Full Browser Control: Modify headers, cookies, user agents, and more for tailored crawling setups.
• 🌍 Multi-Browser Support: Compatible with Chromium, Firefox, and WebKit.
• 📐 Dynamic Viewport Adjustment: Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements.

• 🖼️ Media Support: Extract images, audio, videos, and responsive image formats like srcset and picture.
• 🚀 Dynamic Crawling: Execute JS and wait for async or sync for dynamic content extraction.
• 📸 Screenshots: Capture page screenshots during crawling for debugging or analysis.
• 📂 Raw Data Crawling: Directly process raw HTML (raw:) or local files (file://).
• 🔗 Comprehensive Link Extraction: Extracts internal, external links, and embedded iframe content.
• 🛠️ Customizable Hooks: Define hooks at every step to customize crawling behavior (supports both string and function-based APIs).
• 💾 Caching: Cache data for improved speed and to avoid redundant fetches.
• 📄 Metadata Extraction: Retrieve structured metadata from web pages.
• 📡 IFrame Content Extraction: Seamless extraction from embedded iframe content.
• 🕵️ Lazy Load Handling: Waits for images to fully load, ensuring no content is missed due to lazy loading.
• 🔄 Full-Page Scanning: Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages.

• 🐳 Dockerized Setup: Optimized Docker image with FastAPI server for easy deployment.
• 🔑 Secure Authentication: Built-in JWT token authentication for API security.
• 🔄 API Gateway: One-click deployment with secure token authentication for API-based workflows.
• 🌐 Scalable Architecture: Designed for mass-scale production and optimized server performance.
• ☁️ Cloud Deployment: Ready-to-deploy configurations for major cloud platforms.

• 🕶️ Stealth Mode: Avoid bot detection by mimicking real users.
• 🏷️ Tag-Based Content Extraction: Refine crawling based on custom tags, headers, or metadata.
• 🔗 Link Analysis: Extract and analyze all links for detailed data exploration.
• 🛡️ Error Handling: Robust error management for seamless execution.
• 🔐 CORS & Static Serving: Supports filesystem-based caching and cross-origin requests.
• 📖 Clear Documentation: Simplified and updated guides for onboarding and advanced usage.
• 🙌 Community Recognition: Acknowledges contributors and pull requests for transparency.

Choose the installation option that best fits your needs:

For basic web crawling and scraping tasks:

pip install crawl4ai
crawl4ai-setup # Setup the browser

By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.

👉 Note: When you install Crawl4AI, the crawl4ai-setup should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:

1. Through the command line:

   playwright install

2. If the above doesn't work, try this more specific command:

   python -m playwright install chromium

This second method has proven to be more reliable in some cases.

The sync version is deprecated and will be removed in future versions. If you need the synchronous version using Selenium:

pip install crawl4ai[sync]

For contributors who plan to modify the source code:

git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai
pip install -e .                    # Basic installation in editable mode

Install optional features:

pip install -e ".[torch]"           # With PyTorch features
pip install -e ".[transformer]"     # With Transformer features
pip install -e ".[cosine]"          # With cosine similarity features
pip install -e ".[sync]"            # With synchronous crawling (Selenium)
pip install -e ".[all]"             # Install all optional features

🚀 Now Available! Our completely redesigned Docker implementation is here! This new solution makes deployment more efficient and seamless than ever.

The new Docker implementation includes:

• Real-time Monitoring Dashboard with live system metrics and browser pool visibility
• Browser pooling with page pre-warming for faster response times
• Interactive playground to test and generate request code
• MCP integration for direct connection to AI tools like Claude Code
• Comprehensive API endpoints including HTML extraction, screenshots, PDF generation, and JavaScript execution
• Multi-architecture support with automatic detection (AMD64/ARM64)
• Optimized resources with improved memory management

# Pull and run the latest release
docker pull unclecode/crawl4ai:latest
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest

# Visit the monitoring dashboard at http://localhost:11235/dashboard
# Or the playground at http://localhost:11235/playground

Run a quick test (works for both Docker options):

import requests

# Submit a crawl job
response = requests.post(
    "http://localhost:11235/crawl",
    json={"urls": ["https://example.com"], "priority": 10}
)
if response.status_code == 200:
    print("Crawl job submitted successfully.")
    
if "results" in response.json():
    results = response.json()["results"]
    print("Crawl job completed. Results:")
    for result in results:
        print(result)
else:
    task_id = response.json()["task_id"]
    print(f"Crawl job submitted. Task ID:: {task_id}")
    result = requests.get(f"http://localhost:11235/task/{task_id}")

For more examples, see our Docker Examples. For advanced configuration, monitoring features, and production deployment, see our Self-Hosting Guide.

This release introduces crash recovery for deep crawls, a new prefetch mode for fast URL discovery, and critical security fixes for Docker deployments.

• 🔄 Deep Crawl Crash Recovery:

  • on_state_change callback fires after each URL for real-time state persistence
  • resume_state parameter to continue from a saved checkpoint
  • JSON-serializable state for Redis/database storage
  • Works with BFS, DFS, and Best-First strategies

  from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
  
  strategy = BFSDeepCrawlStrategy(
      max_depth=3,
      resume_state=saved_state,  # Continue from checkpoint
      on_state_change=save_to_redis,  # Called after each URL
  )

• ⚡ Prefetch Mode for Fast URL Discovery:

  • prefetch=True skips markdown, extraction, and media processing
  • 5-10x faster than full processing
  • Perfect for two-phase crawling: discover first, process selectively

  config = CrawlerRunConfig(prefetch=True)
  result = await crawler.arun("https://example.com", config=config)
  # Returns HTML and links only - no markdown generation

• 🔒 Security Fixes (Docker API):

  • Hooks disabled by default (CRAWL4AI_HOOKS_ENABLED=false)
  • file:// URLs blocked on API endpoints to prevent LFI
  • __import__ removed from hook execution sandbox

Full v0.8.0 Release Notes →

This release focuses on stability with 11 bug fixes addressing issues reported by the community. No new features, but significant improvements to reliability.

• 🐳 Docker API Fixes:

  • Fixed ContentRelevanceFilter deserialization in deep crawl requests (#1642)
  • Fixed ProxyConfig JSON serialization in BrowserConfig.to_dict() (#1629)
  • Fixed .cache folder permissions in Docker image (#1638)

• 🤖 LLM Extraction Improvements:

  • Configurable rate limiter backoff with new LLMConfig parameters (#1269):

    from crawl4ai import LLMConfig
    
    config = LLMConfig(
        provider="openai/gpt-4o-mini",
        backoff_base_delay=5,           # Wait 5s on first retry
        backoff_max_attempts=5,          # Try up to 5 times
        backoff_exponential_factor=3     # Multiply delay by 3 each attempt
    )

  • HTML input format support for LLMExtractionStrategy (#1178):

    from crawl4ai import LLMExtractionStrategy
    
    strategy = LLMExtractionStrategy(
        llm_config=config,
        instruction="Extract table data",
        input_format="html"  # Now supports: "html", "markdown", "fit_markdown"
    )

  • Fixed raw HTML URL variable - extraction strategies now receive "Raw HTML" instead of HTML blob (#1116)

• 🔗 URL Handling:

  • Fixed relative URL resolution after JavaScript redirects (#1268)
  • Fixed import statement formatting in extracted code (#1181)

• 📦 Dependency Updates:

  • Replaced deprecated PyPDF2 with pypdf (#1412)
  • Pydantic v2 ConfigDict compatibility - no more deprecation warnings (#678)

• 🧠 AdaptiveCrawler:

  • Fixed query expansion to actually use LLM instead of hardcoded mock data (#1621)

Full v0.7.8 Release Notes →

• 📊 Real-time Monitoring Dashboard: Interactive web UI with live system metrics and browser pool visibility

  # Access the monitoring dashboard
  # Visit: http://localhost:11235/dashboard
  
  # Real-time metrics include:
  # - System health (CPU, memory, network, uptime)
  # - Active and completed request tracking
  # - Browser pool management (permanent/hot/cold)
  # - Janitor cleanup events
  # - Error monitoring with full context

• 🔌 Comprehensive Monitor API: Complete REST API for programmatic access to all monitoring data

  import httpx
  
  async with httpx.AsyncClient() as client:
      # System health
      health = await client.get("http://localhost:11235/monitor/health")
  
      # Request tracking
      requests = await client.get("http://localhost:11235/monitor/requests")
  
      # Browser pool status
      browsers = await client.get("http://localhost:11235/monitor/browsers")
  
      # Endpoint statistics
      stats = await client.get("http://localhost:11235/monitor/endpoints/stats")

• ⚡ WebSocket Streaming: Real-time updates every 2 seconds for custom dashboards

• 🔥 Smart Browser Pool: 3-tier architecture (permanent/hot/cold) with automatic promotion and cleanup

• 🧹 Janitor System: Automatic resource management with event logging

• 🎮 Control Actions: Manual browser management (kill, restart, cleanup) via API

• 📈 Production Metrics: 6 critical metrics for operational excellence with Prometheus integration

• 🐛 Critical Bug Fixes:

  • Fixed async LLM extraction blocking issue (#1055)
  • Enhanced DFS deep crawl strategy (#1607)
  • Fixed sitemap parsing in AsyncUrlSeeder (#1598)
  • Resolved browser viewport configuration (#1495)
  • Fixed CDP timing with exponential backoff (#1528)
  • Security update for pyOpenSSL (>=25.3.0)

Full v0.7.7 Release Notes →

• 🔧 Docker Hooks System: Complete pipeline customization with user-provided Python functions at 8 key points

• ✨ Function-Based Hooks API (NEW): Write hooks as regular Python functions with full IDE support:

  from crawl4ai import hooks_to_string
  from crawl4ai.docker_client import Crawl4aiDockerClient
  
  # Define hooks as regular Python functions
  async def on_page_context_created(page, context, **kwargs):
      """Block images to speed up crawling"""
      await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
      await page.set_viewport_size({"width": 1920, "height": 1080})
      return page
  
  async def before_goto(page, context, url, **kwargs):
      """Add custom headers"""
      await page.set_extra_http_headers({'X-Crawl4AI': 'v0.7.5'})
      return page
  
  # Option 1: Use hooks_to_string() utility for REST API
  hooks_code = hooks_to_string({
      "on_page_context_created": on_page_context_created,
      "before_goto": before_goto
  })
  
  # Option 2: Docker client with automatic conversion (Recommended)
  client = Crawl4aiDockerClient(base_url="http://localhost:11235")
  results = await client.crawl(
      urls=["https://httpbin.org/html"],
      hooks={
          "on_page_context_created": on_page_context_created,
          "before_goto": before_goto
      }
  )
  # ✓ Full IDE support, type checking, and reusability!

• 🤖 Enhanced LLM Integration: Custom providers with temperature control and base_url configuration

• 🔒 HTTPS Preservation: Secure internal link handling with preserve_https_for_internal_links=True

• 🐍 Python 3.10+ Support: Modern language features and enhanced performance

• 🛠️ Bug Fixes: Resolved multiple community-reported issues including URL processing, JWT authentication, and proxy configuration

Full v0.7.5 Release Notes →

• 🚀 LLMTableExtraction: Revolutionary table extraction with intelligent chunking for massive tables:

  from crawl4ai import LLMTableExtraction, LLMConfig
  
  # Configure intelligent table extraction
  table_strategy = LLMTableExtraction(
      llm_config=LLMConfig(provider="openai/gpt-4.1-mini"),
      enable_chunking=True,           # Handle massive tables
      chunk_token_threshold=5000,     # Smart chunking threshold
      overlap_threshold=100,          # Maintain context between chunks
      extraction_type="structured"    # Get structured data output
  )
  
  config = CrawlerRunConfig(table_extraction_strategy=table_strategy)
  result = await crawler.arun("https://complex-tables-site.com", config=config)
  
  # Tables are automatically chunked, processed, and merged
  for table in result.tables:
      print(f"Extracted table: {len(table['data'])} rows")

• ⚡ Dispatcher Bug Fix: Fixed sequential processing bottleneck in arun_many for fast-completing tasks

• 🧹 Memory Management Refactor: Consolidated memory utilities into main utils module for cleaner architecture

• 🔧 Browser Manager Fixes: Resolved race conditions in concurrent page creation with thread-safe locking

• 🔗 Advanced URL Processing: Better handling of raw:// URLs and base tag link resolution

• 🛡️ Enhanced Proxy Support: Flexible proxy configuration supporting both dict and string formats

Full v0.7.4 Release Notes →

• 🕵️ Undetected Browser Support: Bypass sophisticated bot detection systems:

  from crawl4ai import AsyncWebCrawler, BrowserConfig
  
  browser_config = BrowserConfig(
      browser_type="undetected",  # Use undetected Chrome
      headless=True,              # Can run headless with stealth
      extra_args=[
          "--disable-blink-features=AutomationControlled",
          "--disable-web-security"
      ]
  )
  
  async with AsyncWebCrawler(config=browser_config) as crawler:
      result = await crawler.arun("https://protected-site.com")
  # Successfully bypass Cloudflare, Akamai, and custom bot detection

• 🎨 Multi-URL Configuration: Different strategies for different URL patterns in one batch:

  from crawl4ai import CrawlerRunConfig, MatchMode
  
  configs = [
      # Documentation sites - aggressive caching
      CrawlerRunConfig(
          url_matcher=["*docs*", "*documentation*"],
          cache_mode="write",
          markdown_generator_options={"include_links": True}
      ),
      
      # News/blog sites - fresh content
      CrawlerRunConfig(
          url_matcher=lambda url: 'blog' in url or 'news' in url,
          cache_mode="bypass"
      ),
      
      # Fallback for everything else
      CrawlerRunConfig()
  ]
  
  results = await crawler.arun_many(urls, config=configs)
  # Each URL gets the perfect configuration automatically

• 🧠 Memory Monitoring: Track and optimize memory usage during crawling:

  from crawl4ai.memory_utils import MemoryMonitor
  
  monitor = MemoryMonitor()
  monitor.start_monitoring()
  
  results = await crawler.arun_many(large_url_list)
  
  report = monitor.get_report()
  print(f"Peak memory: {report['peak_mb']:.1f} MB")
  print(f"Efficiency: {report['efficiency']:.1f}%")
  # Get optimization recommendations

• 📊 Enhanced Table Extraction: Direct DataFrame conversion from web tables:

  result = await crawler.arun("https://site-with-tables.com")
  
  # New way - direct table access
  if result.tables:
      import pandas as pd
      for table in result.tables:
          df = pd.DataFrame(table['data'])
          print(f"Table: {df.shape[0]} rows × {df.shape[1]} columns")

• 💰 GitHub Sponsors: 4-tier sponsorship system for project sustainability

• 🐳 Docker LLM Flexibility: Configure providers via environment variables

Full v0.7.3 Release Notes →

• 🧠 Adaptive Crawling: Your crawler now learns and adapts to website patterns automatically:

  config = AdaptiveConfig(
      confidence_threshold=0.7, # Min confidence to stop crawling
      max_depth=5, # Maximum crawl depth
      max_pages=20, # Maximum number of pages to crawl
      strategy="statistical"
  )
  
  async with AsyncWebCrawler() as crawler:
      adaptive_crawler = AdaptiveCrawler(crawler, config)
      state = await adaptive_crawler.digest(
          start_url="https://news.example.com",
          query="latest news content"
      )
  # Crawler learns patterns and improves extraction over time

• 🌊 Virtual Scroll Support: Complete content extraction from infinite scroll pages:

  scroll_config = VirtualScrollConfig(
      container_selector="[data-testid='feed']",
      scroll_count=20,
      scroll_by="container_height",
      wait_after_scroll=1.0
  )
  
  result = await crawler.arun(url, config=CrawlerRunConfig(
      virtual_scroll_config=scroll_config
  ))

• 🔗 Intelligent Link Analysis: 3-layer scoring system for smart link prioritization:

  link_config = LinkPreviewConfig(
      query="machine learning tutorials",
      score_threshold=0.3,
      concurrent_requests=10
  )
  
  result = await crawler.arun(url, config=CrawlerRunConfig(
      link_preview_config=link_config,
      score_links=True
  ))
  # Links ranked by relevance and quality

• 🎣 Async URL Seeder: Discover thousands of URLs in seconds:

  seeder = AsyncUrlSeeder(SeedingConfig(
      source="sitemap+cc",
      pattern="*/blog/*",
      query="python tutorials",
      score_threshold=0.4
  ))
  
  urls = await seeder.discover("https://example.com")

• ⚡ Performance Boost: Up to 3x faster with optimized resource handling and memory efficiency

Read the full details in our 0.7.0 Release Notes or check the CHANGELOG.

Our version numbers follow this pattern: MAJOR.MINOR.PATCH (e.g., 0.4.3)

We use different suffixes to indicate development stages:

• dev (0.4.3dev1): Development versions, unstable
• a (0.4.3a1): Alpha releases, experimental features
• b (0.4.3b1): Beta releases, feature complete but needs testing
• rc (0.4.3): Release candidates, potential final version

• Regular installation (stable version):

  pip install -U crawl4ai

• Install pre-release versions:

  pip install crawl4ai --pre

• Install specific version:

  pip install crawl4ai==0.4.3b1

We use pre-releases to:

• Test new features in real-world scenarios
• Gather feedback before final releases
• Ensure stability for production users
• Allow early adopters to try new features

For production environments, we recommend using the stable version. For testing new features, you can opt-in to pre-releases using the --pre flag.

• [x] 0. Graph Crawler: Smart website traversal using graph search algorithms for comprehensive nested page extraction
• [x] 1. Question-Based Crawler: Natural language driven web discovery and content extraction
• [x] 2. Knowledge-Optimal Crawler: Smart crawling that maximizes knowledge while minimizing data extraction
• [x] 3. Agentic Crawler: Autonomous system for complex multi-step crawling operations
• [x] 4. Automated Schema Generator: Convert natural language to extraction schemas
• [x] 5. Domain-Specific Scrapers: Pre-configured extractors for common platforms (academic, e-commerce)
• [x] 6. Web Embedding Index: Semantic search infrastructure for crawled content
• [x] 7. Interactive Playground: Web UI for testing, comparing strategies with AI assistance
• [x] 8. Performance Monitor: Real-time insights into crawler operations
• [ ] 9. Cloud Integration: One-click deployment solutions across cloud providers
• [x] 10. Sponsorship Program: Structured support system with tiered benefits
• [ ] 11. Educational Content: "How to Crawl" video series and interactive tutorials

Theme	Badge
Disco Theme (Animated)
Night Theme (Dark with Neon)
Dark Theme (Classic)
Light Theme (Classic)

HTML code for adding the badges:

<!-- Disco Theme (Animated) -->
<a href="https://github.com/unclecode/crawl4ai">
  <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-disco.svg" alt="Powered by Crawl4AI" width="200"/>
</a>

<!-- Night Theme (Dark with Neon) -->
<a href="https://github.com/unclecode/crawl4ai">
  <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-night.svg" alt="Powered by Crawl4AI" width="200"/>
</a>

<!-- Dark Theme (Classic) -->
<a href="https://github.com/unclecode/crawl4ai">
  <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-dark.svg" alt="Powered by Crawl4AI" width="200"/>
</a>

<!-- Light Theme (Classic) -->
<a href="https://github.com/unclecode/crawl4ai">
  <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-light.svg" alt="Powered by Crawl4AI" width="200"/>
</a>

<!-- Simple Shield Badge -->
<a href="https://github.com/unclecode/crawl4ai">
  <img src="https://img.shields.io/badge/Powered%20by-Crawl4AI-blue?style=flat-square" alt="Powered by Crawl4AI"/>
</a>

• Data Capitalization: Transform digital footprints into measurable, valuable assets.
• Authentic AI Data: Provide AI systems with real human insights.
• Shared Economy: Create a fair data marketplace that benefits data creators.

1. Open-Source Tools: Community-driven platforms for transparent data extraction.
2. Digital Asset Structuring: Tools to organize and value digital knowledge.
3. Ethical Data Marketplace: A secure, fair platform for exchanging structured data.

For more details, see our full mission statement.