Crawl4AI
Open-source crawler and scraper that turns web pages into LLM-ready Markdown, with a self-hosted API server.
Quick answer
Crawl4AI is an open-source crawler and scraper that turns web pages into LLM-ready Markdown, with a self-hosted API server, listed under Web scraping for AI. A typical small-business use: turn your help center, docs or product pages into Markdown for a RAG knowledge base. Official hardware: minimum 4 GB RAM. That fits a Hetzner Cloud CPX22 server at about €19.99/month (server only, excl. VAT). License: Apache-2.0, which allows internal business use.
Last updated · facts verified against official docs on · sources
Activity data comes from GitHub and was last refreshed Oct 29, 2026.
What is Crawl4AI?
Crawl4AI is a Python library and a Dockerized API server that crawls pages with a headless browser and returns clean Markdown, structured JSON (CSS or LLM-based extraction), screenshots or PDFs. The self-hosted server adds a playground, a monitoring dashboard, a job queue with webhooks and an MCP endpoint, so agents and workflows can call it directly. LLM extraction uses the provider keys you supply.
Where it fits in a 10–200 person business
- Turn your help center, docs or product pages into Markdown for a RAG knowledge base
- Give an internal agent or n8n workflow a private web-reading API instead of a paid scraping service
- Pull structured data (prices, specs, listings) from partner or supplier sites on a schedule
These use cases are our editorial suggestions, not claims made by the project.
Can a business use Crawl4AI? (license)
| License | Apache-2.0 |
|---|---|
| Type | Permissive: MIT / Apache-2.0: use, modify and sell with few obligations. |
| Internal business use | Allowed under the license text |
| What to watch | Free for internal and commercial use, modification and redistribution. Keep the license and NOTICE text and state significant changes when you redistribute. |
This is a plain-English summary of the license text, not legal advice. Third-party models you run with it have their own licenses.
Data & compliance: does Crawl4AI phone home?
| Telemetry (usage data sent to the vendor) | Not documented |
|---|---|
| Other calls home by default (updates, marketplaces, default cloud services) | Not documented |
| Single sign-on (OIDC / SAML / LDAP) | Not documented |
| Audit log | Not documented |
| Role-based access control (RBAC) | Not documented |
| Data residency / GDPR notes from the project | Not documented |
Checked against the project's own docs and repository on . "Not documented" means we could not find the answer in official sources; it does not mean the feature is missing or that no data is sent. Third-party services you connect (model APIs, vector databases, search engines) have their own data policies. Compare all apps.
What hardware does Crawl4AI need?
| Minimum | 4 GB RAM |
|---|---|
| Recommended | Not published |
The Docker guide asks for at least 4 GB of RAM available for the container, "more recommended for heavy use". No CPU or disk figure is published.
How much does it cost to host Crawl4AI per month?
| Minimum spec | 4 GB RAM → CPX22 (2 vCPU, 4 GB, 80 GB disk) €19.99/mo |
|---|
How we got this: we took the cheapest Hetzner Cloud Regular Performance (shared AMD vCPU) plan with at least the RAM and vCPU the project lists, using prices read from www.hetzner.com/cloud/regular-performance on 2026-10-25. "From" monthly price cap shown by Hetzner, excluding VAT. Prices vary by location. Covers the server only; it excludes LLM API usage, backups, extra storage, GPUs and your time. Disk is not matched; add a volume if you need more than the plan includes.
How to deploy Crawl4AI
Method: Docker (pre-built image)
export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)"
docker run -d \
-p 11235:11235 \
--name crawl4ai \
--shm-size=1g \
-e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \
unclecode/crawl4ai:latest
# API on http://localhost:11235, playground at /playgroundFrom the official instructions: docs.crawl4ai.com/core/self-hosting/. Always check the source for the current version before you run it.
Known gotchas when self-hosting Crawl4AI
- Since 0.9.0 the server binds to loopback inside the container unless CRAWL4AI_API_TOKEN is set, so the published port answers with "connection reset" while the container still reports healthy. Pass the token explicitly with -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN". source
- Deep crawling over the API is disabled by default (requests with deep_crawl_strategy get HTTP 400). Start the container with CRAWL4AI_ALLOW_DEEP_CRAWL=true to allow it. source
- Upgrading from 0.8.x changes the API (token required, stricter request validation, declarative hooks instead of inline Python). Read the migration guide before upgrading an existing install. source
- The INSTALL_TYPE=all build adds torch and transformers for advanced extraction strategies and makes the image significantly larger; only build it if you need those strategies. source
Alternatives in Web scraping for AI
- Firecrawl API that turns websites into clean Markdown or structured data for LLMs. Compare Crawl4AI vs Firecrawl
Crawl4AI FAQ
How much RAM does Crawl4AI need?
The official minimum is 4 GB RAM. The Docker guide asks for at least 4 GB of RAM available for the container, "more recommended for heavy use". No CPU or disk figure is published.
How much does it cost to self-host Crawl4AI?
The software is free to download. Based on the official minimum 4 GB RAM: That fits a Hetzner Cloud CPX22 server at about €19.99/month (server only, excl. VAT). That covers the server only, not LLM API usage, backups, extra storage or your time.
Is Crawl4AI free for commercial use?
Crawl4AI is licensed under Apache-2.0 (Permissive). Internal business use is allowed under the license text. Free for internal and commercial use, modification and redistribution. Keep the license and NOTICE text and state significant changes when you redistribute. This is a summary, not legal advice.
What are common problems when self-hosting Crawl4AI?
Since 0.9.0 the server binds to loopback inside the container unless CRAWL4AI_API_TOKEN is set, so the published port answers with "connection reset" while the container still reports healthy. Pass the token explicitly with -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN". Deep crawling over the API is disabled by default (requests with deep_crawl_strategy get HTTP 400). Start the container with CRAWL4AI_ALLOW_DEEP_CRAWL=true to allow it. Upgrading from 0.8.x changes the API (token required, stricter request validation, declarative hooks instead of inline Python). Read the migration guide before upgrading an existing install. The INSTALL_TYPE=all build adds torch and transformers for advanced extraction strategies and makes the image significantly larger; only build it if you need those strategies.
Is Crawl4AI still maintained?
Yes, it is actively developed. Last commit: Oct 26, 2026. Latest release: v0.9.5 (Oct 14, 2026). Checked Oct 29, 2026.
How do I install Crawl4AI?
The official method is Docker (pre-built image). The exact commands are in the "How to deploy" section above and in the official instructions: https://docs.crawl4ai.com/core/self-hosting/
What are self-hosted alternatives to Crawl4AI?
Other apps in Web scraping for AI: Firecrawl.
Sources
- GitHub repository: github.com/unclecode/crawl4ai
- Official documentation, Hardware requirements, Deploy instructions, Gotcha 1, Gotcha 3: docs.crawl4ai.com/core/self-hosting/
- License text: github.com/unclecode/crawl4ai/blob/main/LICENSE
- Gotcha 2, Gotcha 4: github.com/unclecode/crawl4ai/blob/main/deploy/docker/README.md
- Hetzner Cloud pricing (cost estimate): www.hetzner.com/cloud/regular-performance
- GitHub stars, commits and releases: GitHub REST API, fetched
Facts on this page were checked against the project's own repository and docs on 2026-10-29. Spotted something out of date? Tell us.