# Fastcrawl

> Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown — with
> crawl, map, search, extract, screenshots, PDF and scheduled monitors included. JavaScript rendering
> built in. One API key works across both REST and MCP (15 tools). Free tier: 1,500 credits/month, no card.

## Product
- [Homepage](/): live demo, pricing, endpoints
- [Dashboard](/app): signup, API keys, billing
- [Comparisons](/vs/): fastcrawl vs SerpApi, ScrapingAnt, Oxylabs, Scrapingdog, Zyte, Nimbleway, Bright Data, Decodo, Apify, Crawl4AI, ScrapingBee, Firecrawl, Jina, Tavily, Exa
- [Free tools](/tools/): HTML to Markdown converter, link extractor, word counter, sitemap checker, robots.txt checker, meta tag checker, email extractor, email verifier, heading outline, token calculator
- [Benchmark](/benchmark/): measured scrape-speed benchmark (curl vs fastcrawl) — median fresh-render latency, cache-hit latency, methodology
- [Token calculator](/token-calculator/): estimate and cut LLM token cost

## Getting started (agent)
- [Auth](/auth.md): how to authenticate (Bearer PAT via API keys page, or Google sign-in)
- [Pricing](/pricing.md): Free / Go ($5/mo) / Enterprise plan limits
- [Agent onboarding SKILL](/agent-onboarding/SKILL.md): agent-facing guide with examples
- [MCP](/mcp): the same API over Model Context Protocol (15 tools, one key)
- [CLI](/docs/cli/): `fastcrawl-cli` — URL to clean markdown from your terminal (zero-dep, free via guest /try)

## API reference
- Endpoint: `https://fastcrawl.net/api/v1/` (REST) · `https://mcp.fastcrawl.net/mcp` (MCP)
- Auth: `Authorization: Bearer <spd_...>` (from https://fastcrawl.net/app)
- Methods: `scrape`, `crawl`, `map`, `search`, `extract`, `batch/extract` (up to 25 URLs, one schema), `screenshot`, `pdf`, `image` (HTML+CSS → hosted PNG), `parse`, `batch/scrape` (up to 50 URLs), `email/verify` (up to 10 addresses), monitors CRUD
- Image rendering: `POST /api/v1/image` (route `/image` under that base) with `{html}` (required, max 1MB) plus optional `css`, `viewport_width`, `viewport_height`, `google_fonts` → `{url}` serving `image/png`. 1 credit per image; an identical repeat within 30 minutes returns `cached: true` and is free. Markup only — for a URL use `screenshot`.
- Batch extract: `POST /batch/extract` with `{urls: [1-25], prompt, schema, maxAge?}` scrapes every URL (10-wide, same pipeline and cache as `/scrape`) and runs one prompt + schema over each (5 LLM calls wide) in ONE call — one round-trip instead of 25. Returns `{success, total, succeeded, failed, results}` with one entry per input URL, in input order: `{url, success, data}` or `{url, success: false, error, error_code, retryable}` — a dead URL is that row's problem and the batch still returns 200. `from_cache: true` on a repeated call means that page cost 0 credits. Pricing: 1 credit per page, cache hits free. Up to 25 URLs (not 50: every page is an LLM call) and the whole request is bounded at ~2 minutes. MCP equivalent: `extract_many` (same body, one credit per page).
- Search + content bundle: `POST /search` with `{query, limit, include_content: 1-5}` returns each result's page markdown in the same call — `content` (markdown) on the first N results, `content: null` + `content_error` (`target_not_found`, `scrape_empty_content`, `target_blocked`, `budget exceeded`, …) when a page fails, and the request still returns 200 (60s whole-request budget). Pricing: results 1 credit; each content page 1 credit, cache hits free. REST only — the MCP `search` tool stays 1 credit per call.
- Completion webhook: `POST /crawl` (and the MCP `crawl` tool) accepts an optional `callback_url` — when the job reaches `succeeded`/`failed` we POST `{event: "crawl.completed", job_id, status, status_url, pages}` to it. Delivery is free (billing unchanged: one credit per crawled page); retried up to 4× (immediately, ~1s, ~4s, ~15s). Every delivery is signed: `X-Fastcrawl-Signature: t=<unix_ts>,v1=<hex hmac-sha256(secret, "<ts>.<body>")>`. Signed with YOUR account's webhook secret: read it at Dashboard → Webhooks or `GET /api/v1/webhooks/secret` (created on first read; `POST /api/v1/webhooks/secret/rotate` replaces it). URLs must be http/https on a public host — loopback, RFC1918, link-local and CGNAT targets are rejected with 400. Monitor `webhook_url` payloads carry the same signature header.
- Scrape formats: `markdown`, `html`, `links`, `json`, `changeTracking` (diff vs your previous scrape: metadata.change_information = {change_status, diff})
- PDF URLs auto-route: scraping a .pdf returns parser=pdf-inspector markdown, no extra call
- Email verification: `POST /email/verify` with `{email}` or `{emails:[≤10]}` → `status` is `valid` (deliverable) | `invalid` (do not send) | `risky` (catch-all domain: the mailbox may or may not exist) | `unknown` (unproven — the probe IP is rate-limited by large providers, so retry later; never treat `unknown` as invalid). Also exposes `gibberish`, `disposable`, `webmail`, `mx_records`, `smtp_check`, `accept_all`, `score`. 1 credit per address; MCP tool: `verify_email`.
- Errors are business results: 422 + `error_code` (target_dns_error | target_not_found | target_blocked | target_rate_limited | target_timeout | target_server_error | scrape_engine_error | scrape_antibot_error | scrape_empty_content | unknown_format ...) + `retryable` flag — retry only when true. Failed requests are never charged. 401 carries error_code + remediation hint + WWW-Authenticate header.
- Cache: `maxAge` seconds (default 172800 = 48h, 0=force fresh). Cache hits are free.
- A2A agent card (no auth): `/.well-known/agent-card.json`

## Guides & tutorials
- [Web scraping best practices for AI training data](/blog/scraping-for-ai-training-data/)
- [Building a price monitoring system with AI agents](/blog/price-monitoring-with-ai-agents/)
- [Anti-bot detection: what sites do and how modern scrapers handle it](/blog/anti-bot-detection/)
- [Verify an email address before you send](/blog/verify-email-before-you-send/)
- [Structured data extraction from e-commerce product pages](/blog/ecommerce-product-page-extraction/)
- [Building a real-time data pipeline with a scraping API](/blog/realtime-data-pipeline/)
- [Crawling etiquette: robots.txt, rate limits, and being a good citizen](/blog/crawling-etiquette/)
- [Why scraping API bills are unpredictable](/blog/why-scraping-api-bills-are-unpredictable/)
- [Screenshot APIs for visual regression and archiving](/blog/screenshot-api-visual-regression/)
- [How to handle pagination in web scraping](/blog/pagination-in-web-scraping/)
- [Web scraping for market research](/blog/scraping-for-market-research/)
- [SEO monitoring with a scraping API](/blog/seo-monitoring-with-a-scraping-api/)
- [Scheduled monitors vs polling](/blog/scheduled-monitors-vs-polling/)
- [API rate limits and credit budgets](/blog/plan-your-scraping-budget/)
- [Extracting structured data from PDFs](/blog/extract-data-from-pdfs/)
- [Scraping job boards at scale](/blog/scraping-job-boards-at-scale/)
- [Bright Data vs Fastcrawl](/blog/brightdata-vs-fastcrawl/)
- [Decodo vs Fastcrawl](/blog/decodo-vs-fastcrawl/)
- [Apify vs Fastcrawl](/blog/apify-vs-fastcrawl/)
- [ScrapingBee vs Fastcrawl](/blog/scrapingbee-vs-fastcrawl/)
- [What is a web scraping API?](/blog/what-is-a-web-scraping-api/)
- [How does web scraping work?](/blog/how-does-web-scraping-work/)
- [How to convert HTML to Markdown](/blog/html-to-markdown/)
- [Crawling vs scraping](/blog/crawling-vs-scraping/)
- [Firecrawl vs Jina Reader vs Fastcrawl](/blog/firecrawl-vs-jina-vs-fastcrawl/)
- [How to scrape JavaScript-heavy sites](/blog/scrape-javascript-heavy-sites/)
- [LLM-ready markdown](/blog/llm-ready-markdown/)
- [Cut LLM token costs](/blog/cut-llm-token-costs/)
- [Monitoring websites with AI agents](/blog/monitoring-websites-with-ai-agents/)
- [HTTP-only vs browser rendering](/blog/http-only-vs-browser-rendering/)
- [Connect an MCP server to your agent](/blog/connect-mcp-to-your-agent/)

## Notes
- Related: `.mcp.json` server config, `/docs` reference, `/.well-known/mcp/server-card.json`