# Octocrawl > Octocrawl turns a public web page into readable Markdown and, on supported pages, fields you can check against the source. It reports blocks, timeouts and missing fields with a reason instead of inventing content. Connect an agent to hosted Octocrawl at https://mcp.octocrawl.dev/mcp (MCP) or call https://api.octocrawl.dev (REST): scrape and map, keyless within a daily allowance; an account adds 1,000 credits a month, crawl jobs of up to 500 pages, batch jobs of up to 1,000 URLs, and with a key the browser lane. The published packages run everything on your own computer, without credits or a daily allowance, and crawls without the 500-page cap (npx octocrawl serve: REST, a TypeScript SDK, a Python client or MCP, with crawl, batch and Monitor), and npx octocrawl serve --hosted --token serves other machines. It is open source (AGPL-3.0). Connect MCP: claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp (other clients: https://octocrawl.dev/docs/connect-mcp/index.md). Try one public page in the browser at https://octocrawl.dev/ (five previews a day). The source code is at https://github.com/77777R7/Octocrawl. ## Get started - [Introduction](https://octocrawl.dev/docs/index.md): Use Octocrawl to extract web pages as Markdown and structured data through REST, SDKs or MCP: hosted scrape, map, crawl and batch jobs, or run it locally. - [Connect MCP](https://octocrawl.dev/docs/connect-mcp/index.md): Connect Octocrawl MCP to Claude Code, Cursor, OpenCode or Codex: hosted scrape and map, crawl and batch jobs with an account, or a local API for Monitors. - [Scrape a page](https://octocrawl.dev/docs/scrape/index.md): Scrape a public web page with the Octocrawl API: Markdown, links, tables and JSON fields, with its status and an Evidence Record of where each value came from. ## Guides - [Amazon.sg product JSON](https://octocrawl.dev/docs/guides/amazon-product/index.md): Turn an Amazon.sg /dp/ page into product JSON with Octocrawl (Beta): ASIN, title, price, currency, seller and delivery region, each checked or marked missing. - [Monitor to HTTPS Webhook](https://octocrawl.dev/docs/guides/monitor-webhook/index.md): Watch a public documentation page with a local Octocrawl Monitor, keep its baseline and run history, and deliver each change as an event to an HTTPS webhook. - [AI SDK tools](https://octocrawl.dev/docs/integrations/ai-sdk/index.md): Give an AI SDK model Octocrawl's scrape and map tools: web pages as Markdown with an Evidence Record, hosted with no key. Install, sample output and options. ## Reference - [Limits and result states](https://octocrawl.dev/docs/limits/index.md): Hosted Octocrawl's daily limits with and without a key, the sites it supports, and what each result means: success, blocked, failed, timeout or incomplete. - [Advanced reference](https://octocrawl.dev/docs/reference/index.md): Reference for Octocrawl's REST API, TypeScript SDK and Python client: scrape, map, batch and crawl routes, JSON extraction, error codes and the Evidence Record. ## Project - [Privacy](https://octocrawl.dev/docs/privacy/index.md): What the Octocrawl page and its preview service record about a visit and a preview, what they never record, how long each record is kept, and who sees it. - [Terms of use](https://octocrawl.dev/docs/terms/index.md): The terms for the free public Octocrawl preview: what it is and is not, its daily limits, results provided as they are, who runs it and how to reach them. - [Acceptable use](https://octocrawl.dev/docs/acceptable-use/index.md): What the hosted Octocrawl preview, API and MCP may not be used for, such as private pages or getting around logins and CAPTCHAs, and how to report misuse. - [Contact](https://octocrawl.dev/docs/contact/index.md): How to reach the Octocrawl operator: public GitHub issues for questions, bugs and ideas, private email for privacy, security, site owners and misuse reports. ## Blog - [Hosted Crawl and Batch Jobs for AI Agents](https://octocrawl.dev/blog/hosted-crawl-and-batch-jobs/index.md): Octocrawl now runs crawls of up to 500 pages and batches of up to 1,000 URLs on its servers, as clean Markdown from any MCP client, on your account's credits. - [Firecrawl to Octocrawl: Test the v1 Migration Boundary](https://octocrawl.dev/blog/firecrawl-v1-migration/index.md): Test the Firecrawl Node SDK against local Octocrawl: v1 paths, supported options, refusal cases and SDK error differences, with exact reproduction steps. - [Why HTTP 200 Is Not Enough to Accept a Scraped Page](https://octocrawl.dev/blog/http-200-task-completion/index.md): Use four reproducible fixtures to separate HTTP success, task verification and readiness, then decide which scraped pages can enter a RAG pipeline. - [Claude Code Web Scraping: Add a Web Tool With MCP](https://octocrawl.dev/blog/claude-code-web-scraping/index.md): Give Claude Code a web scraping tool in one command: it can map a site, read pages as Markdown and cite the source URL and fetch time of every answer. - [Web Scraping for RAG: Markdown Chunks That Keep Sources](https://octocrawl.dev/blog/web-scraping-for-rag/index.md): Scrape web pages into clean Markdown for RAG, split them into chunks that keep the source URL, fetch time and a text hash, and skip pages that did not change. - [How to Find All Pages on a Website: 6 Ways (Free)](https://octocrawl.dev/blog/find-all-pages-on-a-website/index.md): Find all pages on a website six ways: sitemap, site: search, Search Console, a crawler, a one-call map and web archives, then check the list is complete. - [Scrape a Website With Login Without Sharing Your Password](https://octocrawl.dev/blog/scrape-website-with-login/index.md): Read pages that need your login in the Chrome you already use. Your password and cookies stay in your browser, and you pass any check yourself, not a bot. - [Extract Structured Data From a Web Page, Without AI](https://octocrawl.dev/blog/extract-structured-data-from-web-page/index.md): Pull fields like price and availability from a web page into JSON, each with the place on the page it was read from, and a reason when a field is missing. - [How to Scrape a List of URLs and Keep Every Failure](https://octocrawl.dev/blog/scrape-list-of-urls/index.md): Scrape a list of up to 1,000 URLs into Markdown files and a CSV, with one row per URL, the reason for every failure and the evidence behind each page. ## Optional - [All documentation in one file](https://octocrawl.dev/llms-full.txt) - [Changelog](https://octocrawl.dev/changelog/index.md): what changed, version by version