# URLpipe

> URLpipe is a per-URL web data API: POST a URL and get back clean Markdown, rendered HTML, a full-page screenshot, page metadata, an AI summary, keywords, a Lighthouse audit or the JavaScript console output — or several of them from one page visit. Pages are rendered in real Chrome by our own rendering engine, so JavaScript-heavy sites work. One bearer token, no SDK needed — official clients for Python, JavaScript, Ruby and Go if you want one — and an MCP server for AI agents.

URLpipe is for developers who would otherwise write and maintain one-off scrapers: feeding web pages to LLMs and RAG pipelines, giving AI agents a browser, link previews, and performance or error checks on sites you don't control. It does not crawl whole sites or discover URLs — it answers questions about the URL you give it.

Authentication: `Authorization: Bearer YOUR_API_KEY` (a per-project key) on every request. Each request is a POST with a JSON body containing `url`. Requests are asynchronous by default — you get a token back and collect the result at your webhook or with GET /result/:token — and `sync: true` returns the result inline.

## Endpoints

- [POST /html](https://urlpipe.dev/docs/html): Rendered HTML of a page (JavaScript executed, redirects followed). 1 credit.
- [POST /markdown](https://urlpipe.dev/docs/markdown): The page's main content as clean Markdown, converted deterministically from the rendered DOM — no model. 1 credit.
- [POST /screenshot](https://urlpipe.dev/docs/screenshot): Full-page screenshot as PNG, JPEG or WebP; viewport, retina scale, one element, dark mode, hidden elements and custom CSS via `screenshot_options`. 1 credit.
- [POST /pdf](https://urlpipe.dev/docs/pdf): The whole rendered page as a PDF: paper size, orientation, margins, scale, print or screen stylesheet, or one long page via `pdf_options`. 1 credit.
- [POST /meta](https://urlpipe.dev/docs/meta): Everything a page declares about itself — title, description, site name, language, authors, dates, share image, icons, feeds and the raw Open Graph and Twitter tags — as one object with the same keys on every page. 1 credit.
- [POST /summarize](https://urlpipe.dev/docs/summarize): A concise AI summary of the page's main content, in Markdown. 17 credits.
- [POST /keywords](https://urlpipe.dev/docs/keywords): 5–15 keywords and phrases, ranked by relevance, in the page's language. 15 credits.
- [POST /lighthouse](https://urlpipe.dev/docs/lighthouse): A Google Lighthouse audit (mobile or desktop): four category scores and the lab metrics. 2 credits.
- [POST /console](https://urlpipe.dev/docs/console): console.error and console.warn output, uncaught exceptions and unhandled promise rejections during page load. 1 credit.
- [POST /technologies](https://urlpipe.dev/docs/technologies): What a site is built with — CMS, frameworks, analytics, CDN, web server — with versions and a confidence. 1 credit.
- [POST /scrape](https://urlpipe.dev/docs/scrape): Any subset of the operations above from a single page visit. Costs the sum of the operations it runs.
- [GET /result/:token](https://urlpipe.dev/docs/results): Collect a result by its token, for 30 days.

## MCP server

URLpipe is also a remote MCP server at https://urlpipe.dev/mcp (streamable HTTP, `Authorization: Bearer YOUR_TOKEN` with an organization token — nothing to install). Its tools are the endpoints above, with the same arguments, responses and credits, plus tools to list projects, read usage and retrieve past results. Tokens can be read-only, limited to one project, or set to expire. Setup: https://urlpipe.dev/docs/mcp

## Client libraries

Optional: every endpoint is one POST. The official clients wait for the result (polling past the 60-second sync window), retry with an Idempotency-Key so a retry never bills twice, type every documented error and verify webhook signatures. https://urlpipe.dev/docs/client-libraries

- Python: `pip install urlpipe` — https://github.com/URLpipe/urlpipe-python
- JavaScript: `npm install @urlpipe/sdk` — https://github.com/URLpipe/urlpipe-js
- Ruby: `gem install urlpipe` — https://github.com/URLpipe/urlpipe-ruby
- Go: `go get github.com/URLpipe/urlpipe-go` — https://github.com/URLpipe/urlpipe-go
- LangChain: `pip install langchain-urlpipe` — six agent tools, a toolkit and a document loader. https://urlpipe.dev/integrations/langchain
- LlamaIndex: `pip install llama-index-readers-urlpipe` — a reader that loads pages as Markdown Documents. https://urlpipe.dev/integrations/llamaindex
- n8n: the community node `n8n-nodes-urlpipe` (Settings → Community Nodes) — every operation as a node, usable as an AI Agent tool. https://urlpipe.dev/integrations/n8n
- Command line: `brew install urlpipe/tap/urlpipe` (or Scoop on Windows, or `go install github.com/URLpipe/urlpipe-go/cmd/urlpipe@latest`) — `urlpipe markdown <url>`, `urlpipe screenshot <url>`, and CI checks such as `urlpipe lighthouse <url> --min-score 90`.

## Features

- Caching: repeated requests for the same URL and operation within `max_age` (default 7 days, up to 30) are served from cache and spend no credits; a repeat that arrives while the first is still running waits for it, also free. https://urlpipe.dev/docs/caching
- Never pay twice: `Idempotency-Key` makes a retried request return the original answer. https://urlpipe.dev/docs/retries
- Async by default: webhooks retried with backoff and re-sendable from the dashboard, HMAC-SHA256 signed (`X-URLpipe-Signature`) once signing is switched on for the project; or poll GET /result/:token. https://urlpipe.dev/docs/async
- Page options: wait for an element or a delay, block ads, remove cookie banners and any elements you name — removed from every result, not just hidden. https://urlpipe.dev/docs/page-options
- Documents: a URL that serves a PDF or a text, JSON, XML or feed file is read like a page by /markdown, /meta, /summarize, /keywords and /screenshot, at the same price. https://urlpipe.dev/docs/documents
- Labels: tag requests with up to 16 of your own ids (client, project, job); they come back with results and webhooks, and usage is broken down by label. https://urlpipe.dev/docs/labels
- Residential exits: `residential: true` fetches from a home broadband address, +25 credits per page visit, off by default. https://urlpipe.dev/docs/residential
- robots.txt: every project follows the robots.txt of the sites it fetches by default; a customer can turn that off per project, and is then responsible for having the right to fetch those pages. https://urlpipe.dev/docs/robots-txt
- EU: pages are fetched, rendered and stored in the EU on every plan, and AI processing can be switched to EU-only per organization at no charge. https://urlpipe.dev/docs/data-residency
- Failed requests spend nothing: target errors, bot checks that couldn't be cleared, robots.txt refusals. https://urlpipe.dev/docs/credits

## Pricing

Sold in credits. Per call: /console 1, /html 1, /markdown 1, /meta 1, /pdf 1, /screenshot 1, /technologies 1, /lighthouse 2, /keywords 15, /summarize 17, /extract 24; /scrape is the sum of its operations; `residential: true` adds 25 per page visit. Cache hits and failures are free.

- Free: 1,000 credits/month, 1 in parallel, one user, $0, no card.
- Starter: 20,000 credits/month, 3 in parallel, unlimited team members, $19/month or $190/year.
- Pro: 55,000 credits/month, 8 in parallel, unlimited team members, $49/month or $490/year.
- Scale: 175,000 credits/month, 20 in parallel, unlimited team members, $149/month or $1,490/year.

Paid plans are never cut off: past the allowance, extra credits cost $1.50 per 1,000 credits, invoiced after the month closes. The Free plan is refused (HTTP 429) until the 1st. Prices are in USD before VAT. Full detail: https://urlpipe.dev/pricing.md

## Documentation

- [Introduction](https://urlpipe.dev/docs): URLpipe API docs: turn any URL into clean, structured data
- [Quickstart](https://urlpipe.dev/docs/quickstart): URLpipe Quickstart: your first API request in minutes
- [Authentication](https://urlpipe.dev/docs/authentication): Authentication: API keys & bearer tokens — URLpipe docs
- [Client libraries](https://urlpipe.dev/docs/client-libraries): Client libraries for Python, JavaScript, Ruby & Go — URLpipe
- [MCP server](https://urlpipe.dev/docs/mcp): MCP server: connect an AI agent to the URLpipe API
- [Caching & freshness](https://urlpipe.dev/docs/caching): Caching & freshness: max_age and free cache hits — URLpipe
- [Async & sync modes](https://urlpipe.dev/docs/async): Async & sync modes: webhooks or inline — URLpipe API docs
- [Retries & duplicates](https://urlpipe.dev/docs/retries): Retries & idempotency keys: never pay twice — URLpipe
- [Retrieving results](https://urlpipe.dev/docs/results): Retrieving async results by token — URLpipe API docs
- [Page options](https://urlpipe.dev/docs/page-options): Page options: block ads, cookie banners, wait — URLpipe
- [Documents & files](https://urlpipe.dev/docs/documents): PDFs and text files as URLs — URLpipe docs
- [Labels](https://urlpipe.dev/docs/labels): Labels: tag requests with your own ids — URLpipe docs
- [Residential exits](https://urlpipe.dev/docs/residential): Residential proxy exits: fetch from a home IP — URLpipe
- [robots.txt](https://urlpipe.dev/docs/robots-txt): robots.txt: follow or ignore crawl rules — URLpipe docs
- [Credits & rate limits](https://urlpipe.dev/docs/credits): Credits & rate limits: what each call costs — URLpipe docs
- [Data residency](https://urlpipe.dev/docs/data-residency): EU data residency: keep requests in Europe — URLpipe
- [Response headers](https://urlpipe.dev/docs/response-headers): Response headers: cache status, timing & quota — URLpipe
- [Errors](https://urlpipe.dev/docs/errors): API error responses & HTTP status codes — URLpipe docs
- [HTML](https://urlpipe.dev/docs/html): HTML endpoint: fetch a page's rendered HTML — URLpipe API
- [Screenshot](https://urlpipe.dev/docs/screenshot): Screenshot endpoint: full-page PNG, JPEG, WebP — URLpipe
- [PDF](https://urlpipe.dev/docs/pdf): PDF endpoint: convert any web page to PDF — URLpipe
- [Markdown](https://urlpipe.dev/docs/markdown): Markdown endpoint: convert a web page to Markdown — URLpipe
- [Metadata](https://urlpipe.dev/docs/meta): Metadata endpoint: title, description, author — URLpipe API
- [Summarize](https://urlpipe.dev/docs/summarize): Summarize endpoint: AI summary of any page — URLpipe API
- [Keywords](https://urlpipe.dev/docs/keywords): Keywords endpoint: extract page keywords — URLpipe API
- [Extract](https://urlpipe.dev/docs/extract): Extract endpoint: web data in your own schema — URLpipe
- [Lighthouse](https://urlpipe.dev/docs/lighthouse): Lighthouse endpoint: Google performance audit — URLpipe
- [Console](https://urlpipe.dev/docs/console): Console endpoint: capture JS console logs — URLpipe API
- [Technologies](https://urlpipe.dev/docs/technologies): Technologies endpoint: detect a site's tech stack — URLpipe
- [Scrape](https://urlpipe.dev/docs/scrape): Scrape endpoint: multiple analyses in one call — URLpipe

## Free tools (live demos, no signup)

- [Turn any URL into clean Markdown](https://urlpipe.dev/tools/url-to-markdown): Paste a link and get the page's main content as tidy Markdown — headings, lists, links and code kept, navigation and cookie banners stripped. The format LLMs and RAG pipelines work best with.
- [Screenshot any website from its URL](https://urlpipe.dev/tools/website-screenshot): Paste a link and get a sharp PNG of the page as it opens — JavaScript executed, exactly as a real browser would draw it. Great for previews, thumbnails and visual QA.
- [Run a Lighthouse audit on any URL](https://urlpipe.dev/tools/lighthouse-audit): Paste a link and get a real Google Lighthouse audit — performance, accessibility, best practices and SEO scores, plus Core Web Vitals — measured against the live page.
- [Check any website for JavaScript errors](https://urlpipe.dev/tools/console-errors): Paste a link and see the console errors, warnings and uncaught exceptions logged while the page loads in a real browser — no DevTools, no local setup.
- [Find out what any website is built with](https://urlpipe.dev/tools/technology-checker): Paste a link and see the technologies behind the site — its CMS or shop platform, JavaScript frameworks, analytics and tag managers, CDN, web server and more, with versions where the page shows them.
- [Save any web page as a PDF](https://urlpipe.dev/tools/website-to-pdf): Paste a link and get the whole page as one continuous PDF that looks like the site — rendered in a real browser, so JavaScript, web fonts and lazy-loaded images are all there, with its headings as bookmarks.
- [Check a page's Open Graph tags and metadata](https://urlpipe.dev/tools/metadata-extractor): Paste a link and see everything the page says about itself — title, description, share image, authors, dates, language, icons and feeds — as one structured object.
- [Summarize any web page with AI](https://urlpipe.dev/tools/summarize-url): Paste a link and get a concise AI summary of the page's main content — the substance, without the navigation, ads and boilerplate.
- [Extract structured data from any web page](https://urlpipe.dev/tools/ai-web-scraper): Paste a link and get the page's data as JSON — who it belongs to, what it offers, its prices, its contact email and social profiles — with null wherever the page does not say.
- [Get the keywords from any URL](https://urlpipe.dev/tools/keyword-extractor): Paste a link to any website or article and get the 5–15 keywords and phrases that describe it best, ranked — for SEO briefs, tagging and content analysis.
- [Get the rendered HTML of any URL](https://urlpipe.dev/tools/rendered-html): Paste a link and get the page's HTML after JavaScript has run and redirects have been followed — the DOM a real browser sees, not the empty shell curl returns.
- [Everything about a URL, from one visit](https://urlpipe.dev/tools/analyze-url): Paste a link and get a screenshot of the page, its title, description and share image, the main content as Markdown and the console errors it threw — all from the same page load.
- [See how a link unfurls in Slack, X, LinkedIn and Discord](https://urlpipe.dev/tools/link-preview-generator): Paste a link and see the card each app builds from its title, description and share image — with a screenshot of the page standing in when it has no image of its own.
- [Check a page's Core Web Vitals](https://urlpipe.dev/tools/core-web-vitals-checker): Paste a link and get Largest Contentful Paint, Cumulative Layout Shift and Total Blocking Time from a real Lighthouse run — each marked good, needs improvement or poor.
- [Take a full-page screenshot of any website](https://urlpipe.dev/tools/website-screenshot/full-page): Paste a link and get the entire page in one PNG — every section down to the footer, not just what fits on the screen — with lazy-loaded images loaded before the capture.
- [Screenshot a website the way a phone shows it](https://urlpipe.dev/tools/website-screenshot/mobile): Paste a link and get a full-page capture at a 390 px wide viewport — a common phone width — so you see the site's mobile layout, not its desktop one.
- [Screenshot a website in dark mode](https://urlpipe.dev/tools/website-screenshot/dark-mode): Paste a link and get a full-page capture rendered with the browser set to dark — so a site that ships a dark theme shows it, exactly as a visitor with dark mode on sees it.
- [Screenshot one element of a page by CSS selector](https://urlpipe.dev/tools/website-screenshot/element): Paste a link and a CSS selector and get just that element — a header, a pricing table, a chart — cropped to its own box, rendered in real Chrome.
- [Check whether robots.txt lets a bot fetch a URL](https://urlpipe.dev/tools/robots-txt-checker): Paste a link and see what the site's robots.txt says about that exact path — for URLpipe's user agent and for every crawler that has no group of its own — and which rule decides it.
- [Compare two websites side by side](https://urlpipe.dev/tools/compare-urls): Paste two links and get a screenshot of each with its Lighthouse scores and Core Web Vitals next to the other's — your page against a competitor's, or before against after.

## Guides

- [HTML to Markdown for LLMs](https://urlpipe.dev/guides/html-to-markdown-for-llms): Why Markdown is the right format for language models, what a clean conversion keeps and drops, and how to handle client-rendered pages.
- [Reading a Lighthouse audit](https://urlpipe.dev/guides/how-to-read-a-lighthouse-audit): What the four categories and the Performance score mean, how Core Web Vitals are measured, and how to prioritize fixes.
- [Rendered vs. raw HTML](https://urlpipe.dev/guides/rendered-vs-raw-html): Why modern sites return an empty shell to curl, how client-side rendering works, and how to get the post-JavaScript DOM.
- [Extracting page metadata](https://urlpipe.dev/guides/extract-open-graph-metadata): The three metadata standards, why they conflict in practice, and how to reconcile them into one clean object.
- [Why AI agents read empty pages](https://urlpipe.dev/guides/why-ai-agents-read-empty-pages): The reference fetch server does a plain GET, so client-rendered pages reach your agent empty — measured on three real pages.
- [Web access for Claude and Cursor](https://urlpipe.dev/guides/give-claude-and-cursor-web-access-with-mcp): Exact configuration for Claude Code, Claude Desktop and Cursor, token scopes, first prompts and costs.
- [Keyword extraction methods](https://urlpipe.dev/guides/get-keywords-from-a-url): TF-IDF, RAKE and YAKE vs model-based extraction, a competitor's page as a keyword brief, clustering, and the limits.
- [Open Graph image sizes](https://urlpipe.dev/guides/open-graph-image-sizes): The og:image size and file limits each network documents, checked against its own docs, and the one size that works everywhere.
- [Screenshot a website programmatically](https://urlpipe.dev/guides/screenshot-a-website-programmatically): Puppeteer vs Playwright vs an API, and the full-page gotchas: lazy loading, sticky headers, banners and size limits.
- [Core Web Vitals: lab vs field](https://urlpipe.dev/guides/core-web-vitals-lab-vs-field): LCP, INP and CLS, their thresholds, and why a Lighthouse run and real-user data tell different stories.
- [Find JavaScript errors on any site](https://urlpipe.dev/guides/find-javascript-errors-on-any-site): DevTools, headless Chrome and load-time captures — and what each can and cannot see on a site you don't own.
- [Verify webhook signatures (HMAC)](https://urlpipe.dev/guides/verify-webhook-signatures-hmac): The signed string, the raw body, timestamp tolerance, constant-time compares and rotation — with code in five languages.
- [Chunking web pages for RAG](https://urlpipe.dev/guides/chunking-web-pages-for-rag): Markdown first, heading-aware splitting, chunk size and overlap, metadata, and keeping chunks fresh.
- [Polite scraping: robots.txt and pacing](https://urlpipe.dev/guides/polite-scraping-robots-txt-and-rate-limits): What RFC 9309 says about robots.txt, how to pace requests per host, and how to back off when a site struggles.
- [Idempotency keys explained](https://urlpipe.dev/guides/idempotency-keys-explained): What the header does, how servers implement it, and the mistakes that make a retry charge twice.
- [What llms.txt is](https://urlpipe.dev/guides/what-is-llms-txt): The format section by section, a real example, and an honest look at which tools read it.
- [Cookie banners and ads in scraped pages](https://urlpipe.dev/guides/cookie-banners-and-ads-in-scraped-pages): Why automated visits get every banner and ad, and how to remove them without consenting on anyone's behalf.

## Product

- [Web scraping API](https://urlpipe.dev/web-scraping-api): Scrape any URL in real Chrome and get its Markdown, metadata, screenshot and more from one page visit — robots.txt respected, cache hits free.
- [URL to Markdown API](https://urlpipe.dev/url-to-markdown-api): Turn any web page into clean, LLM-ready Markdown — rendered in a real browser, converted without a model, measured against the strategies other APIs use.
- [MCP server](https://urlpipe.dev/mcp-server): A hosted MCP server with 17 tools: your agent reads rendered pages as Markdown, looks at screenshots and pulls metadata, with one bearer token and nothing to install.
- [Screenshot API](https://urlpipe.dev/screenshot-api): Full-page website screenshots as PNG, JPEG or WebP — any viewport, retina, one element, dark mode, banners removed — at one price whatever the options.
- [Lighthouse API](https://urlpipe.dev/lighthouse-api): Run Google Lighthouse 13 on any URL by API — scores, Core Web Vitals lab metrics and every audit — and combine it with a screenshot and console errors.
- [Metadata API](https://urlpipe.dev/metadata-api): One JSON object per URL, the same keys every time — title, description, image, icons, authors, dates, feeds, language — reconciled from Open Graph, Twitter cards, JSON-LD and HTML.
- [Console API](https://urlpipe.dev/console-api): Load any URL in a real browser and get back the JavaScript errors, warnings, uncaught exceptions and unhandled rejections it produced — no SDK, no access to the site.
- [Rendered HTML API](https://urlpipe.dev/rendered-html-api): Get the fully rendered HTML of any URL — scripts run, redirects followed, late content waited for — from a real browser, for the price of one page fetch.
- [URL to PDF API](https://urlpipe.dev/url-to-pdf-api): Turn any URL into a PDF rendered in real Chrome — A4, Letter or one long page, with selectable text, working links and headings as bookmarks — at one price whatever the options.
- [Website technology lookup API](https://urlpipe.dev/website-technology-lookup-api): Detect the CMS, frameworks, analytics, CDN, payment providers and more behind any URL — with versions, confidence and the signals that gave each away — from over 7,000 fingerprints.
- [Structured data extraction API](https://urlpipe.dev/structured-data-extraction-api): Describe the fields you want — names, types, descriptions, lists of objects — and get them filled from any web page or PDF as typed JSON, with null wherever the page does not say.
- [Keyword extraction API](https://urlpipe.dev/keyword-extraction-api): Get the 5–15 keywords and key phrases that describe any web page or PDF, most relevant first, as a JSON array in the page's own language.
- [URL summarization API](https://urlpipe.dev/url-summarization-api): Get a three-to-five-paragraph summary of any web page or PDF, covering the whole page in its own language, as Markdown — nothing added that the page does not say.

## Use cases

- [Web access for AI agents](https://urlpipe.dev/use-cases/ai-agents): Give an agent a tool that reads any page — JavaScript rendered — as Markdown, sees it as a screenshot, and checks it for errors.
- [Web pages into a RAG pipeline](https://urlpipe.dev/use-cases/rag-ingestion): Turn a list of URLs into clean Markdown chunks with titles and dates — deterministic, measured, and free to re-fetch inside your freshness window.
- [Link previews and unfurls](https://urlpipe.dev/use-cases/link-previews): Build the card a pasted link turns into: title, description and image from /meta, and a screenshot when the page has no image of its own.
- [Monitoring client websites](https://urlpipe.dev/use-cases/client-site-monitoring): For agencies: JavaScript errors, Lighthouse scores and screenshots for every client site, tagged per client so usage splits cleanly for billing.
- [Monitoring pages for changes](https://urlpipe.dev/use-cases/content-monitoring): Detect when a web page's content changes: fetch it fresh as deterministic Markdown, hash it, and diff only when the hash moves.
- [Archiving pages as screenshots](https://urlpipe.dev/use-cases/visual-archiving): Keep a visual record of pages over time: full-page WebP captures, light and dark, with the rendered HTML beside each one.
- [URL features inside a multi-tenant app](https://urlpipe.dev/use-cases/multi-tenant-apps): Build link previews, imports and page reading into your own SaaS: one key per environment, a label per tenant, retries that never bill twice, EU processing.

## Integrations

- [Claude Desktop](https://urlpipe.dev/integrations/claude-desktop): Add URLpipe to Claude Desktop as a custom connector, or through a local bridge, so Claude can read, screenshot and audit any page.
- [Claude Code](https://urlpipe.dev/integrations/claude-code): One command connects Claude Code to URLpipe, so it can read docs pages, screenshot a deployed preview and check it for console errors.
- [Cursor](https://urlpipe.dev/integrations/cursor): Give Cursor's agent a rendered-page reader: add URLpipe to mcp.json and it reads docs, SPAs and preview deploys as Markdown.
- [Windsurf](https://urlpipe.dev/integrations/windsurf): Add URLpipe to Windsurf's mcp_config.json with serverUrl and a header, and Cascade can read, screenshot and audit any page.
- [VS Code](https://urlpipe.dev/integrations/vs-code): Add URLpipe to VS Code's mcp.json with a password prompt for the token, and Copilot's agent mode can read and audit any page.
- [Cline](https://urlpipe.dev/integrations/cline): Connect Cline to URLpipe with a streamableHttp entry in cline_mcp_settings.json, and it can read, render and screenshot any page.
- [Zed](https://urlpipe.dev/integrations/zed): Add URLpipe to Zed's context_servers with a url and an Authorization header, and the Agent Panel can read and inspect any page.
- [n8n](https://urlpipe.dev/integrations/n8n): The URLpipe community node for n8n: Markdown, screenshots, metadata, console errors and Lighthouse in any workflow.
- [Make](https://urlpipe.dev/integrations/make): Call URLpipe from Make's HTTP "Make a request" module, and receive async results with a Custom webhook scenario.
- [Zapier](https://urlpipe.dev/integrations/zapier): Call URLpipe from a Zap with Webhooks by Zapier's Custom Request, and catch async results with a Catch Hook.
- [Pipedream](https://urlpipe.dev/integrations/pipedream): Call URLpipe from a Pipedream Node.js step with fetch, and use an HTTP trigger's endpoint as the webhook for async results.
- [Google Sheets](https://urlpipe.dev/integrations/google-sheets): Pull titles, descriptions and Lighthouse scores into a spreadsheet with a short Apps Script and UrlFetchApp.
- [Airtable](https://urlpipe.dev/integrations/airtable): Enrich Airtable records with page metadata and screenshot links from an automation's Run a script action.
- [LangChain](https://urlpipe.dev/integrations/langchain): langchain-urlpipe: six agent tools that read any page, JavaScript included, and a document loader for RAG.
- [LlamaIndex](https://urlpipe.dev/integrations/llamaindex): llama-index-readers-urlpipe: a reader that loads any page, JavaScript included, as a Markdown Document ready to index.

## Code recipes

- [Code recipes in Python](https://urlpipe.dev/code/python): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in Node.js](https://urlpipe.dev/code/node): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in PHP](https://urlpipe.dev/code/php): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in Ruby](https://urlpipe.dev/code/ruby): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in Go](https://urlpipe.dev/code/go): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in Java](https://urlpipe.dev/code/java): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in C#](https://urlpipe.dev/code/csharp): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.
- [Code recipes in cURL](https://urlpipe.dev/code/curl): Screenshots, Markdown, rendered HTML, metadata, Lighthouse and webhook verification.

## Glossary

- [Rendered DOM](https://urlpipe.dev/glossary/rendered-dom): The page as it exists after the browser has run its JavaScript — what a visitor sees, and what curl never returns.
- [Client-side rendering](https://urlpipe.dev/glossary/client-side-rendering): When the browser, not the server, builds the page from JavaScript — which is why curl gets an empty shell.
- [Hydration](https://urlpipe.dev/glossary/hydration): Attaching JavaScript to HTML the server already rendered, so a static page becomes an interactive app.
- [Headless browser](https://urlpipe.dev/glossary/headless-browser): A real browser engine run without a window, driven by code — how you read, screenshot and measure pages at scale.
- [robots.txt](https://urlpipe.dev/glossary/robots-txt): The file at a site's root that tells automated clients which paths they may fetch — a request, not a lock.
- [networkidle](https://urlpipe.dev/glossary/networkidle): The wait condition that treats a page as loaded once its network traffic goes quiet.
- [DOMContentLoaded](https://urlpipe.dev/glossary/domcontentloaded): The event fired once the HTML is parsed and deferred scripts have run — before images, and often before content.
- [Residential proxy](https://urlpipe.dev/glossary/residential-proxy): Sending a request out through a home internet connection's IP address instead of a data center's.
- [Bot detection](https://urlpipe.dev/glossary/bot-detection): How sites decide whether a visitor is a person or a program — and what they do about it.
- [Webhook signature](https://urlpipe.dev/glossary/webhook-signature): A keyed hash sent with each webhook so the receiver can prove who sent it and that nothing changed.
- [Idempotency key](https://urlpipe.dev/glossary/idempotency-key): A unique value sent with a request so retrying it can never do the work — or the charge — twice.
- [Cache TTL](https://urlpipe.dev/glossary/cache-ttl): How long a stored result stays fresh enough to reuse — the knob between cost and freshness.
- [SSRF](https://urlpipe.dev/glossary/ssrf): Tricking a server into fetching a URL on its own internal network — the first risk of any 'fetch this URL' feature.
- [Open Graph](https://urlpipe.dev/glossary/open-graph): The og: meta tags that decide the title, description and image a link preview shows.
- [og:image](https://urlpipe.dev/glossary/og-image): The one tag that decides the picture on every shared link to your page.
- [Twitter Cards](https://urlpipe.dev/glossary/twitter-cards): The twitter: meta tags that shape how a link looks on X — and how they fall back to Open Graph.
- [JSON-LD](https://urlpipe.dev/glossary/json-ld): Structured data in a script tag — the schema.org facts search engines read for rich results.
- [Full-page screenshot](https://urlpipe.dev/glossary/full-page-screenshot): An image of the whole page, top to bottom — not just the part that fits in the window.
- [Device pixel ratio](https://urlpipe.dev/glossary/device-pixel-ratio): How many physical pixels a screen uses for one CSS pixel — why retina screenshots are twice the size.
- [Cookie consent banner](https://urlpipe.dev/glossary/cookie-consent-banner): The consent pop-up European privacy law puts on most sites — and in most screenshots and scraped text.
- [Core Web Vitals](https://urlpipe.dev/glossary/core-web-vitals): Google's three user-experience metrics — loading, responsiveness and visual stability — and their thresholds.
- [Largest Contentful Paint (LCP)](https://urlpipe.dev/glossary/largest-contentful-paint): When the biggest image or text block in the viewport finishes rendering — the loading Core Web Vital.
- [Cumulative Layout Shift (CLS)](https://urlpipe.dev/glossary/cumulative-layout-shift): How much the page's content jumps around unexpectedly while you read it — the visual-stability Core Web Vital.
- [Interaction to Next Paint (INP)](https://urlpipe.dev/glossary/interaction-to-next-paint): How long the page takes to visibly respond to a click, tap or key press — the responsiveness Core Web Vital.
- [Total Blocking Time (TBT)](https://urlpipe.dev/glossary/total-blocking-time): How long long JavaScript tasks block the main thread during load — Lighthouse's heaviest-weighted metric.
- [Speed Index](https://urlpipe.dev/glossary/speed-index): How quickly the visible part of the page fills in during load, averaged over time.
- [Time to First Byte (TTFB)](https://urlpipe.dev/glossary/time-to-first-byte): How long before the server's response starts arriving — the floor under every other loading metric.
- [Unhandled promise rejection](https://urlpipe.dev/glossary/unhandled-promise-rejection): A JavaScript promise that failed with nothing there to catch it — the error that breaks features silently.
- [Keyword extraction](https://urlpipe.dev/glossary/keyword-extraction): Pulling the terms and phrases that say what a page is about out of its text — from any URL.
- [Markdown for LLMs](https://urlpipe.dev/glossary/markdown-for-llms): Converting a web page to Markdown before a model reads it — fewer tokens, same structure.
- [Context window](https://urlpipe.dev/glossary/context-window): How much text a language model can take in at once — the budget every fetched page spends from.
- [Chunking](https://urlpipe.dev/glossary/chunking): Splitting documents into pieces small enough to embed and retrieve — best done along the headings.
- [Embeddings](https://urlpipe.dev/glossary/embeddings): Lists of numbers that place text by meaning, so similar passages can be found without matching words.
- [llms.txt](https://urlpipe.dev/glossary/llms-txt): A Markdown file at a site's root that tells language models what the site is and where its best content lives.
- [Model Context Protocol (MCP)](https://urlpipe.dev/glossary/model-context-protocol): The open protocol that lets AI assistants and agents call external tools — like fetching a web page.
- [Streamable HTTP](https://urlpipe.dev/glossary/streamable-http): The transport remote MCP servers speak: JSON-RPC over plain HTTP POST, with streaming when it's needed.

## Compare

- [URLpipe vs Firecrawl](https://urlpipe.dev/compare/urlpipe-vs-firecrawl): Firecrawl crawls whole sites; URLpipe answers several questions about one URL. Where each wins.
- [URLpipe vs Jina Reader](https://urlpipe.dev/compare/urlpipe-vs-jina-reader): Both turn a URL into Markdown. Jina bills tokens; URLpipe charges a fixed price and does more per URL.
- [Best HTML-to-Markdown APIs](https://urlpipe.dev/best-html-to-markdown-apis): Eight ways to turn a web page into Markdown for a model, with measured output quality and prices.
- [Jina Reader alternatives](https://urlpipe.dev/alternatives/jina-reader): Leaving r.jina.ai? What to look for, the options, measured Markdown quality and how the calls map.
- [Microlink alternatives](https://urlpipe.dev/alternatives/microlink): Looking past Microlink's metadata API? How the endpoints map, what each costs and when to stay.
- [ScreenshotOne alternatives](https://urlpipe.dev/alternatives/screenshotone): When a screenshot API alone isn't enough: options compared, pricing, and how a ScreenshotOne call maps.
- [Firecrawl alternatives](https://urlpipe.dev/alternatives/firecrawl): Leaving Firecrawl? Which alternative fits whether you crawl whole sites or read one URL at a time.
- [Urlbox alternatives](https://urlpipe.dev/alternatives/urlbox): Alternatives to Urlbox for screenshots with Markdown and metadata — with a free tier.
- [PageSpeed Insights API alternatives](https://urlpipe.dev/alternatives/pagespeed-insights-api): When Google's free Lighthouse API isn't enough: async runs, webhooks, and Lighthouse next to the page's errors.
- [ScrapingBee alternatives](https://urlpipe.dev/alternatives/scrapingbee): When you need clean data from a URL more than proxies: alternatives to ScrapingBee compared.
- [Apify alternatives](https://urlpipe.dev/alternatives/apify): When a scraper platform is more than you need: per-URL alternatives to Apify for Markdown and more.
- [OpenGraph.io alternatives](https://urlpipe.dev/alternatives/opengraph-io): Alternatives to OpenGraph.io for Open Graph data and link previews, compared on price and fields.
- [Iframely alternatives](https://urlpipe.dev/alternatives/iframely): When you need a page's data rather than its embed: alternatives to Iframely compared.
- [URLpipe vs Microlink](https://urlpipe.dev/compare/urlpipe-vs-microlink): Two APIs that do several things with one URL. Where Microlink's GET-and-cache model wins, and where URLpipe's does.
- [URLpipe vs ScreenshotOne](https://urlpipe.dev/compare/urlpipe-vs-screenshotone): A screenshot specialist against a per-URL data API: options, pricing and data protection compared.
- [Best screenshot APIs](https://urlpipe.dev/best-screenshot-apis): Six website screenshot APIs compared on options, full-page capture, pricing and what else they return.
- [Best link preview APIs](https://urlpipe.dev/best-link-preview-apis): Five link preview and Open Graph APIs compared on fields, price per lookup and what else they do.
- [Best MCP servers for web browsing](https://urlpipe.dev/best-mcp-servers-for-web-browsing): Eight MCP servers that let an AI agent read the web, compared on JavaScript, hosting and cost.
- [Best Lighthouse APIs](https://urlpipe.dev/best-lighthouse-apis): Five ways to run Lighthouse from code, from Google's free API to monitoring products, compared.

## Research

- [Web-to-Markdown, measured](https://urlpipe.dev/research/markdown-benchmark): Four HTML-to-Markdown strategies on the same 33 real pages: how much of the visible text each keeps, and how much of its output the reader never saw.
- [What stops a headless browser in 2026](https://urlpipe.dev/research/what-stops-a-headless-browser): 46 well-known, frequently protected sites, visited from a data center and a residential address: which refuse outright, which answer 200 with a block page, and what a residential exit changes.
- [Lighthouse variance: how much does one audit move?](https://urlpipe.dev/research/lighthouse-variance): Ten mobile Lighthouse audits of each of five public pages, back to back: the performance score moved by up to 61 points on the same page.

## Engineering notes

- [Why we built our own rendering engine](https://urlpipe.dev/notes/our-own-rendering-engine): One warm Chrome, a fresh browser context per visit, about 900 lines with no dependencies — and the measurements that decided it.
- [One page visit, several answers](https://urlpipe.dev/notes/one-visit-scrape): How /scrape reads several results off one page load and stays identical to calling each endpoint on its own.
- [Pacing a site that can't keep up](https://urlpipe.dev/notes/pacing-slow-sites): When a target site falls over under load, slow down and try again — without failing the request or billing the wait.
- [Why async is the default](https://urlpipe.dev/notes/async-by-default): Pages take seconds to minutes and sometimes much longer; a request that returns a token at once never loses the answer.
- [Billing only what worked](https://urlpipe.dev/notes/billing-only-what-worked): Failures, bot checks, duplicates, cache hits and waits are free — and the harder case, a 200 that is really a block page.
- [Request timing as marks, not spans](https://urlpipe.dev/notes/request-timing-as-marks): Store four moments per request and derive every duration from them — and publish exactly one number.

## Changelog

- [Read the whole of a page that loads as you scroll](https://urlpipe.dev/changelog/scroll-to-bottom): page_options.scroll_to_bottom scrolls a listing or a feed to its end before anything is read, so every result holds all of it.
- [Save any web page as a PDF](https://urlpipe.dev/changelog/pdf-endpoint): POST /pdf prints the whole rendered page to a PDF, on the paper you choose or as one long page, for one credit.
- [See what any site is built with](https://urlpipe.dev/changelog/technologies-endpoint): POST /technologies returns the CMS, frameworks, analytics, CDN and web server behind a page, with versions, for one credit.
- [A page without its newsletter popup](https://urlpipe.dev/changelog/block-popups): page_options.block_popups removes the sign-up prompts, app banners and offers a site lays over its page, from every result.
- [A sign-in form or an error page is a failure, not a result](https://urlpipe.dev/changelog/pages-shown-in-place-of-yours): When a site answers 200 with a block page, a "not found" page or a sign-in form in place of the page you asked for, the request fails unbilled and says which.
- [A URLpipe node for n8n](https://urlpipe.dev/changelog/n8n-node): Install n8n-nodes-urlpipe for Markdown, screenshots, metadata, console errors and Lighthouse in any workflow, or from an AI Agent.
- [Every result names the page it came from, after redirects](https://urlpipe.dev/changelog/final-url): X-Final-Url, and final_url in webhooks, /scrape and /meta, give the address the page ended on once every redirect was followed.
- [Metadata with the same keys on every page, at one credit](https://urlpipe.dev/changelog/metadata-fixed-shape): /meta returns everything a page declares about itself in one fixed shape — authors, dates, image size, icons, feeds and more — for one credit.
- [Bring your team, and keep each client in its own organization](https://urlpipe.dev/changelog/teams-and-organizations): Every paid plan takes as many team members as you need, and one login can belong to several organizations.
- [Official LangChain and LlamaIndex packages](https://urlpipe.dev/changelog/langchain-and-llamaindex-packages): langchain-urlpipe gives agents four tools and a document loader; llama-index-readers-urlpipe loads pages as Documents.
- [The urlpipe command-line tool](https://urlpipe.dev/changelog/urlpipe-cli): Every operation in your terminal and your CI: brew install urlpipe/tap/urlpipe, then urlpipe markdown example.com.
- [Official client libraries for Python, JavaScript, Ruby and Go](https://urlpipe.dev/changelog/client-libraries): pip, npm, gem or go get: typed results and errors, retries that never bill twice, long analyses waited out and webhook verification.
- [A missing or wrong API key answers in JSON](https://urlpipe.dev/changelog/json-authentication-error): The 401 now has the same {error, message} shape as every other error, with the code invalid_api_key.
- [MCP: prompts, a pricing resource, structured results, and two renamed tools](https://urlpipe.dev/changelog/mcp-prompts-and-structured-results): Three ready-made prompts, the price list as a resource, an output schema on every tool — and console_logs and lighthouse_audit renamed.
- [Each project chooses whether to follow robots.txt](https://urlpipe.dev/changelog/robots-txt-per-project): A project setting, on by default: a page the site's robots.txt disallows is not fetched, and the refusal costs nothing.
- [Idempotency-Key, and identical requests share one run](https://urlpipe.dev/changelog/idempotency-key-and-in-flight-sharing): A retried request returns the first one's result and charge; a duplicate that arrives mid-run waits for it for free.
- [Labels: tag requests with your own ids](https://urlpipe.dev/changelog/labels): Attach up to 16 key/value labels to any request, get them back everywhere, and see usage broken down by label.
- [page_options: clean up the page before anything is read](https://urlpipe.dev/changelog/page-options): Block ads, remove cookie banners and your own selectors, or wait for an element — for every result read off the page.
- [screenshot_options: viewport, format, element, dark mode](https://urlpipe.dev/changelog/screenshot-options): PNG, JPEG or WebP, viewport and scale, one element by selector, dark mode, hidden selectors and injected CSS — at one credit.
- [Paid plans, VAT, and paid plans are never cut off](https://urlpipe.dev/changelog/paid-plans-and-overage): Starter, Pro and Scale on sale through Stripe; past the allowance, paid plans carry on and extra credits are billed.
- [Screenshots capture the whole page](https://urlpipe.dev/changelog/full-page-screenshots): Full page by default, with lazy-loaded images loaded first, up to a 16,384 px tall capture.
- [Every screenshot has a link that needs no API key](https://urlpipe.dev/changelog/screenshot-links-without-a-key): X-Result-Url gives a signed link to the image, valid for as long as the result is stored — drop it straight into an image tag.
- [Keep AI processing in the EU, per organization](https://urlpipe.dev/changelog/eu-ai-processing): An organization setting that keeps /meta, /summarize and /keywords inside the European Union, at no extra charge.
- [API keys are stored hashed, and shown once](https://urlpipe.dev/changelog/api-keys-shown-once): Project keys are stored as a digest, displayed only when minted, and can be rotated with a 24-hour overlap.
- [The URLpipe API as an MCP server](https://urlpipe.dev/changelog/mcp-server): Point Claude, Cursor or any MCP client at urlpipe.dev/mcp: 14 tools, the same arguments, results and credits as the API.
- [Fetch a page from a residential address](https://urlpipe.dev/changelog/residential-exits): residential: true routes the page visit through a home internet connection, for sites that answer data center traffic differently.
- [Parallel requests per plan](https://urlpipe.dev/changelog/parallel-requests-per-plan): Each plan runs a set number of requests at once — 1, 3, 8 or 20 — and says so on every response.
- [One credit meter, priced by what each call costs](https://urlpipe.dev/changelog/credits): The two separate AI and web allowances became one monthly credit balance, with each endpoint priced by what it costs to run.
- [/markdown converts without a language model](https://urlpipe.dev/changelog/deterministic-markdown): A DOM walk replaces the model: better coverage, half the stray text, about 20 ms a page, and no model cost.
- [Close your account or delete an organization yourself](https://urlpipe.dev/changelog/account-and-organization-deletion): Account settings closes an account; Organization settings deletes an organization and everything in it.
- [A site that can't keep up is paced, not failed](https://urlpipe.dev/changelog/pacing-slow-sites): When a target site slows down or answers 429/5xx, requests to it wait and retry instead of failing.
- [Filter requests and webhooks in the dashboard](https://urlpipe.dev/changelog/dashboard-filters): Narrow History by duration, URL and outcome, and the Webhooks table by endpoint and delivery time.
- [Send a failed webhook again by hand](https://urlpipe.dev/changelog/resend-a-failed-webhook): A Retry now button on any failed delivery, at any age, so a fixed endpoint can still receive the result.
- [Async requests no longer need a webhook](https://urlpipe.dev/changelog/results-without-a-webhook): report_to is optional: collect async results with GET /result/:token, or set a default webhook per project.
- [Signed webhooks](https://urlpipe.dev/changelog/signed-webhooks): Every delivery can carry an HMAC-SHA256 signature over the timestamp and raw body, with a 24-hour overlap on rotation.
- [Cache, timing and quota on every response](https://urlpipe.dev/changelog/response-headers): X-Result-Token, X-Cache, X-Cache-Age, X-Processing-Time-Ms and the quota headers on every response, and a meta object in webhooks.
- [POST /scrape: several results off one page visit](https://urlpipe.dev/changelog/scrape-endpoint): Ask for any combination of operations on one URL; the page loads once and every result comes back together.
- [Failed requests are never charged](https://urlpipe.dev/changelog/failed-requests-are-free): A timeout, a DNS error or the target's own 4xx/5xx spends nothing — and an error page is no longer returned as a result.
- [Free tools: try every endpoint without signing up](https://urlpipe.dev/changelog/free-tools): A no-signup page for each endpoint that runs it live on a URL you choose and shows the curl that reproduces it.
- [GET /result/:token](https://urlpipe.dev/changelog/get-result-by-token): Fetch any result by its token for 30 days — the way to collect an async result, or a sync one that timed out.
- [Per-project rate limits](https://urlpipe.dev/changelog/per-project-rate-limits): 60 requests a minute and 15 per 10 seconds per project key, with a 429 and Retry-After when you go over.
- [Sign in with Google or GitHub](https://urlpipe.dev/changelog/google-and-github-sign-in): Create an account or sign in with a Google or GitHub account instead of a password.
- [Public API documentation at /docs](https://urlpipe.dev/changelog/api-docs): A docs site with a quickstart, the core concepts and a page per endpoint, open without an account.
- [Request caching with max_age](https://urlpipe.dev/changelog/request-caching): Repeat requests are served from stored results inside a freshness window you choose — free.

## Key pages

- [Homepage](https://urlpipe.dev/)
- [Pricing](https://urlpipe.dev/pricing)
- [Status](https://urlpipe.dev/status): Measured latency and success rate per endpoint.
- [About](https://urlpipe.dev/about)
- [Security](https://urlpipe.dev/security)
- [Terms](https://urlpipe.dev/terms)
- [Privacy](https://urlpipe.dev/privacy)
- [Full documentation as one file](https://urlpipe.dev/llms-full.txt)
- [OpenAPI description](https://urlpipe.dev/openapi.json)

## Contact

- Email: contact@urlpipe.dev
- Provided by Aliat Partner S.L., Barcelona, Spain