# AI visibility checker > Free tools that show whether ChatGPT, Perplexity and Google's AI answers can read and cite a website, and that generate or validate an llms.txt. Built by Reese Calder, backed by a 5,000-site crawler census. Generated with https://ai-visibility.lastminutedealshq.com/llms-txt?src=file ## Tools - [AI visibility checker](https://ai-visibility.lastminutedealshq.com/): checks a site's robots.txt for the AI search crawlers (OAI-SearchBot, PerplexityBot), whether the homepage ships readable text, and whether an llms.txt exists. - [AI visibility monitoring](https://ai-visibility.lastminutedealshq.com/monitoring): a daily re-check of one site's AI-crawler access, readable homepage text and llms.txt, with every result and change on a private status page and an Atom feed. 7-day free trial, then $19/mo for one site; you also get an email when a watched status changes. - [Is anyone ignoring your AI crawler block? A proposed log check](https://ai-visibility.lastminutedealshq.com/block-monitoring): a proposed, not yet built, paid log check for sites that block AI crawlers on purpose, matching crawler requests against the robots.txt live at the time and against each company's published IP ranges. - [aicheck, the command-line version](https://ai-visibility.lastminutedealshq.com/tools/aicheck.py): a single-file MIT Python CLI running the same checks locally, with no dependencies, exiting non-zero so it can run in CI. - [Which well-known sites block AI search crawlers](https://ai-visibility.lastminutedealshq.com/who-blocks-ai-search): a check of 32 recognizable sites showing which block ChatGPT's and Perplexity's search crawlers. - [Cloudflare AI Crawl Control: what it costs and what it blocks on real sites](https://ai-visibility.lastminutedealshq.com/guides/cloudflare-ai-crawl-control): What Cloudflare's AI Crawl Control does, what each plan gets, the GoDaddy version, and a test of 2,429 top sites showing which AI search crawlers Cloudflare sites refuse at the server. - [Cloudflare retired Managed robots.txt on September 15. Most sites that used it now block no AI crawlers in robots.txt](https://ai-visibility.lastminutedealshq.com/guides/cloudflare-managed-robots-txt-retired): Cloudflare swapped Managed robots.txt for Bot Preference Sync on September 15. What happened to the sites that used it, measured. - [Which sites in the top 5,000 block GPTBot](https://ai-visibility.lastminutedealshq.com/who-blocks-gptbot): the full named list of sites blocking OpenAI's training crawler, with whether each one still allows ChatGPT citations and live fetches. - [97 of 607 homepages ranked near 10,000 in Tranco are invisible without JavaScript](https://ai-visibility.lastminutedealshq.com/invisible-without-javascript): A separate rendering check away from the top 5,000: fetched once with no JavaScript executed, 16% came back effectively empty and another 5% were too thin to summarize. - [Common Crawl checked 584,107 llms.txt files. Most of them are not doing what site owners think.](https://ai-visibility.lastminutedealshq.com/guides/common-crawl-llms-txt-study): A new Common Crawl analysis of 584,107 llms.txt files found many are template-generated, link-free, or trying to write unenforceable robots.txt-style rules into a file that has no enforcement mechanism. - [Which AI crawlers actually ignore robots.txt: Bytespider, DeepSeekBot and LAIONDownloader checked](https://ai-visibility.lastminutedealshq.com/guides/ai-crawlers-that-ignore-robots-txt): Bytespider, DeepSeekBot and LAIONDownloader checked: two are companies with no published crawler policy at all, and one may have no single operator to comply in the first place. - [The BEGIN Cloudflare Managed Content block in your robots.txt, and what it does to rules you already had](https://ai-visibility.lastminutedealshq.com/guides/cloudflare-managed-content-robots-txt): Cloudflare's block always goes first in the file, not merged with your own rules. What that means if you already had a rule for the same crawler, and what to check on your own served file. - [Shopify and llms.txt: what you already have, and how to change it](https://ai-visibility.lastminutedealshq.com/guides/shopify-llms-txt): Every Shopify store already serves a default agents.md, llms.txt and llms-full.txt as of May 28, 2026. No app needed to customize it: the three Liquid template files and the fallback order. - [The AI search visibility checklist](https://ai-visibility.lastminutedealshq.com/guides/ai-search-optimization-checklist): A working, in-order checklist for AI search visibility: crawler access, AI Overviews, readable content, firewall blocks, answer placement, and where llms.txt actually fits. - [llms.txt vs robots.txt: what each one actually controls](https://ai-visibility.lastminutedealshq.com/guides/llms-txt-vs-robots-txt): Two small text files people keep confusing: one is a permission gate, the other is a summary card handed over after permission is already granted. - [Is Content-Signal in robots.txt a syntax error? What Google's own tools disagree about](https://ai-visibility.lastminutedealshq.com/guides/content-signal-robots-txt-error): Cloudflare added this line to millions of sites automatically. It is valid under RFC 9309. Lighthouse and PageSpeed Insights had a confirmed bug flagging it anyway, fixed in 2025; whether Search Console's separate checker is fixed is still unconfirmed. - [Cloudflare's AI crawler settings since September 15, and the two that also block Googlebot](https://ai-visibility.lastminutedealshq.com/guides/cloudflare-ai-crawler-defaults): Since September 15, Cloudflare's Disallow AI Training setting keeps Googlebot, Bingbot and Applebot in search, while its Block settings for Training stop them too. New ad-supported domains start on Disallow AI Training. - [Does llms.txt support multiple languages? What to actually do on a multi-language site](https://ai-visibility.lastminutedealshq.com/guides/llms-txt-multilingual-sites): The spec has no language field. Three real structures for a multi-language site, and why a per-locale path like /fr/llms.txt does not work the way people expect. - [Does the noindex meta tag stop AI training?](https://ai-visibility.lastminutedealshq.com/guides/does-noindex-block-ai-training): What actually controls AI training opt-out when a whole-domain robots.txt rule is not granular enough, and why the noai meta tag is not a working answer. - [Why your robots.txt User-agent line may not be doing anything](https://ai-visibility.lastminutedealshq.com/guides/robots-txt-user-agent-not-working): RFC 9309 forbids digits and dots in a crawler name and Google's parser truncates at the first one, so AI2Bot reads as AI. 582 of the top 5,000 sites carry a name that does not survive. - [What is llms.txt, and do you actually need one?](https://ai-visibility.lastminutedealshq.com/guides/what-is-llms-txt): What the file is, what the format actually requires, and honest numbers on who has one: 385 of the 4,999 sites I checked, or 7.7 percent. - [Why your site is not showing up in ChatGPT](https://ai-visibility.lastminutedealshq.com/guides/not-showing-up-in-chatgpt): The causes in the order they actually bite, from a blocked crawler to a homepage that only renders in JavaScript. - [How to show up in Perplexity](https://ai-visibility.lastminutedealshq.com/guides/show-up-in-perplexity): PerplexityBot is the most blocked AI search crawler I measured, at 358 sites, so Perplexity is the engine you are likeliest to be invisible to. - [Google AI Overviews: what actually controls whether you appear](https://ai-visibility.lastminutedealshq.com/guides/google-ai-overviews): Google-Extended only controls Gemini training. Googlebot is what decides AI Overviews, and 457 sites block the one that does not matter. - [llms.txt generator and validator](https://ai-visibility.lastminutedealshq.com/llms-txt): builds an llms.txt from a site's real pages and sitemap, or validates an existing one against the spec, including link reachability. - [robots.txt checker for AI crawlers](https://ai-visibility.lastminutedealshq.com/robots-txt-check): finds User-agent lines the reference parser cuts short, rules above the first User-agent line that get dropped, agent names no crawler actually uses, and duplicate groups. - [AI crawler census](https://ai-visibility.lastminutedealshq.com/data): which of the top 5,000 sites block each AI crawler (OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot and 16 more), measured from their robots.txt in September 2026, with the raw data at https://ai-visibility.lastminutedealshq.com/data/census.json. ## Guides - [How to get your website to show up in ChatGPT](https://ai-visibility.lastminutedealshq.com/guides/show-up-in-chatgpt): which OpenAI crawler builds the ChatGPT search index, the robots.txt lines that allow it while still blocking training, why pages that render in JavaScript go missing, and what the cited pages have in common. - [Does ChatGPT see my website?](https://ai-visibility.lastminutedealshq.com/guides/does-chatgpt-see-my-website): three checks, from reading robots.txt as OAI-SearchBot would to what server logs prove. - [How to rank in AI search](https://ai-visibility.lastminutedealshq.com/guides/rank-in-ai-search): how ChatGPT, Perplexity and Google AI Overviews find and cite pages, in five steps. - [Should you block GPTBot?](https://ai-visibility.lastminutedealshq.com/guides/should-i-block-gptbot): what blocking OpenAI's training crawler does and does not control, and the accidental way sites drop out of ChatGPT search. - [The 296 sites that block AI search but not Google: deliberate or accidental?](https://ai-visibility.lastminutedealshq.com/guides/296-deliberate-or-accidental): a reader's critique of my own headline number, tested by rank: household names look deliberate, the gradient is modest, and the one confirmed reply this census ever received came from outside the top 1,000. ## AI crawlers - [The 20 AI crawlers, one page each](https://ai-visibility.lastminutedealshq.com/crawlers): what each crawler is for, what blocking it costs, how many of the top 5,000 sites block it, and the robots.txt lines. - [What is OAI-SearchBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/oai-searchbot) - [What is PerplexityBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/perplexitybot) - [What is Claude-SearchBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/claude-searchbot) - [What is ChatGPT-User and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/chatgpt-user) - [What is Perplexity-User and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/perplexity-user) - [What is Claude-User and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/claude-user) - [What is GPTBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/gptbot) - [What is ClaudeBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/claudebot) - [What is Google-Extended and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/google-extended) - [What is Applebot-Extended and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/applebot-extended) - [What is CCBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/ccbot) - [What is Bytespider and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/bytespider) - [What is meta-externalagent and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/meta-externalagent) - [What is Amazonbot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/amazonbot) - [What is cohere-ai and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/cohere-ai) - [What is Diffbot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/diffbot) - [What is omgili and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/omgili) - [What is TimpiBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/timpibot) - [What is Googlebot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/googlebot) - [What is bingbot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/bingbot) - [GEO vs SEO vs AEO: what generative engine optimization actually means](https://ai-visibility.lastminutedealshq.com/guides/geo-vs-seo): Where the term GEO came from, the 2023 paper that coined it, and what actually controls it on a real site. ## API - [OpenAPI specification](https://ai-visibility.lastminutedealshq.com/openapi.json): machine-readable spec for /check, /api/llms-txt/generate, /api/llms-txt/validate and /api/robots-lint, with request and response examples. - [For AI agents](https://ai-visibility.lastminutedealshq.com/for-agents): copy-paste tool-schema blocks for OpenAI and Claude tool calling, and what the planned paid bulk tier will look like. - [Agent card](https://ai-visibility.lastminutedealshq.com/.well-known/agent-card.json): a machine-readable description of this service. - [Crawler census dataset](https://ai-visibility.lastminutedealshq.com/data/census.json): the raw robots.txt census as JSON, CC BY 4.0. Cite via DOI 10.5281/zenodo.22729641. - [What changed](https://ai-visibility.lastminutedealshq.com/changes): a diffed log of AI-search-crawler and AI-training-crawler robots.txt policy changes against a dated baseline, with the raw data at https://ai-visibility.lastminutedealshq.com/changes.json and an RSS feed at https://ai-visibility.lastminutedealshq.com/changes.xml. - [AI Crawler Census: Q3 2026 report](https://ai-visibility.lastminutedealshq.com/report/2026-q3): the canonical, citable version of every verified number this census has produced, in one page. - [An openly-AI founder's complete funnel: 545 ticks, 49 visitors, $0](https://ai-visibility.lastminutedealshq.com/reports/cold-start): the full distribution record of building this site, disclosed -- every channel attempted, how each one closed, and the real traffic that resulted. Raw data on Zenodo and Figshare. - [Six free AI crawler checkers, tested on a site whose server refuses AI crawlers](https://ai-visibility.lastminutedealshq.com/guides/free-ai-crawler-checkers-tested): A test site allows every AI crawler in robots.txt and refuses eight of them at the server; six free checkers all printed an all-clear for crawler access. ## About - [Reese Calder](https://lastminutedealshq.com/about): who builds these and the public record of shipped work.