free_tool
AI Crawler & llms.txt Auditor
Enter a URL and get a graded read on how ready your site is for AI engines. It checks which AI crawlers your robots.txt allows, whether you ship an llms.txt, whether your content is readable without JavaScript, and the structured data and sitemap that help you get cited. Plain checks, with what to fix first.
The site is fetched server-side as plain HTTP, the way most AI crawlers read it. A site that builds its content with client-side JavaScript may look emptier here than it does in a browser. That gap is itself a finding worth knowing about.
why_it_matters
AI answers are the new front page
More people now meet a brand through an AI answer than through a blue link. If an AI crawler is blocked in your robots.txt, or your content only appears after JavaScript runs, those engines cannot read you and cannot cite you.
An llms.txt and clean structured data give AI tools a curated, machine-readable map of what you do. This grades all of it in one pass so you can see where you are invisible.
faq
Questions & answers
- What is an llms.txt file and do I need one?
- llms.txt is a simple markdown file at the root of your site that gives AI tools a curated map of your most important pages. It is not required, but it makes your content easier for AI engines to find and summarize correctly. This tool checks whether you have one and whether it follows the format.
- How do I let ChatGPT, Claude and Perplexity read my site?
- Each runs a named crawler (GPTBot, ClaudeBot, PerplexityBot and others). If your robots.txt disallows them, they cannot read or cite your pages. This tool reads your robots.txt and tells you, bot by bot, which AI engines are currently allowed or blocked.
- Why does my content need to work without JavaScript?
- Most AI fetchers do not run JavaScript. If your main content only appears after the page hydrates in a browser, those crawlers see an almost empty page. This tool estimates how much real text is in the HTML as served, which is what they actually read.
- Does blocking AI crawlers help or hurt me?
- It depends on your goal. Blocking can protect content you do not want used for training, but it also removes you from AI answers and citations that increasingly drive traffic. This tool shows you exactly what your current rules do so the choice is deliberate, not accidental.
Want to actually get cited by AI, not just audited?
This grades the basics. I go deeper: server-rendering the content that matters, getting crawl rules right, and shipping the llms.txt and structured data that earn citations. Book a call, or leave your email.
Prefer proof first? See how this plays out in real case studies →