Crawler access
Are the AI search bots, the on-demand fetchers and the training bots allowed or blocked, and was that your choice?
See
Which AI crawlers visit, which pages they read and what your server answers each one.
Beta list open
See what AI does with your website.
LLMReady shows which AI crawlers visit, what they get back and whether they send you customers. Then it serves them a clean, always-current version of every page.
The free tools run in your browser. Nothing you type there is sent anywhere. Try the free tools.
HTTP 403 blocked by bot rule<div id="root"></div><script src="/static/main.4f2a.js"></script>Loading...Accept all cookies Reject ManageNo llms.txt No markdown No structured data
AI readiness
31
Pricing
Acme Supply sells reusable packaging to small food businesses.
Plans
- Starter, $19 a month, one location
- Growth, $59 a month, up to five locations
- Custom, volume pricing and invoicing
llms.txtmarkdownjson-ld
AI readiness
92
Dashboard·acme-supply.example//last 30 days
01 / 06·The loop//See, fix, serve, prove
One loop for a new kind of visitor.
AI crawlers read your site all day. LLMReady closes the loop between what they ask for and what they get.
01
02
Fix
Eight checks, each with the exact change to make, written for the person who has to make it.
03
Serve
Clean agent pages and an llms.txt that stay current on every deploy.
04
Prove
A weekly report and a log of every bot decision you can show a client or a lawyer.
02 / 06·Checks//Eight things AI needs
Eight things decide whether AI can use your pages.
Most sites fail on two or three of these without knowing. The scan checks all eight and explains what to change.
llms.txt file
A short guide to your best pages, written for language models. LLMReady checks that it exists, loads and links to live pages.
Sitemap health
Is every important page listed, current and reachable, with no dead or redirected URLs.
Structured data
Organization, Product, Article and FAQ markup that tells machines what a page is about without guessing.
Content without scripts
Many crawlers do not run JavaScript. LLMReady compares what loads with and without it and flags text that goes missing.
Headings and structure
Clear headings, one topic per page and answers placed near the question, so a model can lift them cleanly.
Markdown versions
A plain markdown copy of each key page is the easiest thing for a model to read. LLMReady tests whether yours exist.
Firewall and bot rules
Security tools often block AI crawlers by accident. LLMReady requests your pages as each bot and reports what comes back.
llms.txt is a proposed standard and not every AI company reads it yet. LLMReady tells you which fixes are widely supported and which are early bets, so you spend time where it counts.
03 / 06·Product//The part a chatbot cannot do
The part a chatbot cannot do for you.
An AI model can write your llms.txt in a minute. It cannot see who crawls your server, keep hundreds of pages in sync on every deploy or show which AI answers send you customers. LLMReady does that every day.
01·analytics
AI traffic analytics
Connect your CDN or send server logs. See every AI crawler, the pages it fetched, what it got back and which AI answers sent visitors. Most of this never shows up in Google Analytics, because crawlers do not run scripts.
- Every request classified by bot and purpose: search, user fetch or training
- Served, refused and failed requests per bot and per page
- Visits from ChatGPT, Perplexity, Gemini and Claude answers
0AI requests
0Refused or failed
0Visits from AI
| Bot | Requests | Served |
|---|---|---|
| GPTBot | 18,402 | 92% |
| ClaudeBot | 9,115 | 90% |
| PerplexityBot | 6,220 | 38% |
| OAI-SearchBot | 5,480 | 98% |
PerplexityBot was refused 62 percent of the time by a firewall rule. Fixing it is the largest single gain on this site.
02·agent pages
Agent pages
When a crawler asks for a page, it gets a clean, structured, current version served from the edge. Human visitors see your normal site. The machine version rebuilds on every deploy, so it never goes stale.
- Markdown and structured data for every key page
- llms.txt written and refreshed from your sitemap and top pages
- Works with any platform, with no changes to your pages
curl -A GPTBot https://acme-supply.example/pricingHTTP/2 200 content-type: text/markdown# Pricing> Acme Supply sells reusable packaging to small food businesses. ## Plans- Starter, $19 a month, one location- Growth, $59 a month, up to five locations llms.txt found Markdown found JSON-LD found
03·policy and proof
Bot policy and proof
Allow, block or limit each bot, with identity checks so a fake user agent cannot walk through. Keep a log you can show a client, a manager or a lawyer.
- Per-bot rules for search, user and training bots
- Verification against each vendor's published IP ranges
- A dated log of every decision
policy
OAI-SearchBotSearch. Finds pages for ChatGPT search.
ChatGPT-UserUser fetch. Opens a page when a person asks.
GPTBotTraining. Collects content that may train models.
Unknown botFails identity check against vendor IP ranges.
Log: 4 decisions today, 0 overrides.
04 / 06·How it works//Scan, connect, measure
Scan, connect, measure.
Setup takes minutes and needs no changes to your pages.
STEP 01
Scan
The scan fetches your pages as each major AI crawler would and records what it receives, what it is refused and what is missing.
STEP 02
Connect
Add LLMReady at your CDN or edge, or send logs. It works with any platform. Every AI request is classified the same day.
STEP 03
Serve and measure
Crawlers get agent pages that stay current. You get a weekly report on crawl success, refused requests and the visits and sales AI answers send back.
05 / 06·Free tools//Run in your browser
Two free tools you can use right now.
Build your llms.txt file and decide which AI crawlers may visit. Both run in your browser.
06 / 06·Pricing//Free to check, paid to run
Free to check. Paid to run.
The scan and the generators are free for everyone. You pay for the ongoing system. Plans grow with the AI requests and pages handled. Prices are subject to change before launch.
Free
$0
- One scan a month for one site
- llms.txt and crawler rule generators
- Plain-language report
Starter
$49 a month
- One site
- AI traffic analytics up to 250,000 requests
- Agent pages for up to 500 pages
- Alerts when crawlers get blocked
Most chosen
Growth
$199 a month
- Five sites
- Up to 2 million requests and 5,000 pages
- Bot identity checks and per-bot policy
- AI referral and sales attribution
- Weekly report for clients or managers
Scale
From $799 a month
- Many sites under one contract
- Log export, API and single sign-on
- Compliance log of every bot decision
- Support with a response time guarantee
FAQ
Questions people ask.
01If AI can write llms.txt for free, why pay?
You can write the files yourself, and the free tools here help. What you cannot get from a chatbot is your own crawler data, pages that stay in sync on every deploy, bot identity checks and proof of what reached AI answers. The paid plans cover that ongoing work.
02Will this get my site cited by ChatGPT or Claude?
Nobody can guarantee a citation. LLMReady removes the technical reasons a crawler might skip or misread your pages. Whether a model cites you still depends on your content and how useful it is.
03What is llms.txt?
A plain text file at the root of your site that lists your most useful pages with short descriptions, written so a language model can find the right page quickly. It is a proposal, and not every AI system reads it yet.
04Should I block AI training crawlers?
That is your call. Many sites allow AI search and block training. The crawler rules tool sets each bot separately and shows what every line does.
05Do you store my content or logs?
The free tools run entirely in your browser. For the paid platform, request records and scan results are kept for your account only, and page content is never published or resold.
06Does it work with any website?
Yes. The scan looks at what your server returns, so it works for any platform. The paid layer connects through your CDN, edge or logs.
Beta·Early access
Join the beta and see what AI does with your site.
Leave your email and you will hear when your spot is ready.