TidyTools

Find out how websites treat AI crawlers, and turn web pages into text an AI can read, with TidyTools' free tools. Check any website now for which of 29 AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) its robots.txt allows, with a ready-made fix; look up any of 10,000 popular websites in the daily AI Crawler Index; list the sites that block a crawler; follow which sites changed their rules since yesterday; and get the overall blocking figures by crawler and industry. Also turns any web page or PDF into clean Markdown. No account or API key required.

Actions

Action	What it does
Check which AI crawlers a website allows	Check any website right now: which of 29 AI crawlers, such as ChatGPT's GPTBot, Claude's ClaudeBot and PerplexityBot, its robots.txt file lets in, partly blocks or shuts out, with the exact rule behind each one.
Find websites that block an AI crawler	List the popular websites that block one AI crawler, such as GPTBot or ClaudeBot, most popular first, from today's AI Crawler Index of 10,000 websites.
Get AI crawler blocking statistics	Get today's figures from the AI Crawler Index, a daily check of 10,000 popular websites: how many block each of 29 AI crawlers, as counts and percentages, the share among the top 100 sites, how many have an llms.txt file, the breakdown by industry, and the day-by-day trend for the best-known crawlers.
List websites that changed their AI crawler rules	See which of 10,000 popular websites changed their rules for AI crawlers since the day before: who started or stopped blocking a crawler, changed their overall stance, added or removed an llms.txt file, or changed their Content Signals.
Look up a website in the AI Crawler Index	Look up one of the 10,000 popular websites in today's AI Crawler Index: its industry, how popular it is, its overall stance on AI, which crawlers it allows and blocks, and every change to its rules the index has recorded.
Turn a web page into Markdown	Read a web page or a PDF and get its text back as clean Markdown, ready to hand to an AI step, save as a note or summarise: headings, links and tables kept, menus and page furniture dropped, up to 100,000 characters.

Check which AI crawlers a website allows

Check any website right now: which of 29 AI crawlers, such as ChatGPT's GPTBot, Claude's ClaudeBot and PerplexityBot, its robots.txt file lets in, partly blocks or shuts out, with the exact rule behind each one. Also says whether it has an llms.txt file and Content Signals, scores how visible it is to AI search out of 100, and gives ready-to-paste robots.txt lines that fix any AI search crawlers blocked by mistake.

Inputs

WebsitewebsitetextRequired: The website to check: a domain like example.com or a full link. A link to a page also checks that page's own rules.

Outputs

ai_access_scorenumber: 0 to 100: how open it is to all 29 crawlers.
ai_search_scorenumber: 0 to 100: how open the site is to the crawlers behind AI search and AI assistants. 100 is fully open.
allowed_countnumber
allowed_crawlerslist: Crawlers allowed in everywhere.
blocked_countnumber
blocked_crawlerslist: Crawlers shut out of the site.
content_signalsobject: The site's Cloudflare Content Signals, when it publishes them: 'yes' or 'no' for appearing in search, being used in AI answers and being used to train AI. Empty when the site sets none.
crawlerslist: Every crawler checked, one row each.
has_llms_full_txttrue/false
has_llms_txttrue/false: Whether the site has an llms.txt file for AI.
has_noai_tagtrue/false: Whether the home page carries a 'noai' tag.
has_robots_txttrue/false
llms_txt_urltext
pages_checkedlist: The paths the rules were tested against.
partly_blocked_crawlerslist: Crawlers kept out of some pages only.
policytext: The site's overall stance, in words.
policy_codetext: The same as a short code: open, search-only, blocks-search, restrictive, no-robots or unknown.
robots_txttext: Whether robots.txt was found and readable.
robots_txt_urltext
sitemapslist: Sitemap links listed in robots.txt.
suggested_fixtext: robots.txt lines that let AI search crawlers back in. Empty when none are blocked.
training_only_blocktext: A complete robots.txt that blocks AI training but keeps the site in AI search answers.
unknown_crawlerslist: Crawlers whose access could not be worked out.
verdicttext: One sentence summing up what AI crawlers can do.
websitetext: The site checked.

Example output

{
  "ai_access_score": 33,
  "ai_search_score": 41,
  "allowed_count": 2,
  "allowed_crawlers": [
    "OAI-AdsBot",
    "Google-Agent"
  ],
  "blocked_count": 3,
  "blocked_crawlers": [
    "GPTBot",
    "OAI-SearchBot",
    "ClaudeBot"
  ],
  "content_signals": null,
  "crawlers": [
    {
      "company": "OpenAI",
      "name": "GPTBot",
      "purpose": "Trains AI models",
      "rule": "Disallow: /",
      "status": "blocked"
    }
  ],
  "has_llms_full_txt": false,
  "has_llms_txt": false,
  "has_noai_tag": false,
  "has_robots_txt": true,
  "llms_txt_url": "https://www.nytimes.com/llms.txt",
  "pages_checked": [
    "/"
  ],
  "partly_blocked_crawlers": [
    "Applebot",
    "Amazonbot"
  ],
  "policy": "Blocks some AI search crawlers",
  "policy_code": "blocks-search",
  "robots_txt": "Found",
  "robots_txt_url": "https://www.nytimes.com/robots.txt",
  "sitemaps": [
    "https://www.nytimes.com/sitemaps/new/news.xml.gz"
  ],
  "suggested_fix": "User-agent: OAI-SearchBot\nUser-agent: Claude-SearchBot\nUser-agent: PerplexityBot\nAllow: /\n",
  "training_only_block": "User-agent: GPTBot\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n",
  "unknown_crawlers": [],
  "verdict": "Some AI search/assistant crawlers are blocked (OAI-SearchBot, Claude-SearchBot, PerplexityBot): the site may not appear in those AI answers.",
  "website": "https://www.nytimes.com"
}

Find websites that block an AI crawler

List the popular websites that block one AI crawler, such as GPTBot or ClaudeBot, most popular first, from today's AI Crawler Index of 10,000 websites. Narrow it to one industry or popularity band, and choose fully blocked, partly blocked or both.

Inputs

AI crawlercrawlertextRequired: The crawler to look for. GPTBot trains OpenAI's models; OAI-SearchBot finds pages for ChatGPT search.Default: GPTBot.
How blockedblock_leveltext: Fully blocked keeps the crawler out of the whole site. Partly blocked keeps it out of some pages only.One of: blocked (Fully blocked), partial (Partly blocked (some pages)), either (Fully or partly blocked). Default: blocked.
Industryindustrytext: Only websites in this industry. Leave empty for every industry.
Popularitypopularitytext: Only websites in this popularity band. Leave empty for all 10,000.One of: 1-100 (Top 100), 101-500 (Ranked 101 to 500), 501-1000 (Ranked 501 to 1,000), 1001-3000 (Ranked 1,001 to 3,000), 3000+ (Ranked beyond 3,000), other (Not ranked).
How manylimitnumber: The most websites to return. The total that match is always reported.From 1 to 500. Default: 50.

Outputs

companytext
countnumber: How many websites came back.
crawlertext
domainslist: Just the domains, for looping over.
index_datetext
purposetext
total_matchingnumber: How many websites match in all.
websiteslist: The matching websites, most popular first.

Example output

{
  "company": "OpenAI",
  "count": 2,
  "crawler": "GPTBot",
  "domains": [
    "nytimes.com",
    "medium.com"
  ],
  "index_date": "2026-10-15",
  "purpose": "Trains AI models",
  "total_matching": 144,
  "websites": [
    {
      "industry": "News",
      "policy": "Blocks some AI search crawlers",
      "popularity": "Ranked 101 to 500",
      "status": "blocked",
      "website": "nytimes.com"
    },
    {
      "industry": "News",
      "policy": "AI search allowed, AI training blocked",
      "popularity": "Ranked 101 to 500",
      "status": "blocked",
      "website": "medium.com"
    }
  ]
}

Get AI crawler blocking statistics

Get today's figures from the AI Crawler Index, a daily check of 10,000 popular websites: how many block each of 29 AI crawlers, as counts and percentages, the share among the top 100 sites, how many have an llms.txt file, the breakdown by industry, and the day-by-day trend for the best-known crawlers.

Inputs

None.

Outputs

crawlerslist: One row per crawler.
daily_trendlist: One row per day, oldest first.
datetext: The day the index was built, YYYY-MM-DD.
index_urltext
industrieslist: One row per industry.
most_blocked_crawlertext: The crawler the most websites block.
policieslist: How many websites take each overall stance.
tracking_sincetext
websites_checkednumber
websites_readablenumber: Websites whose robots.txt could be read.
websites_unreadablenumber
websites_with_content_signalsnumber
websites_with_llms_txtnumber
websites_without_robots_txtnumber

Example output

{
  "crawlers": [
    {
      "allowed": 7029,
      "blocked": 1004,
      "blocked_percent": 12.1,
      "blocked_percent_top_100": 20.2,
      "company": "OpenAI",
      "name": "GPTBot",
      "partly_blocked": 265,
      "purpose": "Trains AI models"
    }
  ],
  "daily_trend": [
    {
      "blocked_percent": [
        {
          "blocked_percent": 12.1,
          "crawler": "GPTBot"
        }
      ],
      "date": "2026-10-15",
      "websites_readable": 8298,
      "websites_with_content_signals": 191,
      "websites_with_llms_txt": 1313
    }
  ],
  "date": "2026-10-15",
  "index_url": "https://tools.yukai.uk/ai-crawler-index",
  "industries": [
    {
      "blocked_percent": [
        {
          "blocked_percent": 31.7,
          "crawler": "GPTBot"
        }
      ],
      "industry": "News",
      "industry_code": "news",
      "readable": 394,
      "websites": 431,
      "with_llms_txt": 33
    }
  ],
  "most_blocked_crawler": "CCBot",
  "policies": [
    {
      "policy": "Open to AI crawlers",
      "policy_code": "open",
      "websites": 5728
    },
    {
      "policy": "AI search allowed, AI training blocked",
      "policy_code": "search-only",
      "websites": 477
    }
  ],
  "tracking_since": "2026-10-02",
  "websites_checked": 10000,
  "websites_readable": 8298,
  "websites_unreadable": 1702,
  "websites_with_content_signals": 191,
  "websites_with_llms_txt": 1313,
  "websites_without_robots_txt": 1147
}

List websites that changed their AI crawler rules

See which of 10,000 popular websites changed their rules for AI crawlers since the day before: who started or stopped blocking a crawler, changed their overall stance, added or removed an llms.txt file, or changed their Content Signals. Narrow it to one crawler or one industry. Run it daily to be told when a competitor or a publisher changes course.

Inputs

Kind of changechange_typetext: Which changes to include.One of: all (Every change), started_blocking (Started blocking a crawler), stopped_blocking (Stopped blocking a crawler), policy (Overall AI policy changed), llms_txt (Added or removed an llms.txt file), content_signals (Changed their Content Signals). Default: all.
AI crawlercrawlertext: Only changes for this crawler. Leave empty for every crawler and for policy, llms.txt and Content Signals changes.
Industryindustrytext: Only websites in this industry. Leave empty for every industry.
How manylimitnumber: The most changes to return. The total that match is always reported.From 1 to 500. Default: 100.

Outputs

changeslist
compared_withtext: The day it was compared with.
countnumber
datetext: The day of the newest check, YYYY-MM-DD.
started_blocking_todaynumber: Crawler blocks added across all websites.
stopped_blocking_todaynumber: Crawler blocks lifted across all websites.
total_matchingnumber
websiteslist: Each website with a change listed, once.

Example output

{
  "changes": [
    {
      "after": "blocked",
      "before": "allowed",
      "change_type": "Started blocking a crawler",
      "change_type_code": "started_blocking",
      "crawler": "GPTBot",
      "industry": "Shopping",
      "website": "aliexpress.com",
      "what": "GPTBot"
    },
    {
      "after": "No llms.txt",
      "before": "Has llms.txt",
      "change_type": "Added or removed an llms.txt file",
      "change_type_code": "llms_txt",
      "crawler": "",
      "industry": "News",
      "website": "time.com",
      "what": "llms.txt file"
    }
  ],
  "compared_with": "2026-10-14",
  "count": 2,
  "date": "2026-10-15",
  "started_blocking_today": 113,
  "stopped_blocking_today": 293,
  "total_matching": 498,
  "websites": [
    "aliexpress.com",
    "time.com"
  ]
}

Look up a website in the AI Crawler Index

Look up one of the 10,000 popular websites in today's AI Crawler Index: its industry, how popular it is, its overall stance on AI, which crawlers it allows and blocks, and every change to its rules the index has recorded. Instant, with no live check. Says found is false for a website the index does not cover; use Check which AI crawlers a website allows for those.

Inputs

WebsitewebsitetextRequired: The website's domain, such as nytimes.com. A full link works too; only the domain is used.

Outputs

ai_search_scorenumber: 0 to 100: how open it is to AI search.
allowed_crawlerslist
blocked_crawlerslist
changeslist: Every recorded change, newest first.
content_signalsobject: The site's Cloudflare Content Signals, when it publishes them: 'yes' or 'no' for appearing in search, being used in AI answers and being used to train AI. Empty when the site sets none.
crawlerslist
foundtrue/false: False when the index does not cover the website.
has_llms_txttrue/false
index_datetext
industrytext
industry_codetext
partly_blocked_crawlerslist
policytext
policy_codetext
popularitytext: Its popularity band, such as Top 100.
robots_txttext
unknown_crawlerslist
websitetext

Example output

{
  "ai_search_score": 100,
  "allowed_crawlers": [
    "OAI-SearchBot",
    "ChatGPT-User",
    "Applebot"
  ],
  "blocked_crawlers": [
    "GPTBot",
    "ClaudeBot",
    "Google-Extended"
  ],
  "changes": [
    {
      "after": "blocked",
      "before": "allowed",
      "change_type": "Started blocking a crawler",
      "change_type_code": "started_blocking",
      "crawler": "GPTBot",
      "date": "2026-10-15",
      "what": "GPTBot"
    }
  ],
  "content_signals": null,
  "crawlers": [
    {
      "company": "OpenAI",
      "name": "GPTBot",
      "purpose": "Trains AI models",
      "status": "blocked"
    }
  ],
  "found": true,
  "has_llms_txt": false,
  "index_date": "2026-10-15",
  "industry": "Shopping",
  "industry_code": "ecommerce",
  "partly_blocked_crawlers": [],
  "policy": "AI search allowed, AI training blocked",
  "policy_code": "search-only",
  "popularity": "Ranked 101 to 500",
  "robots_txt": "Found",
  "unknown_crawlers": [],
  "website": "aliexpress.com"
}

Turn a web page into Markdown

Read a web page or a PDF and get its text back as clean Markdown, ready to hand to an AI step, save as a note or summarise: headings, links and tables kept, menus and page furniture dropped, up to 100,000 characters. Pages that only show their text after running scripts in a browser come back nearly empty, with a note saying so.

Inputs

PageurltextRequired: The link to the web page or PDF to read, such as https://example.com/blog/post.

Outputs

character_countnumber: How long the text is.
descriptiontext: The page's own summary, when it has one.
file_typetext: text/html for a web page, application/pdf for a PDF.
final_urltext: Where the link ended up after any redirects.
languagetext: The page's language code, such as en.
markdowntext: The page's text, as Markdown.
notetext: A warning when the page had almost no text without running scripts. Empty otherwise.
titletext: The page's title.
token_estimatenumber: Roughly how many AI tokens the text is.
truncatedtrue/false: True when the page was cut at 100,000 characters.
urltext: The link asked for.

Example output

{
  "character_count": 156,
  "description": "",
  "file_type": "text/html",
  "final_url": "https://example.com/",
  "language": "en",
  "markdown": "This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.",
  "note": "Almost no text without JavaScript.",
  "title": "Example Domain",
  "token_estimate": 39,
  "truncated": false,
  "url": "https://example.com/"
}

ActionsCopy Anchor Link

Check which AI crawlers a website allowsCopy Anchor Link

Inputs

Outputs

Find websites that block an AI crawlerCopy Anchor Link

Inputs

Outputs

Get AI crawler blocking statisticsCopy Anchor Link

Inputs

Outputs

List websites that changed their AI crawler rulesCopy Anchor Link

Inputs

Outputs

Look up a website in the AI Crawler IndexCopy Anchor Link

Inputs

Outputs

Turn a web page into MarkdownCopy Anchor Link

Inputs

Outputs

On this page

Actions

Check which AI crawlers a website allows

Find websites that block an AI crawler

Get AI crawler blocking statistics

List websites that changed their AI crawler rules

Look up a website in the AI Crawler Index

Turn a web page into Markdown