AI-policy checker

What does a website tell AI crawlers? robots.txt rules for 25 named AI bots, Content-Signal, RSL licence and price, TDMRep, llms.txt, and whether it charges crawlers (HTTP 402 / pay per crawl).

Free. Results are kept for 24 hours; nothing about you is logged, which is more than most of these sites can say. Share a result with its link: wknipe.com/check/?d=example.com

Well-known publishers

Checked live, so each result is at most a day old. Click one for the full report, and the occasional surprise.

Loading…

What it checks, and how

  • robots.txt, evaluated for 25 named AI crawlers under RFC 9309: the group naming the crawler (or User-agent: * if none does), most specific rule wins, ties go to allow. Blocked means the crawler may not fetch the homepage; some paths blocked means it may, but parts of the site are off limits. robots.txt that can’t be fetched (server error) counts as blocked, as the standard says; a missing one means allowed.
  • Content-Signal lines in robots.txt (contentsignals.org): search, ai-input, ai-train yes/no.
  • RSL: License: links in robots.txt or a <link type="application/rsl+xml"> on the homepage (rslstandard.org). I read the first licence file to report its payment terms and permissions.
  • TDMRep: /.well-known/tdmrep.json and the tdm-reservation meta tag (W3C TDMRep).
  • llms.txt at the site root (llmstxt.org).
  • Charging crawlers: one GET of the homepage as an identified crawler, looking for HTTP 402, a Cloudflare pay-per-crawl crawler-price header, or an x402 payment offer. Most sites that charge only do so for verified, signed crawlers, so No here doesn’t rule out a paid deal.

Polite by design. The checker identifies itself as wknipe-policy-check/1.0 (+https://wknipe.com/check/), fetches only the five files above (plus redirects within the same site), waits at most 6 seconds for each, follows redirects only within the same site, and fetches each domain at most once a day. It stores the result for 24 hours and nothing about who asked. It’s also available as a free API: https://api.wknipe.com/v1/check?domain=example.com.

Why this exists. Publishers are starting to write down, in ways a machine can read, what AI may do with their work, and agents are starting to pay for content. Most of those terms are buried in files nobody opens. This page opens them for any site. The x402 Clean Index covers the other side: what agents actually pay for.