Logo of LLM Readiness Check

LLM Readiness Check

Test your site's readiness for AI crawlers in 10 seconds

User reviews

Log in to leave a review

The robots.txt snippet is the part I'd have wanted a month ago. We let every AI crawler in on purpose, and the trouble came from the other side: a few bots hammered our job search URLs with endless filter combinations, each one a heavy database query, and the site started freezing. Disallowing the parameterised search paths while keeping the job pages open fixed it. Does the score treat "allow everything" as the ideal, or does it look at what the bots can actually reach, like infinite filter URLs?

View
igrecu's avatar
igrecuAuthorOct 6, 2026

Every website needs to pick what works best for them in terms of robots and crawler access. Cloudflare made a lot of updates lately which impact different types of AI crawlers (Training, Search Bots, User Bots). For regular use, I'd say excluding URL query variables from robots (if you have a search functionality or categorization) is better than allowing them to index everything. You can find some useful blog articles on AI crawler access on our blog.

SpinHire's avatar
SpinHireOct 7, 2026

@igrecu  thanks, the split into training, search and user bots is useful. We had been treating them as one group, and the ones fetching pages live for a user are exactly the ones we want to keep. Did you see sites block user bots by accident when they turned on the Cloudflare default?