The robots.txt snippet is the part I'd have wanted a month ago. We let every AI crawler in on purpose, and the trouble came from the other side: a few bots hammered our job search URLs with endless filter combinations, each one a heavy database query, and the site started freezing. Disallowing the parameterised search paths while keeping the job pages open fixed it. Does the score treat "allow everything" as the ideal, or does it look at what the bots can actually reach, like infinite filter URLs?