Robots.txt

Simple explanation

A file telling search engine and AI crawlers which parts of a site they can access.

Advanced explanation

A permission signal, not a security measure — disallowed pages can still be indexed if linked elsewhere, just not crawled directly.

Related terms