Robots.txt

A public file at the root of your website that gives search engine bots instructions about which pages they should or should not crawl. It's not a security tool - anyone can read it.