What is a Robots.txt file ?
The Robots.txt file tells search engines which pages they can explore or not. This file is placed at the root of the site at https://exemple.com/robots.txt.
This file can, if misused, cause some of your pages to be indexed in search engine results, even if you don't want them to.
How to check the Robots.txt file ?
- Make sure robots.txt doesn't return an HTTP 5XX status code
- Keep robots.txt smaller than 500 KiB. Search engines may stop processing robots.txt midway through if the file is larger than 500 KiB.
- Fix any format errors
- Provide an absolute URL for sitemap (https://exemple.com/sitemap.xml instead of /sitemap.xml)
