SEO and AI Search
robots.txt is a file at the root of a site that tells crawlers which areas they may or may not visit.
In Nigeria and EMEA
It controls crawling, not whether a page appears in results. Do not use it to hide private content.
Related terms
Stay sharp on marketing news and trends
Get remote roles open to Nigeria and EMEA, the marketing news that matters and one tactic to use, free every Wednesday.
Subscribe freeBrowse the full glossary