Robots.txt
A file that tells search engine crawlers which pages they can and cannot access.
Why It Matters
Robots.txt controls crawl behavior and prevents search engines from wasting resources on unimportant pages.
How It Works
The robots.txt file sits at your domain root and contains directives like Allow and Disallow for different user agents. Crawlers check this file before accessing pages. It controls crawling, not indexing.
Real-World Example
Disallowing /admin/ in robots.txt prevents Googlebot from wasting crawl budget on backend pages.
Common Mistakes
Using robots.txt to try to deindex pages (use noindex instead)
Accidentally disallowing critical CSS or JS files
Robots.txt FAQs
Does robots.txt prevent pages from being indexed?
No, robots.txt only prevents crawling; pages can still be indexed if linked from other crawled pages.
Where should robots.txt be located?
It must be at the root of your domain: yoursite.com/robots.txt.
Need help with robots.txt?
Get matched with a vetted specialist in 48 hours.
Ready to Get Started?
Get matched with a vetted specialist in 48 hours. No recruitment fees, no lengthy hiring process, just results.