About the Robots.txt Generator
The robots.txt file sits in the root of your website (for example example.com/robots.txt) and tells search engine crawlers which parts of your site they may visit. A correct robots.txt keeps crawlers away from admin pages, search result pages and duplicate URLs, points them to your XML sitemap, and can block bots you do not want.
The Robots.txt Generator builds the file for you. Allow or block all robots, list the folders to disallow, set an optional crawl delay, add your sitemap URL, and block specific bots with one click, including AI crawlers like GPTBot, Google-Extended, CCBot and ClaudeBot, and SEO crawlers like AhrefsBot and SemrushBot. Download the file and upload it to your website's root folder.
How to use the Robots.txt Generator
- Choose the default rule for all robots.
- Add your sitemap URL and any folders to disallow.
- Tick the bots you want to block completely.
- Download robots.txt and upload it to your website root.
Frequently asked questions
Does robots.txt hide pages from Google?
It prevents crawling, but blocked URLs can still appear in search if other sites link to them. Use a noindex meta tag to keep a page out of search results.
Where do I upload robots.txt?
In the root folder of your domain, so it is available at https://yourdomain.com/robots.txt. On cPanel this is the public_html folder.
Should I block AI crawlers?
It is your choice. Blocking them prevents your content from being used for AI training by bots that respect robots.txt, and does not affect normal Google search.