Build a robots.txt file that tells search engines which parts of your site to crawl and which to leave alone. Pick your rules per search engine bot, add your sitemap, and download the finished file.
Search engine crawlers visit your site on their own schedule, and without guidance they will crawl everything they can reach: admin pages, script folders, duplicate URLs, and staging areas you never meant to show anyone. That wastes your crawl budget and can put pages in the index that should never appear in search results. A robots.txt file gives crawlers instructions. It sits in your website root and tells them which pages and folders they may visit and which to leave alone. The Robots.txt Generator builds that file for you. You pick your rules, add your sitemap, and download the finished file ready to upload.
Robots.txt is a set of instructions for crawlers, not a lock on a door. Well behaved crawlers from Google, Bing, and other search engines read it and follow it. It does not hide a page from the public: anyone with the URL can still open it, and a blocked URL can still appear in search results if other sites link to it. If a page must stay out of search results entirely, use a noindex meta tag on the page itself. Robots.txt controls crawling, not indexing, and not access. Understanding that limit keeps you from relying on it for privacy or security.
Within that limit, it is a powerful tool. Blocking admin areas, script folders, and duplicate content keeps crawlers focused on the pages that matter. That matters most on larger sites, where crawlers have a limited budget and every wasted visit is a visit not spent on a real page.
The generator lets you write separate allow or disallow rules for 15 search engine bots, so you can treat crawlers differently if you need to. Most sites use the same rules for every bot, which is the simplest and safest choice. A per bot rule is useful when one crawler causes trouble: a bot that hammers your server, a crawler you do not want indexing images, or a search engine you do not serve. The default rule applies to all bots, and specific rules override it for the bot they name.
You can block up to six directories, such as admin or script folders. Choose them carefully. Block only areas that have no value in search results: admin panels, server scripts, staging folders, and duplicate or filtered URL paths. Never block your images, CSS, or JavaScript folders. Google needs those files to see your pages properly, and blocking them can hurt your rankings.
The crawl delay setting, from 5 to 120 seconds, tells crawlers to wait between requests. It is useful on small servers that struggle when a crawler visits many pages at once. Use it only if you need it, because a long delay slows down how fast your pages get crawled. Most sites on normal hosting do not need a crawl delay at all.
The sitemap field adds your sitemap URL to the file, which tells crawlers where to find the full list of your pages. Include it. It costs one line, and it helps crawlers discover new and deep pages faster. If you do not have a sitemap yet, the XML Sitemap Generator can build one for you.
Upload robots.txt to the root folder of your website, so it is reachable at your domain followed by robots.txt. Then check that it loads in a browser, and test it in Google Search Console to confirm the rules do what you expect. A quick test takes a minute and can catch a typo that would otherwise block your whole site. Review the file whenever you add a new section, move content, or change your sitemap, so the rules stay in step with the site they describe.
Use our free SEO tools to analyze, fix and grow your website.
It sits in your website root and tells search engine crawlers which pages or folders they may visit. Crawlers read it before they explore your site, so it controls what gets crawled, though it does not hide pages from the public.
Upload it to the root folder of your website, so it is reachable at yourdomain.com/robots.txt. Most hosting control panels let you upload it through the file manager, or you can use FTP.
No. Blocking your images, CSS, or JavaScript folders stops Google from seeing your pages properly, which can hurt rankings. Only block private areas like admin pages, script folders, or duplicate content.
Not on its own. Robots.txt stops crawling, but Google can still list a blocked URL if other sites link to it. To keep a page out of search results, use a noindex meta tag on the page itself.