free web tracker
Robots.txt Generator

Robots.txt Generator

Build a robots.txt file that tells search engines which parts of your site to crawl and which to leave alone. Pick your rules per search engine bot, add your sitemap, and download the finished file.

  • 100% Free
  • No Registration
  • Accurate Results
  • Fast & Secure

Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

Search engine crawlers visit your site on their own schedule, and without guidance they will crawl everything they can reach: admin pages, script folders, duplicate URLs, and staging areas you never meant to show anyone. That wastes your crawl budget and can put pages in the index that should never appear in search results. A robots.txt file gives crawlers instructions. It sits in your website root and tells them which pages and folders they may visit and which to leave alone. The Robots.txt Generator builds that file for you. You pick your rules, add your sitemap, and download the finished file ready to upload.

What robots.txt can and cannot do

Robots.txt is a set of instructions for crawlers, not a lock on a door. Well behaved crawlers from Google, Bing, and other search engines read it and follow it. It does not hide a page from the public: anyone with the URL can still open it, and a blocked URL can still appear in search results if other sites link to it. If a page must stay out of search results entirely, use a noindex meta tag on the page itself. Robots.txt controls crawling, not indexing, and not access. Understanding that limit keeps you from relying on it for privacy or security.

Within that limit, it is a powerful tool. Blocking admin areas, script folders, and duplicate content keeps crawlers focused on the pages that matter. That matters most on larger sites, where crawlers have a limited budget and every wasted visit is a visit not spent on a real page.

Setting rules per bot

The generator lets you write separate allow or disallow rules for 15 search engine bots, so you can treat crawlers differently if you need to. Most sites use the same rules for every bot, which is the simplest and safest choice. A per bot rule is useful when one crawler causes trouble: a bot that hammers your server, a crawler you do not want indexing images, or a search engine you do not serve. The default rule applies to all bots, and specific rules override it for the bot they name.

You can block up to six directories, such as admin or script folders. Choose them carefully. Block only areas that have no value in search results: admin panels, server scripts, staging folders, and duplicate or filtered URL paths. Never block your images, CSS, or JavaScript folders. Google needs those files to see your pages properly, and blocking them can hurt your rankings.

Crawl delay and sitemap

The crawl delay setting, from 5 to 120 seconds, tells crawlers to wait between requests. It is useful on small servers that struggle when a crawler visits many pages at once. Use it only if you need it, because a long delay slows down how fast your pages get crawled. Most sites on normal hosting do not need a crawl delay at all.

The sitemap field adds your sitemap URL to the file, which tells crawlers where to find the full list of your pages. Include it. It costs one line, and it helps crawlers discover new and deep pages faster. If you do not have a sitemap yet, the XML Sitemap Generator can build one for you.

Mistakes to avoid

  • Do not disallow your whole site by accident. One wrong rule can block every crawler from every page.
  • Do not block folders that hold images, CSS, or JavaScript files.
  • Do not use robots.txt to hide private content. It is a public file, and anyone can read it.
  • Do not list private URLs in the file. A disallow line advertises the path it blocks.
  • Do not forget to update the file when your site structure changes. Old rules can block new sections.

After you upload the file

Upload robots.txt to the root folder of your website, so it is reachable at your domain followed by robots.txt. Then check that it loads in a browser, and test it in Google Search Console to confirm the rules do what you expect. A quick test takes a minute and can catch a typo that would otherwise block your whole site. Review the file whenever you add a new section, move content, or change your sitemap, so the rules stay in step with the site they describe.

Improve Your Website's SEO Today

Use our free SEO tools to analyze, fix and grow your website.

Explore All Tools

Frequently Asked Questions

It sits in your website root and tells search engine crawlers which pages or folders they may visit. Crawlers read it before they explore your site, so it controls what gets crawled, though it does not hide pages from the public.

Upload it to the root folder of your website, so it is reachable at yourdomain.com/robots.txt. Most hosting control panels let you upload it through the file manager, or you can use FTP.

No. Blocking your images, CSS, or JavaScript folders stops Google from seeing your pages properly, which can hurt rankings. Only block private areas like admin pages, script folders, or duplicate content.

Not on its own. Robots.txt stops crawling, but Google can still list a blocked URL if other sites link to it. To keep a page out of search results, use a noindex meta tag on the page itself.