robots.txt

What is a robots.txt file?

• The robots.txt file is a simple text file stored on your web server which tells webcrawlers like Googlebot if they should access a file or not.
  
  

  What is a robots.txt file?

• The robots.txt file is a simple text file stored on your web server which tells webcrawlers like Googlebot if they should access a file or not.

The Importance of a Robots.txt File for Your SEO

Robots.txt file is a text file which tells the search engines to access  pages and index on your website & which pages not to. For example, if you define in your Robots.txt file that you don’t want the search engines to be able to access your thank you page, that page won’t be able to appear in the search results and web users won’t be able to find it. Keeping the search engines from accessing certain pages on your site is foremost for both the privacy of your site and for your SEO.

How to Create Robots.txt Files

   

Use our Robots.txt generator to create a robots.txt file.

Analyze Your Robots.txt File

Google also offers a  tool inside of Google Webmaster Central, and shows Google crawling errors for your site

Example Robots.txt Format

    Allow indexing of everything

   

User-agent: *

Disallow:

   

or

   

User-agent: *

Allow: /

       Disallow indexing of everything

   

   

User-agent: *

Disallow: /

   

   

      Disallow indexing of a specific folder

   

   

User-agent: *

Disallow: /folder/

   

   

Disallow Googlebot from indexing of a folder, except for allowing the indexing of one file in that folder

   

            User-agent: Googlebot

    Disallow: /folder1/

    Allow: /folder1/myfile.html

Why Some Pages Need to Be Blocked

Reasons why you command & want to block a page using the Robots.txt file. , if you have a page on your site which is a very same of another page, you don’t want the robots to index it because that would result in very same content which can hurt your SEO. The second reason is if you have a page on your site which you don’t want users to be able to access unless they take a specific action. For example, if you have a thank you page where users get access to outright information because of the fact that they gave you their email address, you probably don’t want people being able to find that page by performing a Google search. The other time that you’ll want to block pages or files is when you want to look after private files in your site such as your cgi-bin and keep your capacity from being used up because of the robots indexing your image files:

User-agent: *

Disallow: /images/

Disallow: /cgi-bin/

In all of these cases, you’ll need to add a command in your Robots.txt file that tells the search engine spiders not to access that page, not to index it in search results and not to send visitors to it.

Thank you for reading. Continue your journey of learning , join digital marketing course in Chandigarh.