Create and use file robots.txt

Monday, November 19, 2012

Create and use file robots.txt

Here is a guide to create and use a robots.txt file to the website, so that search engines index your website content.

Create and use file robots.txt



What's the robots.txt file? That is a structured text file, when the spider (bot, crawler) of the SE (Search Engine) on the website to collect data will robots.txt file to see the instructions in this file.

Robots.txt can define each different bot the different SE can go to the website or in areas of the website?

Some types of SE bots: Googlebot (Google), Googlebot-Image (Google), Yandex (Russia SE)​​, Bingbot (Bing) / Yahoo Slurp (Yahoo) ...

The common syntax of robots.txt file:

User-agent: bot object is accepted
Disallow / Allow: URL you want to block / allow

*: Represent all

For example: User-agent: * (That means accepting all kinds of bots.)

Lock the entire site
Disallow: /

Block 1 folder and everything in it
Disallow: / wp-admin /

Block 1 page
Disallow: / private_file.html

Remove first image from Google Images
User-agent: Googlebot-Image
Disallow: / images / sexy.jpg

Remove all pictures from Google Images:
User-agent: Googlebot-Image
Disallow: /

Block 1 file any, for example. Gif
User-agent: Googlebot
Disallow: / *. Gif $

Create and use file robots.txt

Things to avoid in the robots.txt file

Is case sensitive.
Do not write balances, lack of white.
Do not insert any character other than the command syntax.
Each command should be on one line.
How to create and where to place the robots.txt file

Use notepad or any other program to create the file, then rename the file robots.txt.
Located in the root directory of the website. (http://yoursite.com/robots.txt)