Create and use file robots.txt

Here is a guide to create and use a robots.txt file to the website, so that search engines index your website content.

What's the robots.txt file? That is a structured text file, when the spider (bot, crawler) of the SE (Search Engine) on the website to collect data will robots.txt file to see the instructions in this file.

Robots.txt can define each different bot the different SE can go to the website or in areas of the website?

Some types of SE bots: Googlebot (Google), Googlebot-Image (Google), Yandex (Russia SE)​​, Bingbot (Bing) / Yahoo Slurp (Yahoo) ...

The common syntax of robots.txt file:

User-agent: bot object is accepted

Disallow / Allow: URL you want to block / allow

*: Represent all

For example: User-agent: * (That means accepting all kinds of bots.)

Lock the entire site

Disallow: /

Block 1 folder and everything in it

Disallow: / wp-admin /

Block 1 page

Disallow: / private_file.html

Remove first image from Google Images

User-agent: Googlebot-Image

Disallow: / images / sexy.jpg

Remove all pictures from Google Images:

User-agent: Googlebot-Image

Disallow: /

Block 1 file any, for example. Gif

User-agent: Googlebot

Disallow: / *. Gif $

Things to avoid in the robots.txt file

Is case sensitive.

Do not write balances, lack of white.

Do not insert any character other than the command syntax.

Each command should be on one line.

How to create and where to place the robots.txt file

Use notepad or any other program to create the file, then rename the file robots.txt.

Located in the root directory of the website. (http://yoursite.com/robots.txt)