Robots.txt File

Robots.txt file in a search engine optimization is a text file that webmasters create to instruct search engines that how to crawl the website.

This is a part of the Robots Exclusion Protocol (REP). REP included directives such as page, subdirectory, meta robots and it is web standards that regulate how to access the content and crawl the web.

This file tells google/bing/yahoo which pages on the website to crawl and which pages on the website not to crawl.

Why do we need it?

If there is no robots.txt file then google/bing/yahoo visit all pages of the website. If any user does not allow the search engine to not visit any page/posts/picture then this is used. Search engines honor this but it is not necessary.

Carefully implement this because if any command goes wrong then it blocks the whole website. If you not an expert in this then use build-in functions or hire a technical resource to build this site.

Command for robots.txt

Sr	Command	Result
1.	User-agent : *	Allow all search engines to crawl a website.
2.	User-agent * Disallow : /	Disallow all search engines to crawl a website.
3.	Disallow : /folder/	Disallow search engines from a particular folder
4.	Disallow: /file.html/	Disallow search engines from a particular file
5.	Disallow: /image.png/	Disallow search engines from a particular image

Rules

Robots.txt file must be in a root directory of a website. It is case sensitive and must be named “robots.txt”, no capital letter is used in this name.

It is publicly available and adds/robots.txt at the end of any website name and if this is available on this website then everyone can see that what is crawled or not crawled. Add the location of sitemaps of the website at the end.

Where it is located?

This file is located at the root of the website, and you can access it at http://www.example.com/robots.txt

Example:

Below is a simple example of a site without robots.tx and site with robots.txt