Today, I am sharing with you a very useful and must aware blogging term that is Robots.txt.
In blogger it is known as Custom Robots.txt that means now you can customize this file according to your choice. In today’s tutorial, i will discuss this term in deep and come to know about its use and benefits. I will also tell you how to add custom robots.txt file in blogger.
So let start the tutorial.
What is Robots.txt?
Robots.txt is a text file that contains few lines of code. It is saved on the website or on the blog server, which instructs web crawlers on how to index and crawl your blog in search results. This means that you can restrict any webpage in your web crawler blog so that it can not be indexed in search engines, such as your blog’s tag page, your demo page or any other page that is not so important to be indexed . Always remember that search crawlers scan the robots.txt file before crawling any web page.
How To Add Custom Robots.txt in your Blogger
1) Login to your Blogger account.
2) Open the blog for which you want to add Robots.txt
3) Navigate to Settings >> Search Preferences ›› Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes.
4) Now paste your robots.txt file code in the box.
5) Click on Save Changes button.
6) You are done!