A Guide On Your WordPress Robots.txt File

Google CrawlingRobots.txt file is used by site owners to stop the search engines from crawling certain part of your blog or website like the admin dashboard, admin folder  and the rest. A Robots.txt always reside at the root of all domains. For instance your domain.com/Robot.txt
If the  search engine bots visits a  blog or website and they have limited resources to crawl the blog or website,  they may  stop crawling, and this Robots.txtcould  affect your blog or website indexing. Also,  there are many parts of your blog or website, that you may not like the search engine to crawl and when you use  robots.txt, you are directing search engine crawlers not to crawl such certain area of your blog or website. Note that this action would  speed up the crawling process of your blog or website and  also help the  deep crawling of your inner pages.


To check your domain robots.txt file  go to www.domain.com/robots.txt.  In most blogging platforms you may see a blank robots.txt file but you can also check your domain Robots.txt file using Google webmaster tools.
To do this Google webmaster tool
Go to site configuration
Click on Crawler Access
To  avoid duplicate content the structure should be like this;
User-agent:  *
Disallow: /wp-
Disallow:  /trackback/
The robot.txt file is only going to stop the search engines from crawling your blogs or websites but not the indexing parts. It will only prevent  aspects like your  feeds, trackbacks, admin folder as well as your  pages and comments from being crawled. If you have some pages or posts you do not want the search engines to index, you can make use of the Yoast SEO plugin to add the Noindex tags to the pages or posts you do not want the search engines to index on your blog or website.
Hope this help? Do share!