To make a wordpress robots.txt file, you need to understand two things.
- User-agent : * The section applies to all robots.
- Disallow :/ Tell the robot about the pages on site that it should not visit.
To prevent the robots from crawling or visiting some pages, simply add this page behind the Disallow /
For example, I don’t want the robot visit my admin page in my website site “ www.dukaweb.net/admin “. I add the url of admin in robots.txt like
"User-agent: * Disallow: /admin/
That’s so easy. Below is the standard wordpress robots.txt file which has been used by many wordpress blogger.
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/themes Disallow: /wp-content/plugins Disallow: /search?q=* Disallow: *?replytocom Disallow: */attachment/* Disallow: /images/ Disallow: /forum/ Sitemap: http://dukaweb.net/sitemap.xml