To make a wordpress robots.txt file, you need to understand two things.

• User-agent : * The section applies to all robots.
• Disallow :/ Tell the robot about the pages on site that it should not visit.

To prevent the robots from crawling or visiting some pages, simply add this page behind the Disallow /

For example, I don’t want the robot visit my admin page in my website site “ www.dukaweb.net/admin “. I add the url of admin in robots.txt like

"User-agent: *
Disallow: /admin/

That’s so easy. Below is the standard wordpress robots.txt file which has been used by many wordpress blogger.

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/themes
Disallow: /wp-content/plugins
Disallow: /search?q=*
Disallow: *?replytocom
Disallow: */attachment/*
Disallow: /images/
Disallow: /forum/
Sitemap: http://dukaweb.net/sitemap.xml