WordPress.org

1. santocommunity
   Member
   Posted 9 months ago #

   Hi

   Iam running wordpress sites and i want to know what are the wordpress files i should disallow in robots.txt file.

   thanks
   santo

2. MickeyRoush
   Member
   Posted 9 months ago #

   http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress

   Example:

   User-agent: *
   Disallow: /cgi-bin
   Disallow: /wp-admin
   Disallow: /wp-includes
   Disallow: /wp-content/plugins
   Disallow: /wp-content/cache
   Disallow: /wp-content/themes
   Disallow: /trackback
   Disallow: /feed
   Disallow: /comments
   Disallow: /category/*/*
   Disallow: */trackback
   Disallow: */feed
   Disallow: */comments
   Disallow: /*?*
   Disallow: /*?
   Allow: /wp-content/uploads
   
   # Google Image
   User-agent: Googlebot-Image
   Disallow:
   Allow: /*
   
   # Google AdSense
   User-agent: Mediapartners-Google*
   Disallow:
   Allow: /*
   
   # digg mirror
   User-agent: duggmirror
   Disallow: /
   
   Sitemap: http://www.example.com/sitemap.xml

3. Ipstenu
   Half-Elf Support Rogue & Mod
   Posted 9 months ago #

   See also: http://perishablepress.com/wordpress-robots-rules/

4. santocommunity
   Member
   Posted 9 months ago #

   hi

   can you please explain me what is this for

   # Google Image
   User-agent: Googlebot-Image
   Disallow:
   Allow: /*

   # Google AdSense
   User-agent: Mediapartners-Google*
   Disallow:
   Allow: /*

   # digg mirror
   User-agent: duggmirror
   Disallow: /

5. Ipstenu
   Half-Elf Support Rogue & Mod
   Posted 9 months ago #

   Those grant specific permissions to those user-agents.

6. santocommunity
   Member
   Posted 8 months ago #

   is there any specific reason to allow /wp-content/uploads

7. Ipstenu
   Half-Elf Support Rogue & Mod
   Posted 8 months ago #

   santocommunity - Not really.

   I use Disallow: /wp-content/ and block the whole thing.

8. MickeyRoush
   Member
   Posted 8 months ago #

   is there any specific reason to allow /wp-content/uploads

   It depends on if you want traffic directed to your site when someone is searching for specific images (or any other file there within that directory) via Google, etc. So this could be a bad or good thing depending on how you look at it.

9. Ipstenu
   Half-Elf Support Rogue & Mod
   Posted 8 months ago #

   Google will STILL find your images, if they're included in a post, actually, so the only reason you'd leave it open is if you want Google trolling for images you've not posted. And I can't think of why you would.

Reply

You must log in to post.