WordPress.org

Forums

what are the files to include in robot.txt? (9 posts)

  1. santocommunity
    Member
    Posted 3 years ago #

    Hi

    Iam running wordpress sites and i want to know what are the wordpress files i should disallow in robots.txt file.

    thanks
    santo

  2. MickeyRoush
    Member
    Posted 3 years ago #

    http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress

    Example:

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    Allow: /wp-content/uploads
    
    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*
    
    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*
    
    # digg mirror
    User-agent: duggmirror
    Disallow: /
    
    Sitemap: http://www.example.com/sitemap.xml
  3. santocommunity
    Member
    Posted 3 years ago #

    hi

    can you please explain me what is this for

    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*

    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*

    # digg mirror
    User-agent: duggmirror
    Disallow: /

  4. Those grant specific permissions to those user-agents.

  5. santocommunity
    Member
    Posted 3 years ago #

    is there any specific reason to allow /wp-content/uploads

  6. santocommunity - Not really.

    I use Disallow: /wp-content/ and block the whole thing.

  7. MickeyRoush
    Member
    Posted 3 years ago #

    is there any specific reason to allow /wp-content/uploads

    It depends on if you want traffic directed to your site when someone is searching for specific images (or any other file there within that directory) via Google, etc. So this could be a bad or good thing depending on how you look at it.

  8. Google will STILL find your images, if they're included in a post, actually, so the only reason you'd leave it open is if you want Google trolling for images you've not posted. And I can't think of why you would.

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags