WordPress.org

Ready to get started?Download WordPress

Forums

robots.txt problem (4 posts)

  1. jetskiron
    Member
    Posted 2 years ago #

    Is there anything in the below robots.txt contents that would stop Googlebot from crawling and indexing http://www.insuranceclaimhelp.org ? This was the message posted by Wayback
    http://wayback.archive.org/web/*/http://www.insuranceclaimhelp.org/

    (you will need to copy paste the entire url in your address bar. It was apparently not all highlighted above?)

    robots.txt :

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/
    Disallow: /trackback/
    Disallow: /images
    Disallow: /feed/
    Disallow: /comments/
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /comments
    Disallow: /peoples-choice
    Disallow: /*?*
    Disallow: /*?
    Disallow: /*?
    Disallow: /*/*rate
    Disallow: /*/*add
    Disallow: /aboutadj.asp
    Disallow: /?*

    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*
    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*
    # Internet Archiver Wayback Machine
    User-agent: ia_archiver
    Disallow: /
    # digg mirror
    User-agent: duggmirror
    Disallow: /

  2. Peter_L
    Member
    Posted 2 years ago #

    You don't need all that stuff.

    The only things I've got in my robots.txt is this

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
  3. jetskiron
    Member
    Posted 2 years ago #

    Thanks Peter,

    What kinds of things would be in the wp-includes that robots should not see?

  4. Peter_L
    Member
    Posted 2 years ago #

    Not sure to be honest. It's just a snippet I picked up somewhere.

    The point being, you don't really need robots.txt in WordPress.

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags