WordPress.org

Ready to get started?Download WordPress

Forums

Robots Error (5 posts)

  1. johnny79
    Member
    Posted 2 years ago #

    When I look at my website with robots.txt in a browser I see the following

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    However, in the root directory of my wordpress installation the robots text contains

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: /go/
    Disallow: /comments/feed/
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /index.php
    Disallow: /xmlrpc.php
    Disallow: ?comments=*
    Disallow: /search?
    Disallow: /?p=*

    User-agent: Mediapartners-Google*
    Allow: /

    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/

    User-agent: Adsbot-Google
    Allow: /

    User-agent: Googlebot-Mobile
    Allow: /

    Does anybody have ideas about why my robots file is not properly working? The only SEO plugin I'm using is WordPress SEO. Thanks

  2. Brandon Kraft
    Happiness Engineer
    Posted 2 years ago #

    To clarify, both the robots.txt files are at the root of your domain? If your WP install isn't in root (e.g. http://example.com with WP installed in http://example.com/wp/), having the robots.txt in /wp/ won't help.

    The WordPress SEO allows you to edit a robots.txt directly within the plugin, in the Files menu. It may make things simpler to copy/paste your desired text into the plugin, then delete the physical file.

    What's your website?

  3. johnny79
    Member
    Posted 2 years ago #

    I figured out what the problem was. Thank you for the help.

  4. Brandon Kraft
    Happiness Engineer
    Posted 2 years ago #

    Do share the solution. It could help someone else searching for a fix later on.

    Glad it is fixed!

  5. johnny79
    Member
    Posted 2 years ago #

    Apparently it had something to do with my hosting company not updating my robots file as fast I would have liked.

    All I did was delete the old hosts file that was in the root directory of the public-_html folder and uploaded a new file within the root directory of the specific domain I was working on.

    Thanks for reaching out to help. It's good to see another Texan!

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags

No tags yet.