WordPress.org

Ready to get started?Download WordPress

Forums

[resolved] robots.txt gets a 404 (6 posts)

  1. benwaynet
    Member
    Posted 9 months ago #

    I have a single site, there is a robots.txt in the root of my wordpress install. But when I visit http://www.benway.net/robots.txt it gives a 404

    I'm using pretty permalinks.

    Anything else to look at why this isn't working

    google and bing are complaining they can't find the file.

    thanks,jb

  2. leejosepho
    Member
    Posted 9 months ago #

    I see this:

    # BEGIN XML-SITEMAP-PLUGIN
    Sitemap: http://www.benway.net/sitemap.xml.gz
    # END XML-SITEMAP-PLUGIN

    Robots.txt can have that Sitemap line, but not those others. Got to any website anywhere and add robots.txt to its URL to see some examples.

  3. benwaynet
    Member
    Posted 9 months ago #

    you saw that when you accessed the /robots.txt ?? strange. I just renamed the rebots.txt file in the root and now I'm getting

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    which is better than a 404.

    Guess wordpress is generating it and so I won't have a way to tweak my robots file if I wanted to...

  4. leejosepho
    Member
    Posted 9 months ago #

    I cleared my Firefox cache and got the same as you this time. To get my own file working properly, I had to un-check "Add sitemap URL to the virtual robots.txt file." Otherwise, it seems WordPress does still send out its virtual file.

  5. leejosepho
    Member
    Posted 9 months ago #

    Also, putting out your sitemap without the .gz will let you use its link like this, if you wish:
    http://www.nnysandbox.net/sitemap.xml

    And, I have found it best to "Rebuild Sitemap Manually" after any changes since the plugin seems to miss any new "Ignore these pages" if I do not.

  6. benwaynet
    Member
    Posted 9 months ago #

    I got it cleaned up, I found the lines you where referring to and once i removed those. I could manually create my robot.txt in the root directory.

    thanks for the help!

    jb

Reply

You must log in to post.

About this Topic

Tags