Support » How-To and Troubleshooting » [Resolved] robots.txt gets a 404

[Resolved] robots.txt gets a 404

  • benwaynet


    I have a single site, there is a robots.txt in the root of my wordpress install. But when I visit http://www.benway.net/robots.txt it gives a 404

    I’m using pretty permalinks.

    Anything else to look at why this isn’t working

    google and bing are complaining they can’t find the file.


Viewing 5 replies - 1 through 5 (of 5 total)
  • leejosepho


    I see this:

    Sitemap: http://www.benway.net/sitemap.xml.gz

    Robots.txt can have that Sitemap line, but not those others. Got to any website anywhere and add robots.txt to its URL to see some examples.



    you saw that when you accessed the /robots.txt ?? strange. I just renamed the rebots.txt file in the root and now I’m getting

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    which is better than a 404.

    Guess wordpress is generating it and so I won’t have a way to tweak my robots file if I wanted to…



    I cleared my Firefox cache and got the same as you this time. To get my own file working properly, I had to un-check “Add sitemap URL to the virtual robots.txt file.” Otherwise, it seems WordPress does still send out its virtual file.



    Also, putting out your sitemap without the .gz will let you use its link like this, if you wish:

    And, I have found it best to “Rebuild Sitemap Manually” after any changes since the plugin seems to miss any new “Ignore these pages” if I do not.



    I got it cleaned up, I found the lines you where referring to and once i removed those. I could manually create my robot.txt in the root directory.

    thanks for the help!


Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘[Resolved] robots.txt gets a 404’ is closed to new replies.