WordPress.org

Ready to get started?Download WordPress

Forums

[resolved] Word Press privacy & robots.txt creating indexing problem (5 posts)

  1. JeriMerrell
    Member
    Posted 7 years ago #

    When I first set up my site, I set wordpress to private, no indexing by search engines, while I worked on development. When I launched it, I changed the setting to allow all search engines.

    Now Google won't index my site. The error message is this:

    http://www.ungeekit.com/
    URL restricted by robots.txt [?]
    Jun 12, 2007

    The robots.txt file contents, as reported by google and by accessing the page http://www.ungeekit.com/robots.txt are this:

    User-agent: *
    Disallow:

    It seems to me those entries should allow all and not restrict any indexing whatsoever. (It's kind of a meaningless robots.txt file, actually.)

    And on top of all that… when access the site files directly via FTP, I don't actually SEE the robots.txt file to delete it, or I would!

    I'm probably just not comprehending something basic about the use of robots.txt or the privacy setting in WordPress, or missing something in my file structure, but this is driving me nuts. Any suggestions would be welcome – thanks!

    Jeri

  2. Marcel Brinkkemper
    Member
    Posted 7 years ago #

    a robots.txt file only describes the paths that are not allowed. You don't have to use one if you allow all agents to go everywhere. You could use an empty file if you don't want 404 errors in your logfile.

  3. Samuel Wood (Otto)
    Tech Ninja
    Posted 7 years ago #

    WordPress 2.2 returns that robots.txt file when you have the blog set to public. Perhaps Google has simply not noticed the change yet, but that robots.txt is perfectly valid and should allow it to index everything.

  4. JeriMerrell
    Member
    Posted 7 years ago #

    Google caching a dated version appears to be the issue. (BTW - I can't seem to find the file in my FTP listings to delete it - annoying.)

    When I ask Google to test the current version - it tests fine with today's date.

    But it still shows the dated, week-old restricted version on its main listing for my site and won't index it. I wish there was some tool or command to kick off a refresh or re-index of the site. So much for Google's claim to crawl 'about once a day'.

    Thanks much, folks.

  5. Samuel Wood (Otto)
    Tech Ninja
    Posted 7 years ago #

    (BTW - I can't seem to find the file in my FTP listings to delete it - annoying.)

    There is no file. WordPress generates it on the fly, like it generates everything else.

Topic Closed

This topic has been closed to new replies.

About this Topic