Word Press privacy & robots.txt creating indexing problem

  • Resolved jerimerrell

    (@jerimerrell)


    16 years, 10 months ago

    When I first set up my site, I set wordpress to private, no indexing by search engines, while I worked on development. When I launched it, I changed the setting to allow all search engines.

    Now Google won’t index my site. The error message is this:

    http://www.ungeekit.com/
    URL restricted by robots.txt [?]
    Jun 12, 2007

    The robots.txt file contents, as reported by google and by accessing the page http://www.ungeekit.com/robots.txt are this:

    User-agent: *
    Disallow:

    It seems to me those entries should allow all and not restrict any indexing whatsoever. (It’s kind of a meaningless robots.txt file, actually.)

    And on top of all that… when access the site files directly via FTP, I don’t actually SEE the robots.txt file to delete it, or I would!

    I’m probably just not comprehending something basic about the use of robots.txt or the privacy setting in WordPress, or missing something in my file structure, but this is driving me nuts. Any suggestions would be welcome – thanks!

    Jeri

Viewing 4 replies - 1 through 4 (of 4 total)
  • Marcel Brinkkemper

    (@macbrink)

    16 years, 10 months ago

    a robots.txt file only describes the paths that are not allowed. You don’t have to use one if you allow all agents to go everywhere. You could use an empty file if you don’t want 404 errors in your logfile.

    Moderator Samuel Wood (Otto)

    (@otto42)

    WordPress.org Admin

    16 years, 10 months ago

    WordPress 2.2 returns that robots.txt file when you have the blog set to public. Perhaps Google has simply not noticed the change yet, but that robots.txt is perfectly valid and should allow it to index everything.

    Thread Starter jerimerrell

    (@jerimerrell)

    16 years, 10 months ago

    Google caching a dated version appears to be the issue. (BTW – I can’t seem to find the file in my FTP listings to delete it – annoying.)

    When I ask Google to test the current version – it tests fine with today’s date.

    But it still shows the dated, week-old restricted version on its main listing for my site and won’t index it. I wish there was some tool or command to kick off a refresh or re-index of the site. So much for Google’s claim to crawl ‘about once a day’.

    Thanks much, folks.

    Moderator Samuel Wood (Otto)

    (@otto42)

    WordPress.org Admin

    16 years, 10 months ago

    (BTW – I can’t seem to find the file in my FTP listings to delete it – annoying.)

    There is no file. WordPress generates it on the fly, like it generates everything else.

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Word Press privacy & robots.txt creating indexing problem’ is closed to new replies.