WordPress.org

Ready to get started?Download WordPress

Forums

no robots.txt generated when there are no published posts (15 posts)

  1. RavanH
    Member
    Posted 4 years ago #

    Hi,

    I noticed that the /robots.txt that is automatically generated by WP (to allow or disallow site crawling by search engines) is only generated when there are published posts. On any install that contains only pages and no posts (or only concepts) visiting /robots.txt will result in a WP generated 404 error page.

    Is this a bug?

    Allard

  2. Samuel B
    moderator
    Posted 4 years ago #

    are you using some plugin? because wordpress does not update robots.txt natively

  3. RavanH
    Member
    Posted 4 years ago #

    hmmm.. that is weird.

    i am not aware of any of my plugins to do such a thing! let me check...

  4. RavanH
    Member
    Posted 4 years ago #

    no, with all plugins switched off, i still see

    User-agent: *
    Disallow:

    when i visit my domain root appended with /robots.txt in a browser. if i change the public status of my blog (under Privacy) to 'Block search engines...' it changes to

    User-agent: *
    Disallow: /

    (which seems logical)

    however, if change the status of all posts to 'Draft', visiting myblogs.url/robots.txt results in a 404 page.

  5. Samuel B
    moderator
    Posted 4 years ago #

    ahhh..ok...my apologies - it does in this case
    admin - settings - privacy
    go here and set your blog for:
    "I would like my blog to be visible to everyone, including search engines (like Google, Sphere, Technorati) and archivers"

  6. RavanH
    Member
    Posted 4 years ago #

    well, yes. that's what i have selected normally.

    but my point is: when there are NO posts with status "published" on the blog (only pages) there is NO robots.txt content generated, no matter which option is selected on Settings > Privacy.

    what's the problem? you'd ask... i am working on a small sitemap plugin that automatically adds the sitemap url to the generated robots.txt content. however, if someone uses WP as a CMS with only pages (and no posts; which happens quite often i am sure) the is no auto-generated robots.txt available.

    pretty sure it's a bug. or am i mistaken?

  7. Samuel B
    moderator
    Posted 4 years ago #

    you need to create your own robots.txt file
    wordpress only creates it when you have it set not to be visited by search engines
    adding posts and such will never update a robots.txt

    http://codex.wordpress.org/Search_Engine_Optimization_for_Wordpress#Robots.txt_Optimization

  8. RavanH
    Member
    Posted 4 years ago #

    strange... in my experience WP always generates a robots.txt whether visibility is set to exclude search engines or not! it just changes the content from

    User-agent: *
    Disallow:<code>to</code>User-agent: *
    Disallow: /

    except... except when i have no posts (just pages) on the blog. in that case, there is no robots.txt generated but a 404 shown.

  9. RavanH
    Member
    Posted 4 years ago #

    anybody? (bump)

  10. blovett
    Member
    Posted 4 years ago #

    I can confirm this behavior. The virtual robots.txt file is not generated until you have posts. This is independent of your privacy settings.

    As a workaround, you can create a placeholder post that is privately published. Or, skip the virtual file altogether by creating a real robots.txt file.

  11. RavanH
    Member
    Posted 4 years ago #

    ahhhh... finaly! thanks blovett, for confirming. i was thinking i was mad ;)

    that private post solution is a good tip.

  12. cbeaudin
    Member
    Posted 4 years ago #

    Same behavior (2.9.1), no robots file regardless of privacy setting, haven't actually tried adding a post to see if it generates, will use my own.

  13. GregF
    Member
    Posted 3 years ago #

    I too have a site with only pages and no posts. I discovered that publishing a private post causes the virtual robots.txt to be generated again but only if you are logged in. However, publishing a password-protected post allows robots.txt to be generated for anonymous requests as well.

  14. RavanH
    Member
    Posted 3 years ago #

    thanks for that tip greg.

    the same behaviour is seen in WP 3.0 beta... i really wonder why they chose to make robots.txt dependant on posts.

  15. myBuddyonMaui
    Member
    Posted 3 years ago #

    I LOVE YOU GREG!!!

    I've been trying for a week to fix the problem with my robots.txt file returning a 404 error. Your tip on publishing a password-protected post is sheer GENIUS!

    I published a Post named "robots" set a password and WHAM! Works like a charm.

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags