WordPress.org

Ready to get started?Download WordPress

Forums

sitename.com/robots.txt error 404 - not found (4 posts)

  1. Julius7
    Member
    Posted 5 years ago #

    Hello,

    whenever I type mysitename.com/robots.txt in my browser I get error 404 - not found. It's a new, clean installation of wordpress 2.8 with the following plugins active: contact form 7, customize(for freshy 2.0 theme), google xml sitemaps, wordpress.com stats and wp-mythumshot
    site is not yet properly indexed by google although it is only a few days old.
    I tried installing PC Robots.txt plugin but still get error 404 when trying to reach sitename.com/robots.txt
    why is this and what should I do? should I create robots.txt manually and add all content including sitemaps.xml to it? (and disable "Add sitemap URL to the virtual robots.txt file." in google sitemaps xml plugin)
    My site is not big, it's only like 5 or so pages in size.

  2. esmi
    Forum Moderator
    Posted 5 years ago #

    should I create robots.txt manually and add all content including sitemaps.xml to it? (and disable "Add sitemap URL to the virtual robots.txt file." in google sitemaps xml plugin)

    That's what I would do but then I'm used to creating a robots.txt file anyway.

  3. Julius7
    Member
    Posted 5 years ago #

    But why it's not created automatically(or virtual version of it)? And if I would to create it manually, what examples I can look at to learn how to create the best robots.txt file for search engines?

  4. esmi
    Forum Moderator
    Posted 5 years ago #

    A robots.txt file can cover an entire site - not just the directory containing WP.

    http://www.robotstxt.org/orig.html is the definitive guide. And there's a validator at http://tool.motoricerca.info/robots-checker.phtml.

    It's basically just a plain text file. I normally use a boilerplate file along the lines of:

    # robots.txt for http://your_domain/
    
    Sitemap: http://your_domain/sitemap.xml.gz
    
    User-agent: *
    Disallow: /css/
    Disallow: /cgi-bin/
    Disallow: /images/
    Disallow: /temp/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /themes/
    Disallow: /plugins/
    Disallow: /upgrade/
    Disallow: /uploads/

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags

No tags yet.