WordPress.org

Ready to get started?Download WordPress

Forums

What is a good robots.txt file for WordPress? (5 posts)

  1. John Hoff
    Member
    Posted 5 years ago #

    I have a basic idea what to put in my robots.txt file for WordPress, but I'd like to hear what others have to say. After doing a search online for a good "WordPress robots.txt file", it seems no one can agree on what to put in there.

    Some say to disallow wp-content, while others don't.

    Some say to disallow /categories while others don't.

    Anyone one care to chime in on what the best robots.txt file might be for WordPress?

    Thanks.

  2. Douglas

    Posted 5 years ago #

    I've always left mine untouched personally and have never had a problem with ranking well for appropriate search queries. I don't think you have much to lose by leaving it untouched and just letting the robots crawl your site.

  3. John Hoff
    Member
    Posted 4 years ago #

    Thanks douglashanna. For the most part I've always done the same. I was considering adding one, though. But I'll probably hold off as far as my WP blog goes.

  4. Saqib
    Member
    Posted 4 years ago #

    Sitemap: example.com/sitemap.xml

    User-agent: *
    
    Disallow: /wp-content/
    Disallow: /wp-icludes/
    Disallow: /trackback/
    Disallow: /wp-admin/
    Disallow: /archives/
    Disallow: /category/
    Disallow: /tag/*
    Disallow: /tag/
    Disallow: /wp-*
    Disallow: /login/
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.php$
    
    User-agent: All
    Allow: /
    
    User-agent: Googlebot-Image
    Disallow: /
    
    User-agent: ia_archiver
    Disallow: /
    
    User-agent: duggmirror
    Disallow: /
  5. BoganMarketing
    Member
    Posted 4 years ago #

    boganmarketing.com/blog/ultimate-wordpress-robots-txt-for-silo-seo.html

    WordPress Robots.txt for Silo SEO might be what your after.

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags

No tags yet.