WordPress.org

1. John Hoff
   Member
   Posted 2 years ago #

   I have a basic idea what to put in my robots.txt file for WordPress, but I'd like to hear what others have to say. After doing a search online for a good "WordPress robots.txt file", it seems no one can agree on what to put in there.

   Some say to disallow wp-content, while others don't.

   Some say to disallow /categories while others don't.

   Anyone one care to chime in on what the best robots.txt file might be for WordPress?

   Thanks.

2. Douglas
   Administrator
   Posted 2 years ago #

   I've always left mine untouched personally and have never had a problem with ranking well for appropriate search queries. I don't think you have much to lose by leaving it untouched and just letting the robots crawl your site.

3. John Hoff
   Member
   Posted 2 years ago #

   Thanks douglashanna. For the most part I've always done the same. I was considering adding one, though. But I'll probably hold off as far as my WP blog goes.

4. saaqi
   Member
   Posted 2 years ago #

   Sitemap: example.com/sitemap.xml
   

   User-agent: *
   
   Disallow: /wp-content/
   Disallow: /wp-icludes/
   Disallow: /trackback/
   Disallow: /wp-admin/
   Disallow: /archives/
   Disallow: /category/
   Disallow: /tag/*
   Disallow: /tag/
   Disallow: /wp-*
   Disallow: /login/
   Disallow: /*.js$
   Disallow: /*.inc$
   Disallow: /*.css$
   Disallow: /*.php$
   
   User-agent: All
   Allow: /
   
   User-agent: Googlebot-Image
   Disallow: /
   
   User-agent: ia_archiver
   Disallow: /
   
   User-agent: duggmirror
   Disallow: /

5. BoganMarketing
   Member
   Posted 1 year ago #

   boganmarketing.com/blog/ultimate-wordpress-robots-txt-for-silo-seo.html

   WordPress Robots.txt for Silo SEO might be what your after.

Topic Closed

This topic has been closed to new replies.