WordPress.org

Ready to get started?Download WordPress

Forums

robot.txt question (9 posts)

  1. startribe
    Member
    Posted 7 years ago #

    Hey Community,

    I am putting together a robot.txt file, and I wanted to confirm something in regards to some advice from this link:
    < a href="http://www.askapache.com/seo/wordpress-robotstxt-optimized-for-seo.html">
    wordpress-robotstxt-optimized-for-seo

    I noticed the disallow command for these locations:
    Disallow: /cgi-bin/
    Disallow: /z/j/
    Disallow: /z/c/
    Disallow: /stats/
    Disallow: /dh_

    These are not located in my html folder, i.e., my server goes something like this:
    var/www/html

    Now would I need to locate and change the paths to these if they are not in my html public folder?

    Also, for the disallow location of the wp directories:
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    & etc...

    Should I update my path to stem from my html/web folder to the location of these directories, e.g., if my wp files are located in "content" then change:
    /wp-admin/
    to
    /content/wp-admin/

    Any advice is greatly appreciated.

    Sincerely,
    Orion

  2. whooami
    Member
    Posted 7 years ago #

    is this your box?

    /var/www is usually a symlink, and shouldnt for these purposes used that way.

    In other words, your 'path' isnt www, it's public_html or something equivalent. I mention this because your robots.txt should NOT be placed inside the symlink www/ it needs to go inside public_html or its equivalent

    as far as your questions:

    Should I update my path to stem from my html/web folder to the location of these directories, e.g., if my wp files are located in "content" then change:
    /wp-admin/
    to
    /content/wp-admin/

    yes.

    really to simplify this, there is a plugin:

    http://wordpress.org/extend/plugins/kb-robotstxt/

  3. startribe
    Member
    Posted 7 years ago #

    Thanks for the reply, whooami. Ok, I checked out the plugin and I uploaded it. I then entered my robots.txt with out the added directory info and clicked the update button, but when I viewed the robots.txt file, the plugin did not add "/Content" to "/wp-admin/". Should I expect the plugin to add the associated directory?

    My WordPress Address is "example.com/content" and my Blog Address URL is "example.com"... Does this mean that "/content" isn't needed for my type of install?

    Thank you again.

  4. whooami
    Member
    Posted 7 years ago #

    thats corrent, HOWEVER, your robots.txt needs to go inside the upper most direcotry at your web-root.

    In other words, if you have a blog @ example.com/content you still need that ..

    Follow?

    If you are using the "wordpress in its own folder" option and redirecting so that example.com/ leads to your blog, you still need to have the robotx.txt available here:

    example.com/robots.txt

    And since you are redirecting the /content is unecessary.

    Remember, though, robots.txt files go in your web_root, which isnt always your blog_root.

  5. startribe
    Member
    Posted 7 years ago #

    oh man... ok, I think I got it...

    1. robots.txt always goes in the root regardless of the install
    2. the associated paths to the directories in the robots.txt folder do not need the actual path of the wp-install if I am redirecting from the root, i.e., if my wp url is victorybelt.com and my location of the install is "/content".

    Thank you so much for your help. Your knowledge is appreciated. I am also interested in your thoughts on these two methods of robots.txt SEO:
    Simple Robots.txt
    Elaborate Robots.txt

    Which would you prefer, the simple or more elaborate?

  6. whooami
    Member
    Posted 7 years ago #

    I got askimeted, heres my reply:

    yes, you got it exactly! Think about what you see when you go to your site, and you are all set :)

    As far as seo and a robots.txt goes.. There are going to be those that disagree with me, BUT **generally speaking**, using a robots.txt will not help SEO, if you define SEO by an increase in traffic.

    Think about googlebot for a second. It follows links. Nothing else nothing more. In the case of the second post you linked above, they suggest something similar to this:

    Disallow: /wp-content/themes/askapache/4
    Disallow: /wp-content/themes/askapache/c
    Disallow: /wp-content/themes/askapache/d
    Disallow: /wp-content/themes/askapache/f
    Disallow: /wp-content/themes/askapache/h
    Disallow: /wp-content/themes/askapache/in
    Disallow: /wp-content/themes/askapache/p
    Disallow: /wp-content/themes/askapache/s

    Where on any wordpress install is there a LINK to themes/ that Googlebot has standard access to ???

    There isnt.

    And none of what that poster writes provides anything that might aid SEO.

    How **might** one use robots.txt to help SEO?

    By preventing your feed from showing up in Google.

    There's an example of that on the first link.

    By potentially blocking multipage views, in other words, directing searchers to the permalink post instead of the pages views we so often see with categorys and archives.

    I'll tell you .. I can even show you what I do.

    http://www.village-idiot.org/robots.txt

    I dont have and never have had an issue with duplicate posts in Google. Case in point:

    http://www.google.com/search?hl=en&q=wp-chunk&btnG=Search

    Yes, my feed is there, thats by choice :) Googlebot is smarter than we all think.

  7. startribe
    Member
    Posted 7 years ago #

    edit: your complete reply just showed up... reading it now.

  8. whooami
    Member
    Posted 7 years ago #

    no, it means my first reply was caught by askimet on here, so I reposted it.

    I should add that when speaking about duplicate posts, one can ALWAYS do the following:

    http://www.google.com/search?as_q=&hl=en&num=10&btnG=Google+Search&as_epq=wp+chunk&as_oq=&as_eq=&lr=&as_ft=i&as_filetype=&as_qdr=all&as_nlo=&as_nhi=&as_occt=any&as_dt=i&as_sitesearch=village-idiot.org&as_rights=&safe=images

    which will, of course, return duplication.

    My point in the google link in the previous post is to show that Google and its bot have a way of taking care of things if your site is set up properly and 'standard' search queries are used..

  9. startribe
    Member
    Posted 7 years ago #

    Awesome, well said. Thanks!

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags