Default wordpress robots.txt?

  • Resolved Peter_L

    (@peter_l)


    10 years, 8 months ago

    I have a site running with a custom theme. It has a robots.txt file (in root of theme folder) saying:

    User-Agent: *
Disallow: /wp-content/plugins/

    However, on calling the robots.txt from the url (www.mysite.com/robots.txt) it shows a robots.txt file containing following:

    User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

    So, it’s clearly not the one I added in the theme. Does anybody have an idea where it’s coming from (virtual or actual file).
    There’s no actual other robots.txt in any other folder, or at least I didn’t find one.
    Plugins: Better WP Security and WordPress SEO.

Viewing 5 replies - 1 through 5 (of 5 total)
  • Thread Starter Peter_L

    (@peter_l)

    10 years, 8 months ago

    Well, things have gotten worse. Google webmaster tools downloaded a version of robots.txt:

    User-agent: *
Disallow: /

    I’m not too concerned about this though. I reckon it’s just a previous version showing up and google is a bit slow to use the new version.

    What does concern me however is the above problem. Anybody ideas? I’m clueless.

    Thread Starter Peter_L

    (@peter_l)

    10 years, 8 months ago

    Ah, nevermind. Proper place for robots.txt is root of site, not theme folder.

    Closing the topic.

    As a final sidenote, WP does create it’s own default robots.txt. For more info see this http://wordpress.org/support/topic/robotstxt-set-to-disallow-cant-change?replies=35

    Krishna

    (@1nexus)

    10 years, 8 months ago

    In your robots.txt file,
    If you do NOT want to block anything use only:
    User-Agent: *

    If you want to block access to admin and includes files, use:

    User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

    As some plugin authors may be including URLs to other sites, some site owners exclude plugins also in which case, you may also add:
    Disallow: /wp-content/plugins/

    More info on can be had here:
    http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress
    https://support.google.com/webmasters/answer/156449?hl=en
    http://stackoverflow.com/questions/17029811/how-to-set-up-robots-txt-file-for-wordpress

    sidhusharma

    (@sidhusharma)

    10 years, 1 month ago

    I experienced a problem with my wordpress website. It took 20 days for me to build a website that suits my profession and looks classy but when I search it in google, it appears nowhere in SERP.
    I think my robots.txt file is not appropriate, it is showing ‘noindex, follow’
    When I tried changing the coding for my site, wordpress is not allowing me to change the coding .What shall I do and how can I make my website visible in the SERP?

    iwd1

    (@iwd1)

    9 years, 8 months ago

    In the WordPress admin section check Settings > Reading. At the bottom of the page there is Search Engine Visibility. Make sure the box is unchecked so search engines can index your site. A lot of times this is checked during development and not unchecked when the site goes live thus causing the entire site to not get indexed by search engines.

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Default wordpress robots.txt?’ is closed to new replies.