To Allow Robots to search WP-ADMIN and WP-INCLUDES?

  • russface

    (@russface)


    11 years, 7 months ago

    I got an email from Adsense saying:
    ———————–
    We detected 39390 failed crawl requests last week, caused by a misconfigured robots.txt file that’s blocking certain pages of your site from the AdSense crawler. Since these pages already have AdSense code on them, you’ll want to allow the AdSense crawler so that the most relevant ads will be displayed.

    You can fix this issue today by editing your robots.txt file and granting the AdSense crawler the ability to access these blocked pages. To do this, find your robots.txt file and add the following two lines of text at the very top:

    User-agent: Mediapartners-Google
    Disallow:
    ———————–

    Right now my Robots.txt file is the default…

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    So is there a risk to allowing google to search EVERYTHING? Should I ADD those lines? Or just replace what is already there?

    Thanks!

Viewing 1 replies (of 1 total)
  • MickeyRoush

    (@mickeyroush)

    11 years, 7 months ago

    Try this instead:

    # Allow AdSense bot on entire site
User-agent: Mediapartners-Google
Disallow: /*.php$
Disallow: /*.html$
Disallow: /*.doc$
Disallow: /*.flv$
Disallow: /*.swf$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Allow: /index.php
Viewing 1 replies (of 1 total)
  • The topic ‘To Allow Robots to search WP-ADMIN and WP-INCLUDES?’ is closed to new replies.