WordPress.org

Ready to get started?Download WordPress

Forums

To Allow Robots to search WP-ADMIN and WP-INCLUDES? (2 posts)

  1. russface
    Member
    Posted 1 year ago #

    I got an email from Adsense saying:
    -----------------------
    We detected 39390 failed crawl requests last week, caused by a misconfigured robots.txt file that's blocking certain pages of your site from the AdSense crawler. Since these pages already have AdSense code on them, you'll want to allow the AdSense crawler so that the most relevant ads will be displayed.

    You can fix this issue today by editing your robots.txt file and granting the AdSense crawler the ability to access these blocked pages. To do this, find your robots.txt file and add the following two lines of text at the very top:

    User-agent: Mediapartners-Google
    Disallow:
    -----------------------

    Right now my Robots.txt file is the default...

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    So is there a risk to allowing google to search EVERYTHING? Should I ADD those lines? Or just replace what is already there?

    Thanks!

  2. MickeyRoush
    Member
    Posted 1 year ago #

    Try this instead:

    # Allow AdSense bot on entire site
    User-agent: Mediapartners-Google
    Disallow: /*.php$
    Disallow: /*.html$
    Disallow: /*.doc$
    Disallow: /*.flv$
    Disallow: /*.swf$
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Allow: /index.php

Topic Closed

This topic has been closed to new replies.

About this Topic