Google Robots & Indexing

  • Curtis Stevens

    (@gotmerchant)


    16 years, 5 months ago

    If I use these parameters in my robots file, do you see it creating any problems for Google to craw my blog? The reason why I’m doing this is Google has mentioned several times they may put an emphasis on how you rank in organic results on how fast your site loads and these areas of my blog seem not to be that fast loading, according to webmaster tools…

    Disallow: /blog/wp-admin/
    Disallow: /blog/wp-content/
    Disallow: /blog/wp-includes/

    Thanks
    Curtis

Viewing 3 replies - 1 through 3 (of 3 total)
  • WebStyley

    (@webstyley)

    16 years, 5 months ago

    it shouldn’t.

    i Use the following:

    User-agent: *
    # disallow all files in these directories
    Disallow: /cgi-bin/
    Disallow: /z/j/
    Disallow: /z/c/
    Disallow: /stats/
    Disallow: /dh_
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /contact/
    Disallow: /tag/
    Disallow: /wp-content/
    Disallow: /trackback/
    Disallow: /*?*
    Disallow: */trackback/

    User-agent: Googlebot
    # disallow all files ending with these extensions
    Disallow: /*.php$
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.gz$
    Disallow: /*.cgi$
    Disallow: /*.wmv$
    Disallow: /*.png$
    Disallow: /*.gif$
    Disallow: /*.jpg$
    Disallow: /*.cgi$
    Disallow: /*.xhtml$
    Disallow: /*.php*
    Disallow: */trackback*
    Disallow: /*?*
    Disallow: /z/
    Disallow: /wp-*
    Allow: /wp-content/uploads/

    User-agent: Googlebot-Image
    Allow: /*

    # allow adsense bot on entire site
    User-agent: Mediapartners-Google*
    Disallow: /*?*
    Allow: /z/
    Allow: /about/
    Allow: /contact/
    Allow: /wp-content/
    Allow: /tag/
    Allow: /manual/*
    Allow: /docs/*
    Allow: /*.php$
    Allow: /*.js$
    Allow: /*.inc$
    Allow: /*.css$
    Allow: /*.gz$
    Allow: /*.cgi$
    Allow: /*.wmv$
    Allow: /*.cgi$
    Allow: /*.xhtml$
    Allow: /*.php*
    Allow: /*.gif$
    Allow: /*.jpg$
    Allow: /*.png$

    # disallow archiving site
    User-agent: ia_archiver
    Disallow: /

    # disable duggmirror
    User-agent: duggmirror
    Disallow: /

    askmequestion

    (@askmequestion)

    16 years, 1 month ago

    It shouldn’t prevent google from crawling your website and should not affect your ranking. using these commands you mentioned:

    Disallow: /blog/wp-admin/
Disallow: /blog/wp-content/
Disallow: /blog/wp-includes/

    you are simply disabling a login-based webpages that google cann’t crawl in the first place, so if anything you are doing them a favor.

    ____________
    Mike
    SEO Software

    Susancai

    (@susancai)

    15 years, 9 months ago

    It’s a good site.

Viewing 3 replies - 1 through 3 (of 3 total)

The topic ‘Google Robots & Indexing’ is closed to new replies.