WordPress.org

Ready to get started?Download WordPress

Forums

robots.txt (2 posts)

  1. westie
    Member
    Posted 7 years ago #

    i'm trying to do some seo on my wordpress blogs

    i'm thinking of adding a robots.txt file

    - i use xsitepro to create my websites - this automatically creates a robots.txt file - can i upload the same one to the blog?

    - is there a plugin that would create the robots.txt file for me?

    - i've been doing some searching and found a download wordpress robots.txt file - see below

    --------------

    User-agent: *
    # disallow all files in these directories
    Disallow: /cgi-bin/
    Disallow: /z/
    Disallow: /stats/
    Disallow: /dh_
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /contact/
    Disallow: /tag/
    Disallow: /wp-content/b
    Disallow: /wp-content/p
    Disallow: /wp-content/themes/askapache/4
    Disallow: /wp-content/themes/askapache/c
    Disallow: /wp-content/themes/askapache/d
    Disallow: /wp-content/themes/askapache/f
    Disallow: /wp-content/themes/askapache/h
    Disallow: /wp-content/themes/askapache/in
    Disallow: /wp-content/themes/askapache/p
    Disallow: /wp-content/themes/askapache/s
    Disallow: /trackback/
    Disallow: /2006/
    Disallow: /2007/
    Disallow: /*?*
    Disallow: */trackback/

    User-agent: Googlebot
    # disallow all files ending with these extensions
    Disallow: /*.php$
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.gz$
    Disallow: /*.cgi$
    Disallow: /*.wmv$
    Disallow: /*.png$
    Disallow: /*.gif$
    Disallow: /*.jpg$
    Disallow: /*.cgi$
    Disallow: /*.xhtml$
    Disallow: /*.php*
    Disallow: */trackback*
    Disallow: /*?*
    Disallow: /z/
    Disallow: /wp-*
    Allow: /wp-content/uploads/

    # allow google image bot to search all images
    User-agent: Googlebot-Image
    Allow: /*

    # allow adsense bot on entire site
    User-agent: Mediapartners-Google*
    Disallow: /*?*
    Allow: /about/
    Allow: /contact/
    Allow: /wp-content/
    Allow: /tag/
    Allow: /manual/*
    Allow: /docs/*
    Allow: /*.php$
    Allow: /*.js$
    Allow: /*.inc$
    Allow: /*.css$
    Allow: /*.gz$
    Allow: /*.cgi$
    Allow: /*.wmv$
    Allow: /*.cgi$
    Allow: /*.xhtml$
    Allow: /*.php*
    Allow: /*.gif$
    Allow: /*.jpg$
    Allow: /*.png$

    # disallow archiving site
    User-agent: ia_archiver
    Disallow: /

    # disable duggmirror
    User-agent: duggmirror
    Disallow: /

    Sitemap: http://www.askapache.com/sitemap.xml

    --------------------

    is this an 'across the board' robots.txt file, or do i need to customise/personalise it in any way?

    do robots.txt files help in seo?????????????

    would appreciate your advice!

    kathryn

  2. AskApache
    Member
    Posted 7 years ago #

    That is more of a learning example to show what is possible and how it works. Check out the Updated robots.txt ffor WordPress.

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/askapache/
    Disallow: /trackback/
    Disallow: /2006/
    Disallow: /2007/
    Disallow: */trackback/
    Disallow: */feed/
    
    User-agent: Googlebot
    Allow: /wp-content/uploads/
    
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*
    
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*
    
    User-agent: ia_archiver
    Disallow: /
    
    User-agent: duggmirror
    Disallow: /
    
    # Does anyone care I love Google Apache htaccess
    
    Sitemap: http://www.askapache.com/sitemap.xml

Topic Closed

This topic has been closed to new replies.

About this Topic