WordPress.org

Ready to get started?Download WordPress

Forums

Categories and Tags - Crawl or Not to Crawl (1 post)

  1. robertallen
    Member
    Posted 1 year ago #

    The /category/ and /tag/ folders in WordPress (or I should say, URL's), is it SEO friendlier to crawl them, or omit them via the robots.txt file?

    This is my current robots.txt file...

    User-agent:  Googlebot
    # Slow the bot down to speed up my site
    Crawl-delay: 5
    # Stop access to these directories
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: /wp-admin/
    Disallow: /cgi-bin/
    # Avoid duplicate content and files with certain extensions
    Disallow: /index.php
    Disallow: /*?
    Disallow: /*.php$
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.gz$
    Disallow: /*.cgi$
    Disallow: /*.wmv$
    Disallow: /feed/
    Disallow: /*/feed/
    Disallow: /trackback/
    Disallow: /*/trackback/
    Disallow: /tag/
    Disallow: /*/tag/
    
    #For other bots I allow feed and slow it down a bit
    User-agent: *
    Crawl-delay: 10
    Disallow: /wp-content/
    Disallow: /wp-includes/
    Disallow: /wp-admin/
    Disallow: /cgi-bin/
    Disallow: /trackback/
    Disallow: /index.php
    Disallow: /tag/
    
    Sitemap: http://www.mydomain.com/sitemap.xml.gz

Topic Closed

This topic has been closed to new replies.

About this Topic