WordPress.org

Ready to get started?Download WordPress

Forums

What is the best way to handle robots.txt on Multisite? (4 posts)

  1. jtt89
    Member
    Posted 2 years ago #

    I know there are plugins, and some other things that people do, but is it straight the best just to keep one robots.txt in the root folder for all the sites in the network?

    Thank you.

  2. Depends on how your site is built. Most people only need the one.

  3. jtt89
    Member
    Posted 2 years ago #

    Thanks. I have the following cofiguration (I did not look at the names not to confuse anything):

    define( 'SUBDOMAIN_INSTALL', false );

    and I basically make new sites of main_domain.com, like

    main_domain.com/001
    main_domain.com/002
    main_domain.com/003 etc...

    I use domain mapping plugin on shared hosting, so I just went to "Parked Domains" in cPanel and then added this domain in Multisite configuration...

    I am gonna be starting with something like this:

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: /tag/
    Disallow: */trackback
    Disallow: */comments

    Disallow: /2006/
    Disallow: /2007/
    Disallow: /2008/
    Disallow: /2009/
    Disallow: /2010/
    Disallow: /2011/

    (Second part is just an example)But that's probably gonna be changing too...

    Thank you.

  4. So you're mapping domains? Yeah, one robots is great for that :) Full domains and subdomains are great with the one robots. If you put it in your root HTML, it'll easily handle all sites.

    (BTW, I wouldn't disallow the yearly archives, Google's smart enough to know it's not duplicate content, and it helps people find things.)

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags

No tags yet.