Support » Networking WordPress » Disabling the index process on the Internet with one of the sites

  • Resolved dumitrudany

    (@dumitrudany)


    Hi all,

    I wanna know if I using multisite like mydomain.com and the other one mydomain.com\smtg\ could be a problem for the first site (of been indexed) if in the last one I disable the search engine to index my site? I doing that through the path Settings>Reading>Discourage search engines from indexing this site to the second site.

    I mean, in the second site there is another wordpress installed :). Is this enough to not receiving visitors? or how can I simply point someone to a page from the last site
    mydomain.com\smtg\link(.html)
    and nobody from the internet should find it. Only with the link to that location.

Viewing 3 replies - 1 through 3 (of 3 total)
  • A search engine may only look for a robots.txt in the main domain. WordPress will make one in the subdir you have set up, but search engines may not look for it there.

    Search engine will take your link:

    mydomain.com/smtg/link.html

    but look for robots.txt here:

    mydomain.com/robots.txt

    Therefore you need to add the subdir smtg into your main robots.txt disallow rules:

    User-agent: *
    Disallow:
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-login.php
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /comments
    Disallow: /smtg

    How to add rules to the automagic robots.txt? Plugins will do that.

    Examples: http://wordpress.org/plugins/tags/robotstxt

    Thread Starter dumitrudany

    (@dumitrudany)

    you’re right !
    Many thanks. I used WP Robots Txt plugin, easily to modify the virtual robots.txt file. I understand the stuffs 😉

    Do you know if it is necessary to introduce in robots.txt each link ?
    I mean I wrote this “Disallow: /smtg” and the question is:
    Would be any subpage to the main /smtg as in the example
    mydomain.com/smtg/link.html skipped by the robot crawler, isn’t it? Or will be necessary to introduce in robots.txt each subpage of the /smtg/ ?

    Regards,
    Dan

    Google webmaster tools has a robots.txt checker, there are others “out there”, too.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Disabling the index process on the Internet with one of the sites’ is closed to new replies.