Robots.txt for subdomains
-
Hello guys,
I have a question regarding robots.txt in multisite.
When setting up a multisite network and creating a subdomain, you must create a Vhost for each subdomain. However, WP does not create any folder /name-subdomain/ within the root folder (main site), and even with the subdomain creation, the robots.txt will continue to exist only one that is located at the root of the site. Already the robots.txt of the subdomain, access the same robots.txt from the main site.
For example, my root site robots.txt is http://proddigital.com/robots.txt and mine if I try to access the robots.txt from my subdomain http://idiomas.proddigital.com.br/robots.txt is the same robotx.txt from the root site.
Is this normal and created by WP itself?
As the robots.txt subdomain accesses the robots.txt from the root site, then it is accessing the root site’s sitemap as well. Will not that confuse search engines? Because the search engines, when accessing http://idiomas.proddigital.com.br/robots.txt will see the sitemap http://proddigital.com.br/sitemap_index.xml. Is not that okay? Because the sitemap of the subdomain exists, and this is http://idiomas.proddigital.com.br/sitemap_index.xml.
If you notice, the sitemap of the root site (http://proddigital.com.br/sitemap_index.xml) there is no link to the pages of the subdomain. So, if the search engines access the robots.txt of the subdomain (http://idiomas.proddigital.com.br/robots.txt), it will crawl the sitemap of the root site (http://proddigital.com.br/sitemap_index.xml) and not the sitemap http://idiomas.proddigital.com.br/sitemap_index.xml
Should not the correct one have robots.txt only for the subdomain, so that there is a subdomain only sitemap?
The topic ‘Robots.txt for subdomains’ is closed to new replies.