WordPress.org

1. ltaliga
   Member
   Posted 1 year ago #

   Hi, I'm a bit confused about robots.txt and sitemaps, and can't figure this out from looking at previous answers.

   I have a blog in a subfolder of my domain.

   1. I created a site.xml file for the main domain. But it didn't pick up the pages in the blog folder.
   2. I also created a site.xml for the blog using a Wordpress plugin. So do I need to put a robots.txt file in the root folder AND the blog folder?
   3. Something isn't right because Google Webmaster tool keeps saying robots.txt unreachable.

   Hope you can help?
   Thanks

2. adamrbrown
   Member
   Posted 1 year ago #

   I also created a site.xml for the blog using a Wordpress plugin. So do I need to put a robots.txt file in the root folder AND the blog folder?

   Sitemaps and robots.txt are entirely different technologies. There is no connection between them for all intents and purposes.

   No search engine will look for a robots.txt anywhere except for http://example.com/robots.txt (or http://blog.example.com/robots.txt). If it's in http://example.com/blog/robots.txt, they won't look for it.

   3. Something isn't right because Google Webmaster tool keeps saying robots.txt unreachable.

   URL?

3. ltaliga
   Member
   Posted 1 year ago #

   http://www.virtualpabusiness.com

4. adamrbrown
   Member
   Posted 1 year ago #

   When I load up http://www.virtualpabusiness.com/robots.txt I see nothing. That's why Google says it's not getting anything.

   That's the only URL that Google will check for a robots.txt file.

5. ltaliga
   Member
   Posted 1 year ago #

   I had uploaded a blank text file to the root, because someone advised me to, as an interim measure.

   So it's not OK to have a blank text file as a robots.txt file?

6. adamrbrown
   Member
   Posted 1 year ago #

   Well, it's okay, but it means that there's not really a robots.txt file there as far as the search engines are concerned.

   But if you don't intend to write anything in robots.txt, then it really doesn't matter whether google can find it. Just having it there doesn't help your pagerank in any way.

7. ltaliga
   Member
   Posted 1 year ago #

   I'll have to keep trying with Google Webmaster because it says robots.txt unreachable although I do have a blank text file in the root folder.

   I'm doing this so that my sitemap is submitted to Google but it won't let me because it can't reach the robots.txt file at the moment.

   Thanks

8. ltaliga
   Member
   Posted 1 year ago #

   Thanks for your help, everything is fine now!

Topic Closed

This topic has been closed to new replies.