Support » Plugin: Virtual Robots.txt » [Plugin: PC Robots.txt] Why isn't virtual robots.txt file detected by Google Webmaster Tools?

  • Resolved jfl1

    (@jfl1)



    Google Webmaster Tools doesn’t detect the robots.txt file created by PC-robots.txt

    Is that normal?

    Also, when I click « preview your robots.txt file » in the extension’s page, I get a 404 page. could that be because my WordPress is installed in a folder (not in the domain root)? If that’s the case, is there a workaround?

Viewing 4 replies - 1 through 4 (of 4 total)
  • Have a link so we can see the Robots.txt or error?

    The link is :
    http://www.sceptique.ca

    Do not see any robots.txt. Do not know what PC-robots.txt
    is. If I can’t see it, Google can not either. Also you seem to redirect everything in root to /wp2. Not sure why you do that but up to you.
    And since your sitemap.xml is in /wp2 you need a robots.txt in the root for autodiscovery to point to the sitemap like this:
    Sitemap: http://www.sceptique.ca/wp2/sitemap.xml

    Do not know how you are doing the redirect but you may (after creating one) need to exclude robots.txt in it.

    Just looked at the plugin. The plugin probably tries to create a robots.txt in the WP directory which is in /wp2 on yours. That will not work as it needs to be in the root. You can just create a robots.txt with any text editor but you will need to place it in the root of your site. They are very easy to setup. Simple rules:
    *User-agent: the robot the following rule applies to
    *Disallow: the URL you want to block

    As an example for yourself:

    User-agent: *
    Disallow: /privatedirectory/
    User-agent: baiduspider
    Disallow: /
    Sitemap: http://www.sceptique.ca/wp2/sitemap.xml

    Allows all spiders to everything except /privatedirectory
    Blocks Baidu from spidering anything on your site.
    Points to your Sitemap.xml

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘[Plugin: PC Robots.txt] Why isn't virtual robots.txt file detected by Google Webmaster Tools?’ is closed to new replies.