bloggin360Hello, my blog url is techscrunch.com. I recently installed XML Sitemap Generator for WordPress. I have checked this option in Privacy in WordPress :"I would like my site to be visible to everyone, including search engines (like Google, Bing, Technorati) and archivers" but my site is not accessible by Google crawler. When i go to techscrunch.com/robots.txt it says User-agent: *
Disallow:
Sitemap: http://techscrunch.com/sitemap.xml.gz
Temporarily, I have added a real robots.txt file which allows the crawler to acces my site which was created through Analytics. I believe that Xml Sitemap Generator does not work when a real robots.txt file exists, but it shows no error in the settings page. What values should I change in sitemap.xml.gz file to allow crawler to acces my site when I am using only a virtual robots.txt. Why isnt WordPress allowing crawlers to crawl my site with the virtual robots.txt even though i have checked the option in the privacy settings?
