magicalwondersI'm working on a design at this url: http://www.mylessinclair.com/about/ and wanted to see how it looked in other browsers. The site I used for this function returned an error saying that a robots.txt file disallowed access.
When I FTP the site, there is no robots.txt file present in the /public_html folder. It seems the file is being created dynamically through the rewite.php file in the wp-includes directory.
My first thought was that the output is being affected by the selection under Privacy Settings in the WP Admin panel. However, when I change the selection to show
I would like my blog to be visible to everyone, including search engines (like Google, Bing, Technorati) and archivers
The output of the robots.txt remains exactly the same. I'm guessing the above settings only affects the meta information on the pages produced.
So, I'm a bit confused as to what the currently produced robots.txt file is actually achieving? and how do I edit it? Although the rewrite.php file seems to be generating the file, it doesn't appear to include the current output!
Hope someone can advise.
Myles
