So I'm new to robot.txt, and have a question;
I read that it's good SEO to add the following code to your robot.txt file;
OK, so I didn't really know how to do that, but luckily I found a place to enter this with the "SEO Ultimate" plugin. Easy, you just go to the File Editor section and enter the code into the correct field and it loads the code to your robot.txt file. I checked the file at http://www.example.com/robot.txt and all looked like it was supposed to.
Then I went to do the same thing on my blog. I have a separate WP installation on a subdirectory like http://www.example.com/blog. Now, I also read for this, the following code should be placed;
Now, the problem is, when I do this on the blog, and try to look up my http://www.example.com/blog/robot.txt file, I get a 404 error instead of the correct script.
Should I be putting the above code into the http://www.example.com/robot.txt file and not the subdirectory?
What am I doing wrong?
Any help would be great!