Hey iceraven721,
Hope you are well!
From your posting it seems that you have multisite installation.
The problem might be if you forgot in your main site the option to discourage the web crawlers to index your site.
You can check this by going to your main site
Settings -> Reading
There should be a checkbox on the bottom of the page, just before the save button.
Another reason is that you might created your file with FTP client with diffent user than the web server uses, and now it can delete it, because it doesn’t have rights to do so.
If you did this, open your FTP client and check if robots.txt has the same permissions and ownership as the other files in your install.
Thanks,
Ivan
Thanks Ivan for the response. It’s only a single WordPress instance that is installed under an addon domain in cpanel which was the reason for the links. The search engine visibility option is unchecked to allow search engines in the reading section.
Thoughts?
Thanks for your help
I’m not seeing a website at http://thewebsite.com/, I’m getting redirected to http://thewebsite.com/cgi-sys/defaultwebpage.cgi
For future reference, creating an actual robots.txt file should always override the WordPress generated robots.txt response. The htaccess rules for WordPress only execute if the file does not exist.
Hey Sal,
That was just an example. The actual link is http://stj23.com/robots.txt. The disallow shows up in there but if I follow the cPanel path of http://208.167.238.68/~valorblo/stj23.com/robots.txt it shows the correct file.
I’m not sure what is going on but here are the next steps I’d suggest for troubleshooting.
1. While logged into your WordPress site, I’d go to Settings > Reading and confirm Search Engine Visibility is UNchecked.
2. I’d create a new file called test.txt in the stj23.com root directory that contains the words “Test file”
I’d then confirm this file is accessible at both
http://stj23.com/test.txt
and
http://208.167.238.68/~valorblo/stj23.com/test.txt
3. Can you please post the contents of your .htaccess file
That box is unchecked and I did a test file and it worked ok.
Here is the htaccess file
Thanks!
[Large code excerpt removed by moderator per forum rules. Please use the pastebin for all large code excerpts. It works better anyway.]
Can you please repost your .htaccess file using http://pastebin.com/ or https://gist.github.com/
I’m guessing the iThemes Security plugin and htaccess rules is causing this behavior.
I suggest:
1. make a backup file of your .htaccess file
2. turn off the iThemes Security plugin
3. confirm your .htaccess file has updated removing the iThemes lines (it should look like http://pastebin.com/vg37Pit0)
4. Check http://stj23.com/robots.txt and http://208.167.238.68/~valorblo/stj23.com/robots.txt
Something in there did it. Thanks for your help
