nemo-maritime
Member
Posted 11 months ago #
I have set a robots.txt so that all robots ignore my wp-content folder as well as wp-admin and wp-includes and google still indexes them. I want to make sure I am doing it right. This is my robots.txt file
User-agent: *
Disallow: /wp-login.php/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
# BEGIN XML-SITEMAP-PLUGIN
Sitemap: http://mywebsite.com/sitemap.xml.gz
# END XML-SITEMAP-PLUGIN
It should be following my robots.txt shouldnt it? I looked around and I see that this is how its supposed to be written. Please help, because its also indexing my plugins and other files I dont want people to have access to.
The file looks to be correct. I would also verify that the "robots.txt" file is in the root of your web directory. Here are some nice articles to read over for more info.
http://codex.wordpress.org/Search_Engine_Optimization_for_Wordpress
http://www.askapache.com/seo/wordpress-robotstxt-seo.html
Hope this helps!
Derek M.
nemo-maritime
Member
Posted 11 months ago #
Got it working, I beleive I had to take out the last forward slash, so instead of /wp-admin/ I put /wp-admin and it looks like its working now. By the way, thanks for the articles.
oceanofp
Member
Posted 9 months ago #
I have /wp-content disallow/ in my robot.txt too, but google is still displaying my images in their image search results.
