WordPress.org

1. maxlk
   Member
   Posted 11 months ago #

   Hi this is my 1st post here. just a quick question. is it normal for spiders to go and index admin-ajax.php and plugins folder?

2. stvwlf
   Member
   Posted 11 months ago #

   Paste this into a text file
   

   User-agent: *
   Disallow: /

   and save it with the name robots.txt

   Put a copy of that file in all folders you don't want spiders in, such as /wp-admin/ and /wp-content/plugins/

   Normally no they don't index those folders. Rather than figure out why they have, putting the robots.txt file in the folders you don't want indexed will cause all behaving spiders ( google, bing, etc) from indexing content in that folder and any files/folders beneath it. So putting it in /wp-content/plugins/ prevents any folders/files below it from being indexed also.

   Spammer spiders etc do not honor the robots.txt file. Legit search engines do.

3. ClaytonJames
   Member
   Posted 11 months ago #

   You may want to confine the robots.txt file to just your top level directory. Here is some info on how to create the file, and where it should be placed. http://www.robotstxt.org/robotstxt.html

   There is also a good article in the codex that helps to explain how to optimize the robots.txt file for WordPress, with a link to an example file.
   http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress#Robots.txt_Optimization

4. maxlk
   Member
   Posted 11 months ago #

   hey guys thanks for the replies. i did that before i posted here. but i have lot of wordpress sites. non of the other sites have this problem. so was wondering why is that. thats why i ask. thanks for all ur replies.

5. stvwlf
   Member
   Posted 11 months ago #

   Most likely it has something to do with the configuration on that site or server. That's why i suggested the robots.txt. It resolves the problem without needing to explore why it exists.

   Assuming its an Apache site, do you have the line
   Options -Indexes
   in the .htaccess file in the root level of the webspace folder?

6. maxlk
   Member
   Posted 11 months ago #

   root level doesn't have a .htaccess. and i do run 2 more WP sites on the same sever. this is the site. http://www.genxscripts.com

   i did disallow most of the things. i didnt do for plugins folder. so u can see if u google site:www.genxscripts.com. last result shows

   Index of /wp-content/plugins/mingle-forum/default-skin/Default

   Index of /wp-content/plugins/mingle-forum/default-skin

   thanks

7. maxlk
   Member
   Posted 11 months ago #

   BTW site was on 2nd page of the keyword "web hosting review script" till 15th of this month. and now i can find my site for that keyword after the last cowling on 15th. do you think its something to do with the theme. coz i change the theme few days before that happen

8. stvwlf
   Member
   Posted 11 months ago #

   I went to http://www.genxscripts.com/wp-content/plugins/ and see no listing of folder contents so you don't need to add the line to htaccess

   It takes Google a while when you put robots.txt file in place before they take your files out of the index. It doesn't happen overnight.

   I am not an SEO expert so I hesitate to answer your question about ranking. It depends on how the new theme displays your existing content. The content is the same. Its the tags that may change - H1, H2, etc. Depending on how the new theme is set up.

9. maxlk
   Member
   Posted 11 months ago #

   ah ok. thank you so much. i will wait and see how it goes and post back here.

10. ClaytonJames
    Member
    Posted 11 months ago #

    @maxlk

    You may already be aware of this, but there is some information and tools located here - Webmaster Tools Help - that might be of interest to you. The link for instructions on how to test your robots.txt file is under the "What do you want to do?" heading.

11. maxlk
    Member
    Posted 11 months ago #

    @ClaytonJames thanks. already done that. just wanted to know why it happen.

Reply

You must log in to post.