WordPress.org

1. Optimus
   Member
   Posted 11 months ago #

   i just need to ask is this roboto,txt is fien for my wordpress?

   User-agent: *
   Disallow: /cgi-bin
   Disallow: /wp-admin
   Disallow: /wp-includes
   Disallow: /wp-content/plugins
   Disallow: /wp-content/cache
   Disallow: /wp-content/themes
   Disallow: /trackback
   Disallow: /feed
   Disallow: /author
   Disallow: /comments
   Disallow: */trackback
   Disallow: */feed
   Disallow: */comments
   Disallow: /*?*
   Disallow: /*?
   Allow: /wp-content/uploads
   Alloww: /category/*/*
   
   # Google Image
   User-agent: Googlebot-Image
   Disallow:
   Allow: /*
   
   # Google AdSense
   User-agent: Mediapartners-Google*
   Disallow:
   Allow: /*
   
   User-agent: Googlebot-Mobile
   Allow: /
   
   User-agent: ia_archiver-web.archive.org
   Disallow: /
   
   # digg mirror
   User-agent: duggmirror
   Disallow: /
   
   Sitemap: http://omgbaby.net/sitemap.xml

   and wanna ask what this code does?

   Disallow: /*?*
   Disallow: /*?

   Please note: i made changes to my robots.txt and allowed Categories,i need to ask did i done it right way?

2. Optimus
   Member
   Posted 11 months ago #

   here i have another example. please advice which one is better.

   Sitemap: http://www.yoursite.com/sitemap.xml

   User-agent: *
   Disallow: /cgi-bin
   Disallow: /wp-content/plugins
   Disallow: /wp-content/cache
   Disallow: /wp-content/themes
   Disallow: /wp-admin/
   Disallow: /wp-icludes/
   Disallow: /trackback/
   Disallow: /archives/
   Disallow: /author
   Disallow: /*.js$
   Disallow: /*.inc$
   Disallow: /*.css$
   Disallow: /*.php$
   Allow: /wp-content/uploads

   User-agent: All
   Allow: /

   # Google Image
   User-agent: Googlebot-Image
   Disallow:
   Allow: /*

   # Google AdSense
   User-agent: Mediapartners-Google*
   Disallow:
   Allow: /*

   User-agent: Googlebot-Mobile
   Allow: /

   User-agent: ia_archiver-web.archive.org
   Disallow: /

   # digg mirror
   User-agent: duggmirror
   Disallow: /

3. Ipstenu
   Half-Elf Support Rogue & Mod
   Posted 11 months ago #

   Read this: http://perishablepress.com/wordpress-robots-rules/

4. SEO
   Member
   Posted 11 months ago #

   Disallow: /*.js$ instructs google and other search engines not to crawl any page with extension .js.

Reply

You must log in to post.