Hi I am new to this WordPress and don't know what would be the normal Robots.txt file would be ?
My website is http://www.wideinfo.org and I have googled the search and add something to my Robots.txt the content is given below
User-agent: * Disallow: /wp-content/ Disallow: /wp-icludes/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /archives/ Disallow: /category/ Disallow: /tag/* Disallow: /tag/ Disallow: /wp-* Disallow: /login/ Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*.php$ Sitemap: http://www.wideinfo.org/feed
Now will the search engine Crawl the Post pages ?
At present what are the effects in search engine by the following above mentioned content in Robots.txt