WordPress.org

Support

Support » How-To and Troubleshooting » What is the normal Robots.txt file should be and will search engine crawl my po?

What is the normal Robots.txt file should be and will search engine crawl my po?

  • Hi I am new to this WordPress and don’t know what would be the normal Robots.txt file would be ?

    My website is http://www.wideinfo.org and I have googled the search and add something to my Robots.txt the content is given below

    User-agent: *
    Disallow: /wp-content/
    Disallow: /wp-icludes/
    Disallow: /trackback/
    Disallow: /wp-admin/
    Disallow: /archives/
    Disallow: /category/
    Disallow: /tag/*
    Disallow: /tag/
    Disallow: /wp-*
    Disallow: /login/
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.php$
    
    Sitemap: http://www.wideinfo.org/feed

    Now will the search engine Crawl the Post pages ?

    At present what are the effects in search engine by the following above mentioned content in Robots.txt

Viewing 1 replies (of 1 total)
Viewing 1 replies (of 1 total)
  • The topic ‘What is the normal Robots.txt file should be and will search engine crawl my po?’ is closed to new replies.
Skip to toolbar