WordPress.org

Ready to get started?Download WordPress

Forums

What is the normal Robots.txt file should be and will search engine crawl my po? (2 posts)

  1. sujoydhar
    Member
    Posted 2 years ago #

    Hi I am new to this WordPress and don't know what would be the normal Robots.txt file would be ?

    My website is http://www.wideinfo.org and I have googled the search and add something to my Robots.txt the content is given below

    User-agent: *
    Disallow: /wp-content/
    Disallow: /wp-icludes/
    Disallow: /trackback/
    Disallow: /wp-admin/
    Disallow: /archives/
    Disallow: /category/
    Disallow: /tag/*
    Disallow: /tag/
    Disallow: /wp-*
    Disallow: /login/
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*.php$
    
    Sitemap: http://www.wideinfo.org/feed

    Now will the search engine Crawl the Post pages ?

    At present what are the effects in search engine by the following above mentioned content in Robots.txt

  2. Default is just to block wp-includes and wp-admin. Blocking wp-content can cause issues.

    I'd use http://perishablepress.com/wordpress-robots-rules/

Topic Closed

This topic has been closed to new replies.

About this Topic