WordPress.org

Ready to get started?Download WordPress

Forums

Robots.txt problem Need help (9 posts)

  1. noumanniazi
    Member
    Posted 2 years ago #

    Hi,
    I am new at WordPress. My site is working fine from last few days and Google crawls all my posts but now i got red mark on robots.txt in webmaster and my last few posts doesn't crawl. I also check privacy and select "Allow search engines to index this site." I have also installed Sitemap.xml plugin and select make virtual robots.txt. Now i have installed another Special plugin for Robots.txt named as "WP Robots Txt". When i open mysite.com/robots.txt it says

    # Added by SEO Ultimate's Link Mask Generator module
    User-agent: *
    Disallow: /go/
    # End Link Mask Generator output

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Sitemap: http://www.mysite.com/sitemap.xml.gz

    I want Google to crawl my site what should i do now.

  2. Krishna
    Volunteer Moderator
    Posted 2 years ago #

    Go here for Google's guidelines.

  3. noumanniazi
    Member
    Posted 2 years ago #

    Thanks for response..
    I just want to know that my site is enable for crawling or not. In case of above Google can crawl my site???...

  4. Krishna
    Volunteer Moderator
    Posted 2 years ago #

    Read through the link I provided. Various options are discussed there. Select the one you want. For instance, to block the entire site, use a forward slash (Disallow: /). Like this see other options.

  5. noumanniazi
    Member
    Posted 2 years ago #

    According to your link my site is crawling by Google because i only select admin and includes in robots.txt. But somehow my site isn't crawling by Google what should i do.Now some of my recent posts are displaying in Google and some are not.

  6. Krishna
    Volunteer Moderator
    Posted 2 years ago #

    Use Webmaster Tools and Google Analytics to know what is really happening to your site and which posts/ pages are indexed/ crawled by Google. You will get all reports, answers, etc. there. Forget what others say about it.

  7. noumanniazi
    Member
    Posted 2 years ago #

    I tried but and now i am conform that robot.txt file is fine but still getting red mark on Robots.txt fetch in Webmaster/tools.......

  8. Krishna
    Volunteer Moderator
    Posted 2 years ago #

    It takes time.

  9. noumanniazi
    Member
    Posted 2 years ago #

    Thanks for response.

Topic Closed

This topic has been closed to new replies.

About this Topic