Support » Fixing WordPress » Robots.txt problem Need help

  • Hi,
    I am new at WordPress. My site is working fine from last few days and Google crawls all my posts but now i got red mark on robots.txt in webmaster and my last few posts doesn’t crawl. I also check privacy and select “Allow search engines to index this site.” I have also installed Sitemap.xml plugin and select make virtual robots.txt. Now i have installed another Special plugin for Robots.txt named as “WP Robots Txt”. When i open mysite.com/robots.txt it says

    # Added by SEO Ultimate’s Link Mask Generator module
    User-agent: *
    Disallow: /go/
    # End Link Mask Generator output

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Sitemap: http://www.mysite.com/sitemap.xml.gz

    I want Google to crawl my site what should i do now.

Viewing 8 replies - 1 through 8 (of 8 total)
  • Krishna

    (@1nexus)

    Member

    Go here for Google’s guidelines.

    Thanks for response..
    I just want to know that my site is enable for crawling or not. In case of above Google can crawl my site???…

    Krishna

    (@1nexus)

    Member

    Read through the link I provided. Various options are discussed there. Select the one you want. For instance, to block the entire site, use a forward slash (Disallow: /). Like this see other options.

    According to your link my site is crawling by Google because i only select admin and includes in robots.txt. But somehow my site isn’t crawling by Google what should i do.Now some of my recent posts are displaying in Google and some are not.

    Krishna

    (@1nexus)

    Member

    Use Webmaster Tools and Google Analytics to know what is really happening to your site and which posts/ pages are indexed/ crawled by Google. You will get all reports, answers, etc. there. Forget what others say about it.

    I tried but and now i am conform that robot.txt file is fine but still getting red mark on Robots.txt fetch in Webmaster/tools…….

    Krishna

    (@1nexus)

    Member

    It takes time.

    Thanks for response.

Viewing 8 replies - 1 through 8 (of 8 total)
  • The topic ‘Robots.txt problem Need help’ is closed to new replies.