Support » Plugin: SEO Ultimate » robots.txt

  • I just finished a new site with SEO Ultimate installed & released Search Engine Visibility las night so I could submit a sitemap. Webmaster tools tells me that Googlebot is blocked & showed the following robots.txt

    # Added by SEO Ultimate's Link Mask Generator module
    User-agent: *
    Disallow: /go/
    # End Link Mask Generator output
    User-agent: *
    Disallow: /

    I have tried disabling Link Mask Generator & added a new robots.txt but that didn’t work. Then I disabled the plugin totally, which still didn’t work. I have implemented a physical robots.txt in my root as per this thread. in browser shows that the new robots.txt

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content

    but Webmaster tools is continuing to tell me that Googlebot is blocked.

    Help appreciated.

Viewing 2 replies - 1 through 2 (of 2 total)
  • Have you tried this:

    User-agent: Google
    User-agent: *
    Disallow: /

    It’s not quite what you are looking for but my thought process would be disable the plug-in and remove the txt file. If thats OK add back this txt file which should specifically allow google. If thats OK add back your file and so on.

    I use this resource when stuck

    Thank you Andrew,

    I did try that but to no avail. So I did a google fetch & got this result.

    Fetch as Google
    This is how Googlebot fetched the page.
    Date: Wednesday, March 20, 2013 at 9:57:37 AM PDT
    Googlebot Type: Web
    Download Time (in milliseconds):
    The page could not be crawled at this time because it is blocked by the most recent robots.txt file Googlebot downloaded. Note that if you recently updated the robots.txt file, it may take up to two days before it’s refreshed. You can find more information in the Help Center article about robots.txt.

    So it can take a few days to register the new robots.txt.

    Im so impatient 🙂

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘robots.txt’ is closed to new replies.