Support » Fixing WordPress » Fetch as google, robots.txt, .htaccess?

  • I will paste code from robots.txt and htaccess on pastebin.

    Here is robots.txt:
    And .htaccess:

    When I “fetch as google” urls like

    Sometimes it work, but most time keeps refreshing and after hour or so it would fail.

    I changed permalinks from %postname% to %category%/%postname% last week, don’t know should I wait more for google to update to my new permalinks, maybe the time is answer, or maybe robots.txt or .htaccess aren’t configured on the right way.

Viewing 6 replies - 1 through 6 (of 6 total)
  • So what’s your question?

    So what’s your question?

    Are my .htaccess and robots.txt correct, in order to allow << and subdirectories, sublinks.
    and to allow some of links like this one is

    That’s the question.
    I don’t know do I have my side problem or Google side.

    Moderator kmessinger


    What theme are you using? Please post a url to your site. Are you verified by Google?

    Are my .htaccess and robots.txt correct

    This is not a WordPress problem. You may or may not get assistance

    Hi, kmessinger. is my site.

    I was easily adding URLs to Google in Webmaster tools before I did changed my permalinks and blocked some content by robots.txt

    Now, whenever I add link, allowed by robots.txt, google page keeps refreshing, and after hour it would fail.
    Not all but 90% of added links.

    Yes, maybe this is not a WordPress problem, I just want to be sure are my robots.txt and .htaccess ok, as my site is built with WordPress.

    Theme that I’m using is a child of the Hueman theme.

    Moderator kmessinger


    Do you have a backup of your robots.txt before you changed things?

    Yes, I do have.
    But backup of robots.txt are not for %category%/%postname% permalinks structure.

    I wanted to hide some whole categories from Google.
    That’s why I changed to this permalinks structure.
    I can block all or none of posts with %postname% permalinks structure.
    With this new I can block all category posts, without blocking some another category posts.
    That’s why I did this change.

    All is triggered by Google Bot that was eating my bandwidth way too much.
    I had to hide some parts of the site, and show just a key content.

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘Fetch as google, robots.txt, .htaccess?’ is closed to new replies.