index, no follow

  • hj

    (@hj)


    8 years, 3 months ago

    I’ve installed WP on a live server, but while in development and building I DON’T want google or any other search engines crawling it.

    I’ve added ‘user-agent/disallow’ to the robots.txt file. I’ve also checked the box under settings/read which says “Discourage search engines from indexing the site”.

    So it is with some consternation that I view the source of my visible page and see in the header: <meta name=”robots” content=”index, follow” />

    ???

    Doesn’t seem like a discouragement to me!!

    Can someone with SEO nous please advise?

    Thanks
    HJ

Viewing 4 replies - 1 through 4 (of 4 total)
  • Marios Alexandrou

    (@marios-alexandrou)

    8 years, 3 months ago

    If you have an entry in your robots.txt that disallows crawling, search engines won’t crawl the pages and therefore won’t see the meta tag so its value doesn’t really matter.

    Note that robots.txt directives and meta tag directives don’t guarantee that search engines won’t crawl or index content. It’s a request that they typically follow, but sometimes things don’t go as expected.

    As for the meta tag, are you using an SEO plugin that may be overriding your settings? Or is there something in your theme that outputs that tag?

    Thread Starter hj

    (@hj)

    8 years, 3 months ago

    Hi Marios, thanks for the thoughts.

    No, I’ve heard that robots.txt isn’t google-proof! I’m trying to find what is.

    Regardless of the presence of my robots file, I still think it ‘matters’ that WP has a specific checkbox effectively saying “don’t index me”, but then outputs files that say “do index me”. I think that matters, because it’s craziness!

    As for the meta tag, no I’m not using any SEO plugin – not yet, anyway. I have no idea if there’s anything in the theme that outputs the tag, ‘twould need a geek to know that 😉

    regards
    HJ

    Tarun

    (@scrapur)

    8 years, 3 months ago

    Verify your site in Google webmaster tools and that google tool will guide you in best way !

    Thread Starter hj

    (@hj)

    8 years, 3 months ago

    thanks Tarun.

    I just had a look at Webmaster Tools…..I’m already an hour in and none the wiser. Could take me all day!

    I really don’t need a complex analytics tool…..I just want to stop Google at the front door. Shouldn’t require an IT degree, should it?….

    regards
    HJ

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘index, no follow’ is closed to new replies.