Support » Fixing WordPress » Search Engines

  • Dear Sirs,

    How can I prevent Search Engines from crawling and indexing specific urls of a website?

    Regards,
    Ed

Viewing 2 replies - 1 through 2 (of 2 total)
  • Disallow them in robots.txt https://en.wikipedia.org/wiki/Robots_exclusion_standard

    Example to stop indexing of a specific post of mine.

    
    User-agent: *
    Disallow: /2016/12/06/blog-updated-to-wordpress-v4-7/
    

    There’s probably some plugins that will allow to work with it. WordPress Robots.txt File — WordPress Plugins It’s also some of the SEO Plugins will let you determine which things to index or not.

    How can I prevent Search Engines from crawling and indexing specific urls of a website?

    The only way to stop a page from being fetched by bots and spiders is to have the page require a user login.

    Note: Once your site is ‘crawled’, that content may be indexed and it may prove difficult to get removed from all indexes.

    Robots.txt files are ‘suggestions’ to search engines, but are not followed by all.

    If you have an existing page you do not want a site visitor to see, you can use a Redirect Rule to redirect anyone to another page.

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Search Engines’ is closed to new replies.