WordPress.org

Ready to get started?Download WordPress

Forums

Simple URLs
Google indexed hundreds of /go/ pages (18 posts)

  1. Deon B
    Member
    Posted 10 months ago #

    Hi there,

    1) I have set a disallow /go/ in my robots.txt
    2) Using Yoast's plugin I have set "no-index" for Simple Urls

    Results:
    Google indexed all my /go/ pages that now appear on search results like this:

    Affiliate product 1
    mywebsite.com/go/affiliate-product1
    A description for this result is not available because of this site's robots.txt – learn more.

    The only thing I haven't done is to set rel=nofollow for those links, because I thought that "disallow /go/" would have been enough. Is there anything else I can do?

    Thanks
    Deon

    http://wordpress.org/plugins/simple-urls/

  2. Ron Rennick
    MultiSite Guru
    Plugin Author

    Posted 10 months ago #

    If you do not want the search engines to index your URLs use your robots.txt to tell the SE not to index them.

    Anything that is publicly accessible on your website will be indexed by search engine unless you tell them not to.

  3. Ron Rennick
    MultiSite Guru
    Plugin Author

    Posted 10 months ago #

    You also have to allow the search engines some time to read your robots.txt & update their listings.

  4. Deon B
    Member
    Posted 10 months ago #

    Hi Ron,
    here's the foreword of my previous post, since I think you missed it:

    Hi there,

    1) I have set a disallow /go/ in my robots.txt
    2) Using Yoast's plugin I have set "no-index" for Simple Urls

    I've already done the things you say (before publishing a single URL), but google has indexed them anyways.

  5. Ron Rennick
    MultiSite Guru
    Plugin Author

    Posted 10 months ago #

    You may need to add rel="noindex" to the links in your content. nofollow tells the bot not to follow them vs not to index them.

  6. neo7
    Member
    Posted 9 months ago #

    You can't noindex a link. That would have to be defined on-page, in the meta data.

  7. darrencss
    Member
    Posted 7 months ago #

    You can remove the /go/directory for 90 days via webmaster tools. You will have to keep doing this every 90 days in order to keep the links out of google's index.

  8. mackaaij
    Member
    Posted 3 weeks ago #

    I have the exact same setup and submit URL's to remove to Google weekly manually:
    https://www.google.com/webmasters/tools/removals?pli=1

    It would same me some time if we could prevent indexing :)

    Has anybody determined whether this keeps the links out? As Neo7 suggests it's not possible to keep links out.

    Adding rel="noindex" to existing links would then be a hell of a job but could probably be done with a regex. From then on it would be great if Simple URLs would offer those URLs via the interface.

  9. Deon B
    Member
    Posted 3 weeks ago #

    Hey mate,
    I'm not John Mu, but as far as I know submitting URLs removal weekly is not a good thing to do. That should be a tool to be used every now and again.

    I just got rid of this plugin and set up 301 redirects through a 3rd domain. It makes it much easier, cleaner and faster. Plus I don't have to deal with the hormonal changes in the plugin developers moods and their willingness to fix problems.

    Good luck

  10. mackaaij
    Member
    Posted 3 weeks ago #

    Thanks, I might rethink this setup indeed :) SimpleURLs integrates nicely in WordPress, counts clicks in WordPress and Google Analytics. Migrating by creating a .htaccess shouldn't be too hard. But your 3rd domain URLs are still indexed?

  11. Deon B
    Member
    Posted 3 weeks ago #

    Ok so this what I'm doing and it works like a charm.

    Get a new short domain, that looks like you have your own private URL shortening service. I got some .cc domains

    Let's say you get mack.cc

    Now on mack.cc install:
    http://wordpress.org/plugins/redirection/

    Then upload an excel file with all the redirections following the instruction you'll find on the plugin site or somewhere else.

    For example:
    mack.cc/1 ---redirects to--> affiliate link 1
    mack.cc/2 -----> affiliate link 2

    Upload a robots.txt to block the entire mack.cc website from being indexed.

    On mack.cc, on the redirection plugin setting you'll find statistics telling you how many times your redirect links have been clicked.

    Plus on your website if you're using Jetpack stats all your mack.cc clicks will be tracked as external clicks and you can easily check how many times they have been clicked.

    All your redirection are on mack.cc, so that lightens up your main site. All you have to do is to nofollow your mack.cc links on your main site because you don't want to send "link juice" to an empty noindexed site.

    That's it.

    Hope that helps.

  12. mackaaij
    Member
    Posted 3 weeks ago #

    It does, thanks!

    Regarding the redirection plugin I heard it grows your database quickly because of all the 404's that are generated by sites/bots. Any experience with that?

  13. Deon B
    Member
    Posted 3 weeks ago #

    I'm not sure what you mean about growing my database,
    could you please expand?

    Thanks

  14. mackaaij
    Member
    Posted 3 weeks ago #

    I haven't experiences it myself of course but heard/read somewhere (long time ago) every 404 hit creates a new database record causing at least the database to grow very fast and maybe even slow down WordPress.

  15. Deon B
    Member
    Posted 3 weeks ago #

    Hmmm,
    I don't know what you mean by "database record", where's this database? I'm not sure what you're referring to.

  16. mackaaij
    Member
    Posted 3 weeks ago #

    As I heard the redirection plugin adds records to the WordPress database on each hit.

  17. Deon B
    Member
    Posted 3 weeks ago #

    I'm ignorant about this,

    I don't know what a WordPress database is, I don't know what adding records to it means, and I don't know why adding records to the database would be a problem.

    It would be nice if you could explain so that I learn something today, but don't worry if it's too much of a trouble :)

  18. mackaaij
    Member
    Posted 3 weeks ago #

    Everything you store in WordPress is saved in a database. So posts, pages, categories, tags, ... A database consists of tables. Just like an Excel worksheet. WordPress has a table for posts for instance. In this table, each post is a record.

    You should know about this database and create a backup of it so you can restore in case of problems.

    I searched for redirection found this post about a log you should purge manually. That might be the issue I heard or regarding performance:

    http://mydesultoryblog.com/2011/11/wordpress-and-my-slow-mysql-database-issue-is-solved/

Reply

You must log in to post.

About this Plugin

About this Topic

Tags

No tags yet.