Support » Plugin: Slimstat Analytics » v4.8.3 : googlebot still logged

  • Resolved swissspaceboy

    (@swissspaceboy)


    googlebot is back. This is the entry log

    66.249.92.70, rate-limited-proxy-66-249-92-70.google.com

    Still mobile device, Android, Chrome from Redmond.

    Was this not supposed to be fixed?

    Thanks,

    Didier.

    • This topic was modified 1 month, 2 weeks ago by  swissspaceboy.
Viewing 12 replies - 1 through 12 (of 12 total)
  • Plugin Author Jason Crouse

    (@coolmann)

    Hi @swissspaceboy,

    we rewrote this code in version 4.8.4, to make it even more effective and avoid false positives like the one you experienced. Unfortunately, this happens when the Browscap Library doesn’t have a match for a specific user agent. We are introducing a fallback function for situations like this one. The new version is already on Github, if you want to test it in advance.

    Cheers,
    Jason

    Thanks Jason for the fast reply. I will wait for the official update, test and let you know as usual 🙂

    Best regards,

    Didier.

    Plugin Author Jason Crouse

    (@coolmann)

    Fair enough.

    Thank you,
    Jason

    Bad news.

    I updated yesterday to the latest plugin version, and the log is still full of Googlebot. Now the visitors are tagged as “default browser”. mobile, Android

    66.249.89.161, rate-limited-proxy-66-249-89-161.google.com

    I see also some pings from MacOS from Google.

    Exclusion of “bots” is still enabled. Browsercap library version 60200.

    An idea ?

    Didier.

    Plugin Author Jason Crouse

    (@coolmann)

    Hi @swissspaceboy

    this is quite strange. I’ve personally tested a few user agent strings and they are filtered as expected when the option to exclude bots is enabled. Would you mind sending me some of the user agent strings associated to those pageviews, so that I can run some more tests on my end?

    Best,
    Jason

    Hi Jason,

    I extracted my googlebot visites from yesterday 11 July from the raw access log. I extracted the user agent from it. Here is the unique list:

    Mediapartners-Google
    Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Mediapartners-Google) Chrome/41.0.2272.118 Mobile Safari/537.36
    Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/[WEBKIT_VERSION] (KHTML, like Gecko; Mediapartners-Google) Chrome/[CHROME_VERSION] Mobile Safari/[WEBKIT_VERSION]
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko, Mediapartners-Google) Chrome/41.0.2272.118 Safari/537.36
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/[WEBKIT_VERSION] (KHTML, like Gecko, Mediapartners-Google) Chrome/[CHROME_VERSION] Safari/[WEBKIT_VERSION]
    Mozilla/5.0 (compatible; Google-Site-Verification/1.0)

    These user agents are all coming from IP 66.249.89.*

    Thanks for your support,

    Didier.

    Plugin Author Jason Crouse

    (@coolmann)

    Hi,

    thank you for the strings. Did you mask the version number in some of them or is that how it was recorded in the database? Just making sure I’m using the exact same user agent in my tests.

    Jason

    Hello,

    Here is the full Apache line for one of the first visits of 11 July. This visite is captured in Slimstat

    Line 26805: 66.249.89.161 - - [11/Jul/2019:18:38:17 +0200] "GET /la-gestion-de-projet-marketing-agile/ HTTP/1.1" 200 17631 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/[WEBKIT_VERSION] (KHTML, like Gecko, Mediapartners-Google) Chrome/[CHROME_VERSION] Safari/[WEBKIT_VERSION]"

    I searched bit in the database and here are the 2 lines with the IP and string “agile”:

    (1313, '66.249.89.161', '', NULL, NULL, 'us', '37.751,-97.822', NULL, 'https://www.xxx.ch/la-gestion-de-projet-marketing-agile/', '/la-gestion-de-projet-marketing-agile/', NULL, NULL, 'hostbyaddr:rate-limited-proxy-66-249-89-161.google.com', 1215, 0, 40, 'Default Browser', '0', 2, 'android', 'en-us', 'Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Mediapartners-Google) Chrome/41.0.2272.118 Mobile Safari/537.36', '360x640', 360, 640, 'post', '303,973,701,975,392,187,828,26', 'xxx', 10324, NULL, 0, 1562870516),
    (1339, '66.249.89.161', '', NULL, NULL, 'us', '37.751,-97.822', NULL, 'https://www.xxx.ch/la-gestion-de-projet-marketing-agile/', '/la-gestion-de-projet-marketing-agile/', NULL, NULL, 'hostbyaddr:rate-limited-proxy-66-249-89-161.google.com', 1240, 0, 40, 'Default Browser', '0', 2, 'android', 'en-us', 'Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Mediapartners-Google) Chrome/41.0.2272.118 Mobile Safari/537.36', '360x640', 360, 640, 'post', '303,973,701,975,392,187,828,26', 'xxx', 10324, NULL, 0, 1562871346);

    Didier.

    • This reply was modified 1 month, 1 week ago by  swissspaceboy.
    Plugin Author Jason Crouse

    (@coolmann)

    So, it looks like a bug in the third-party library we use to detect browsers. I posted a note on the developer’s Github page, so that they can look into it:

    https://github.com/browscap/browscap/issues/2130

    In the meanwhile, you can add an exclusion rule in Slimstat to not track any browsers containing Mediapartners-Google in the user agent string. Go to Slimstat > Settings > Exclusions > User Agents, and enter:

    *Mediapartners-Google*

    Keep an eye on the Browscap Github page to see if/when they decide to fix this issue.

    Best,
    Jason

    Plugin Author Jason Crouse

    (@coolmann)

    You can also use their tool to see how any given user agent string is decoded by their library:

    http://browscap.org/ua-lookup

    If it says crawler: false on the output page, then it means that they don’t consider that user agent string a bot.

    Best,
    Jason

    swissspaceboy

    (@swissspaceboy)

    Thanks Jason for your analysis. I also suspected the “media partners” of Google to scrap my website to propose more “appropriate” ads 🙂

    I added the new useragent string as exclusion, and I will check for results.

    I will track the bug report for Browsercap too…

    Many thanks for your fast & good support !

    Didier.

    Plugin Author Jason Crouse

    (@coolmann)

    We’ve been considering moving away from Browscap and adopting a different solution. Unfortunately their library has become increasingly unreliable and heavy over time, and there are more nimble alternatives out there. Hopefully this will solve all these problems once and for all!

    Jason

Viewing 12 replies - 1 through 12 (of 12 total)
  • You must be logged in to reply to this topic.