HTTP HEAD – Not human!

  • devonsmarty

    (@devonsmarty)


    1 week, 1 day ago

    Hi,

    I’ve been using BotObservatory to investigate unusually high visitor figures on my WordPress site and have found what looks like a classification issue that may be of interest.

    BotObservatory is recording requests from 192.0.84.117 as:

    Human → Direct → Home

    The requests occur almost exactly every five minutes. For example, BotObservatory records them at 09:02, 09:07, 09:12, 09:17, 09:22 and so on.

    I investigated the same IP in Cloudflare. Over the last 24 hours Cloudflare recorded 285 requests, all to /, and importantly all 285 use the HTTP HEAD method rather than GET.

    285 requests per day is almost exactly the number expected from a five-minute automated health/availability check (maximum 288/day), so these don’t appear to represent human visits.

    At present, however, BotObservatory includes them in its Human statistics. On a relatively low-traffic community website this has quite a significant effect on the figures.

    Would it make sense for BotObservatory to exclude HEAD requests from the Human category, or perhaps classify them separately as System/Monitoring traffic?

    I’m happy to provide further logs or Cloudflare information if that would help.

    Regards,
    John

You must be logged in to reply to this topic.