Can you PLEASE help define these settings from the Rate Limiting section?
I have read the detailed descriptions which seem to sometimes contradict.
“If a crawler’s page views exceed…”
Is this truly PAGE based or is it REQUEST based.
Many crawlers/bots are directly targeting/phishing for single files many per minute without going directly to a PAGE.
“If a crawler’s pages not found (404s) exceed…”
Again, is this PAGE-based-404 or FILE-based-404. Huge difference.
A page can be “200” whilst included files and links may be “404”.
“If a human’s page views exceed…”
Is this strictly PAGE based?
In the description, it recommends 240…but how can a human navigate even close to 240 pages in a 60secs?
That sounds more like REQUEST/FILE based (as seen in Raw Access logs)
“If a human’s pages not found (404s) exceed…”
Again, is this PAGE-based-404 or FILE-based-404.
In the description….it focuses on FILE-based-404’s which sounds different.
“If 404s for known vulnerable URLs exceed …”
In summary of all these settings, I see the terms used:
Maybe it’s just me but it is a bit confusing…especially when a setting says PAGE and the detailed description talks about FILES.
- The topic ‘Pages vs Requests vs URLs vs Files – Rate Limiting’ is closed to new replies.