Hey there.
Could you check your access logs, what’s being hit the most to cause this load?
Are you tracking analytics?
Where is the traffic coming from?
Any large images? or downloads?
robots would only work if what it was hitting your site respected that.
Cheers.
Thread Starter
Panos
(@panosk)
I am sorry what is the access logs? These in Cpanel? Here is a screenshot of the latest visitors. http://postimg.org/image/9e6rnoqcz/
But I can’t manage to find which one is causing more traffic. Would it help if I share a raw file of the access logs ?
Also I enabled the WB_DEBUG and I found only php Notices and according to the themes author there are not important.
Do you mean google analytics? I use google webmasters tools.
No downloads or any large images.
Andrew Nevins
(@anevins)
WCLDN 2018 Contributor | Volunteer support
Your hosting providers should be able to help you find these logs.
Hey again.
As Andrew mentioned you can ask your host to work with you on this.
Looking at your image I see that your 404 page is around 1MB in size. If your 404 is hit approximately 1024 times then you have a GB right there.
Lots of hits there from the IP starting with 66.249.69, a check shows them for example crawl-66-249-69-204.googlebot.com so you could maybe use Googles Webmaster Tools to reduce the crawl rate:
https://support.google.com/webmasters/answer/48620?hl=en
Note this means the site is not indexed as quickly when you reduce the crawl rate.
If you have the raw logs then you’re looking for groups of IPs that are causing the bulk of the load. You also want to find anything that looks abnormally large on a page load and then investigate that page.
Also I enabled the WB_DEBUG and I found only php Notices and according to the themes author there are not important.
Ya, notices are not errors and shouldn’t really be a problem, but they could be fixed still.
Do you mean google analytics? I use google webmasters tools.
Yes, I was interested to see where they emanate from. When I’ve had similar things in the past they often came from the same region and although not ideal I’d geoblock their IP if my target market wasn’t in that region. Some people warn against this as it can return false positives.
Take care.
Thread Starter
Panos
(@panosk)
You were right the crawl rate was over a thousand. I don’t know why…! Other sites I manage were significantly lower than this. I reduced it to the lowest level in order to see results in traffic.
Thanks anyway. I will tell you if this worked.
Ah, well if they keep hitting pages like the 404 and that’s just around a MB then it won’t take long at well.
You could look at optimising that 404 page, if it’s not loading any real content then there is no need for it to be so large. I’d personally tackle that as well and then monitor how it goes.
Take care.
Thread Starter
Panos
(@panosk)
Finally the problem was the crawl rate of google webmaster tools..
For some unknown reason it was set to maximum (almost 2000 requests per second!!)
I didn’t set it in such a speed and I don’t know why this was the default.
Anyway as soon I set it to the lowest rate the traffic was reduced to normal bandwidth.
Thanks very much for your useful help!!!
I will try to reduce as you said the 404 page. Is there any tool to see the size of each page?
Hey there.
Ouch, that’s a lot and I’ve never had them hit one of my sites so hard in the past, and we’re usually pretty good with it over at WPMU DEV too. At least it’s resolved! 🙂
There are indeed tools you can use, for example Chrome Developer tools and PageSpeed Insights which is an extension for Chrome:
https://chrome.google.com/webstore/detail/pagespeed-insights-by-goo/gplegfbjlmmehdoakndmohflojccocli
You may also find this resource helpful:
https://developers.google.com/speed/pagespeed/
Or if you use Firefox with FireBug then you could use YSlow:
https://addons.mozilla.org/en-US/firefox/addon/yslow/
Anyway, glad to hear things are going more smoothly.
Have a great day! 🙂
