Did you add AhrefsBot to your robots.txt within the past day?
Dark Visitors takes some time to re-read your robots.txt, which it currently does every 24 hours. It does this in order to not spam your website by reading it constantly. Rule-breaking visits will start to show up as expected after it re-reads the updated version.
Hope that helps!
Gavin
OK, I’ll wait longer, the other issue is whether or not it is breaking the rules or continuing a scan, I’ll let it sit for a couple of days….
I didn’t see a 24 hour delay after adding some of my other entries but who knows? If you don’t scan it everytime I refresh the “/robots-txt” page, then it should probably say when it was scanned and perhaps have a rescan button….
Hi @dbareis, checking back in. Let me know if things haven’t fixed themselves as expected. In the meantime, I’ll mark this question as resolved.
I can’t tell any more and without knowing the timestamp of the robots.txt file you use there is no resolving this one way or the other. At least “AhrefsBot” is not ignoring the “robots.txt” file so it must have been continuing an existing scan at the time.