Great question!
From the various threads I found searching Google, it’s a bot for a search engine. Apparently, they don’t have a good reputation for obeying the robots.txt file so manually blocking is probably the best way to handle it. Most indexing services I’ve seen offer a way to not be indexed or a way to opt out but I can’t find it on their site.
Hope this helps.
Mia
Hi Mia,
Thanks for looking into this. Great find.
Should ByteDance and/or ByteSpider be added to Wordfence’s WAF’s auto-block list? Not sure how that would fit into Wordfence’s WAF block criteria.
Upon replying, it’s OK to close this topic as “Resolved.”
Again, thank you.
We wouldn’t necessarily add it because some users may need it if that’s a crawler for the preferred search engine in their or their customers’ area. Yandex used to be aggressive like this too but have made some improvements over the years. Yandex is predominantly used in Russia, but it’s also a popular search engine in other countries like Belarus, Kazakhstan, Turkey and Ukraine.
side note : I like Yandex because they have an image translation service that is super helpful!
https://translate.yandex.com/en/ocr
Mia
