Bots Crawling Nonexistent URLs!
-
Hi,
I don’t know why, but I noticed that bots from google, bing and others are crawling weird, nonexistent, urls on my site.
My site is a multisite and has a couple different domain names. The urls the bots are crawling looks like this:
http://www.domain_name_one.com/domain_name_two.com/random-phases
So these are the random phases I have been getting:
newborn-cub
radwan-ziadeh
baju-songket-putih
norman-smitherman
and hundreds more coming in every minutes or soOf course, these urls don’t exist and in fact, the second domain name is an old site that I already deleted from my multisite. There are no links between any of the sites I have so there shouldn’t be any reason why it would even reference the two sites together.
I have been hacked a couple of times the past few months and I am not sure if this is related but I want to know if I should try fixing this and if so, how I can fix it.
i tried using robots.txt to block the bots access but I haven’t seen any improvements.
I used the following:
Disallow: /domain_name_two.com/*Thanks in advance.
The topic ‘Bots Crawling Nonexistent URLs!’ is closed to new replies.