It might be nice to see (or even edit) the agent that is passed to the crawling so that I can see it more easily in my logs and hence discount it.
It might be nice to see (or even edit) the agent that is passed to the crawling so that I can see it more easily in my logs and hence discount it.
You must log in to post.