We have over four years of posts in the archives, several hundred in number. Would removing or somehow, compressing the archives or even archiving the archives(?) if that’s possible be a potential solution?
I have added a robots.txt file in yet some of the crawlers are ignoring it and accessing the /archive/ files anyway, according to the hosting company.
Is there an app or plugin that will correct this or simply block all crawlers? Or specify which crawlers I can allow?
When I looked at the plugins, I didn’t see a specific one for my needs. I will look again, however. Any suggestions would be appreciated.
We were running Global Translator which I believe can eat up some server time, but I’ve disabled it and there still seem to be problems. I’ll go look for other archive plugins. I don’t have a bot stopping text anywhere. I prolly could use something like that. Any recommendations?
