I run the site http://www.agletdesign.com and recently noticed that things got super sluggish and very often my dashboard wouldn't even load, giving me a internal server error. I then thought of the last thing that I had done and that was register with webmaster tools. I then went to the 'crawler access' in the settings and saw these lines in the robot.txt file:
I deleted this and did a test then went to my dashboard and everything was back to lightening quick. I have now reinserted these lines back into the robot.txt in webmaster tools and wordpress is still quick. Can someone explain to me why this slow down happened, could it be webmasters fault? Also, do I need those lines of code in the robot.txt, how does it help?