All WP Sites Downed: Hacked But Doesn't Explain All

I have done a tremendous amount of searching here and on Google to figure out in full what caused 20+ installations to melt down.

Tracking down whether it is truly one root cause or possibly three is important in how I take the next steps.

Let me say up front that I am totally self-taught and am certain I have made a long list of mistakes but I’ve done the best that I can in 20 months to go from 2002 HTML to putting together a nice WordPress site. I know many of my mistakes already of course through what has occurred recently. Finally, all of the stuff I’ve done up to this point is for volunteer groups…no one is making any money on these sites, including me. (Making money is great! It’s just not the case here.)

When I say the sites melted down, I mean that if one attempts to pull them up, they either:
1) Show a “Fatal Error: ‘many varieties of text goes here’ ending with line XXX” / Parsing error
2) Internal Server
3) Blank White Screen
4) Site will come up but going to wp-admin is a dicey situation (results in #1) OR can get into the dashboard but get a variety of either a line code / parse error OR can go one level in and THEN get a line code / parse code error.

An examination of downloaded sites shows files scattered throughout that have Zero bytes (too many to list as examples). When one opens the files, of course they are entirely blank.

Obviously, that would explain the line code errors on something citing “missing”, but that is not the exact problem stated in all errors.

—–

I am on shared unlimited hosting with a Godaddy reseller (didn’t know this when I first signed up).

My first website – WordPress – was installed in the root directory. Later, many installs were added, so their folders sit in the root directory with the WP install. But all have their own databases. WordPress is by far the predominant type of installation, but there are two Joomlas, a Drupal, and a few other applications on my account (all are PHP based from what I can tell)

This is important to know because I discovered some information that indicates Godaddy’s servers may not have adequate protection to prevent interactivity between users / files who have shared hosting on their servers. THIS post and the discussion comments have some information in that regard.

The above linked post references massive hacking, but, from what I’m reading about a proper server configuration in a shared environment (suexec), it seems there are broader implications.

Godaddy, I’ve discovered, have had repeated attacks to WordPress (and some other php based) sites, apparently dating back to April / May of this year. The post linked to below relates that there were attacks on September 18 and 21.
GoDaddy Hacked Again: Another Way To Cure Your Site

This, of course, leads to:

CAUSE #1

EVAL hacking code in most php file headers in the root directory wordpress installation

I have not been able to look in every file from every WordPress install I downloaded, but I *did* find the EVAL base_64 code in a jsquery file in the wp-uploads folder of another install.

Problem with this as entire explanation: I cannot find anything in any of my searches that connects the EVAL base_64 malicious hacking to the blanking out of files.

CAUSE #2

A plugin destabilized on of the sites and its subsequent manual deletion wreaked havoc on all the installations.

I had a WP 3.0.1 install with Antisnews Theme that I had spent a lot of time developing over the past couple of months.
There were a number of plugins I was experimenting with, but was doing so one at a time. I made sure every plugin was rated as 3.0.1 compatible because I was asking a lot of the install by using a number of them.

The install was entirely stable with these plugins:
Wordpress.com Stats
Askimet
Contact Form 7
SEO All In One Pack
Add to Any
FD Feedburner
Twitter Tools

Was still stable having added, one at a time:
FD Footnotes
Citation Manager
Greg’s High Performance SEO (removed All in One)
WP SlimStat
List Category Posts

These were added in a short period, but one at a time:
W3 Total Cache
Simple Music
MP3 Player FX

I started to see some instability when I reached the last three, specifically in a couple of line code errors above the Dashboard. A refresh seemed to clear the problem and the site was otherwise functional.

I disabled MP3 Player FX, Citation Manager, and SlimStat to see if that helped.

The next day – September 21 – I discovered that although I could get into the dashboard, I could not access any of the menus.

I pulled up some of my other websites to see if it was a possible server issue, starting with the wordpress install in the root directory. It and others seemed fine (it pulled up, I could go to the dashboard, click links there and make my destination)

I did a bit of Googling quickly on the last few plugins I installed. I noticed the W3 Total Cache had just had a number of error reports and in the few days prior a major upgrade to correct issues.

Believing the plugin was the source of the problem, I felt I had no choice but to go into my c-panel and delete it. I have done that before with a couple of other plugins and it was fine.

After I deleted it, my site seemed improved, but not “perfect” – could go into the dashboard and minimally click around.

I refreshed the page on the root directory install and that’s when I got the first line code error. I started checking other sites and saw the same thing.

I discovered that every wordpress site I went to, besides the one for which I had deleted the plugin was done. I checked most (but not all) of my installations and all were throwing errors.

I called my hosting provider who said that my deleting the plugin should NOT cause an account wide melt down, but then again…it could. (???)

After spending two hours on the phone with them, they had no answers and said the server was fine after they said they ran tests.

Fast forwarding several days down the road, I discovered the EVAL base_64 embeds in the files as noted above.

I have spent hours on the phone with my hosting provider who says they have run multiple testings, including for malware. They said some of my databases sit on separate database servers. They have had no outages, server problems or otherwise.

They don’t what happened, can’t explain how my deleting the plugin could have caused account wide failure, etc., etc.

In efforts to sort this out, I found a thread here on the WordPress forums regarding questions some people were having about whether or not W3 Total Cache was changing the configuration of the Apache server. (LINK HERE)

I believe the W3 was simply interacting with other plugins – not that it is poorly coded, etc. Again, I was demanding a lot from my installation.

CAUSE #3

There was server damage, corruption, failure.

Based on the complete lack of assistance from hosting service and an overall degrading level of service from them over the last 9 months or so, it is very difficult to trust their competence, and unfortunately, even their honesty.

A programming developer took a look at a number of files from several sites and the overall situation and it is his opinion that there is something very wrong with the server.

It is hard to know the answer to this in some respects, since information from the provider would have to be totally rejected (they ran tests, my databases are on separate servers, etc.)

CURRENT STATUS / REMEDIES ATTEMPTED:

On the third call to my provider, they informed I had the ability to restore my accounts by rolling them back in time. The restoration was haphazard as despite using two different browsers, I kept getting a message: “Notice: Action Was Not Able To Complete Or Timed Out”. Examining the files after the process on all my installations, I saw updated file dates, and over the course of a few hours SOME of the sites did come back up, in various states of stability.

The programmer went through much of the code on one of the sites and essentially laid over fresh WordPress files throughout. After that, we deactivated, deleted, and reinstalled all the plugins and theme. The site seemed “back to normal”.

The following day, that site was down again. The programmer examined the files and noted many had been changed all at a time when no one had access to the site or back end.

All sites are down again, even those which had some form of functionality following a restoration.

It is true that some of these installations were out of date – the majority are either not currently having traffic driven to them or are for testing purposes. Of course I am well aware of the hazards of having them sitting online and out of date. Practices will change in the future.

QUESTIONS:

Has anyone any information regarding whether the EVAL Base64 hacking causes the wiping out of a number of file contents in WordPress?

If it does not, there are additional issues.

I have read the removing Malware and “Has WordPress Been Hacked” links here and elsewhere. I am already resigned to the fact that the databases and wordpress installs are “shot” due to the malware.

The core question is whether or not our problem is “limited” to the malware hack – and the key to that is the consideration that the files were wiped out.

We’re going to be moving to a new provider, but if the blanking out of the code is a server fault OR if they have poor configurations on the servers that allows files to interact (i.e. the W3 plugin), before I notify them of the move, I am going to attempt to get some kind of recompense from them.

Finally, it will assist us in the moving process to know what we should be looking for in the exports from the site contents in case the most recent backups of those have any possible corruption or damage, especially if it goes beyond the EVAL hacking problem.

I theorize that the servers are poorly configured, which allowed not only the massive hacking across GoDaddy, but a very sloppy interactivity of user files on the server. The plugin is not configured to do damage, but it seems that it may likely did, only because of the poor configuration.

I have tried to provide absolutely everything I thought relevant, but if anyone has questions, I will be checking this thread. Thank you for your attention.

Giving site links is not incredibly helpful at this stage, since none will pull up, but for purposes of examples of line code errors, here are two: (It is possible, due to the incredibly random behavior we’ve witnessed that either of these MAY come up and go back down. If the sites are up, clicking on things will likely provoke an error eventually…unless a miracle occurs.)

http://grassrootsne.com – where the W3 cache plugin was installed.
http://clibertyc.com – the site that had been edited for errors, fresh wp installed, plugins/theme deactivated, deleted, and reinstalled and was functional until “hit” again.