Dreamhost down? (18 posts)

  1. justex07
    Posted 10 years ago #

    I cannot get dreamhost to load up, much less my site. Plus, in other news, myspace.com is down too. Coincidence? I think not! Actually, probably is. ;-)

    Anyone have some news/clues on this downtime with dreamhost?

  2. archshrk
    Posted 10 years ago #

    Message from my host (powweb)
    We have experienced network outages in the last twenty minutes. The city of Los Angeles lost power which affected many high-rise buildings, traffic signals, LAX airport, and other large infrastructures reaching as far as the San Fernando Valley.

    Hopefully the city officials have this under control. Our admins are doing their part monitoring our network to ensure its availability as best they can during this time.

    Here is a link to an AP story regarding this situation: http://www.breitbart.com/news/2005/09/12/D8CIUQFO0.html

  3. alberto
    Posted 10 years ago #

  4. justex07
    Posted 10 years ago #

    Wow, you guys are great! My powweb is working just fine... actually loading faster than normal haha.

    Well thanks for the help guys!

  5. manstraw
    Posted 10 years ago #

    ahem .. battery backup? UPS? I'm not saying any more.

  6. djchester
    Posted 10 years ago #

    So let me get this straight...DirectNIC survived the largest hurricane damage in modern US history, and dreamhost can't make it through a California power outage. Hmmm...

  7. manstraw
    Posted 10 years ago #

    Did you read the directnic guy's blogging during the hurricane? pretty interesting. http://www.livejournal.com/users/interdictor/

  8. Alex Mills (Viper007Bond)
    Posted 10 years ago #

    Sounds like a crappy host with no generator to me.

    My host is in the state of New York and I didn't loose my site for 1 minute during the whole blackout across the east a few years back. They had their big generator running the whole time. :D

  9. carburetor
    Posted 10 years ago #

    They said their backup system failed. They should be up in 45-90 mins.
    Then tomorrow their building will fall victim to a a gust of wind which will cause them to go down again.

  10. justex07
    Posted 10 years ago #

    i got their service, domain and hosting, for only $9 a year so i can't really complain. i pay $120ish for PowWeb and Dreamhost's MySQL servers are way more reliable.

  11. jalenack
    Posted 10 years ago #

    I'm on dreamhost too, so I'm down.

    I saw pictures on their blog of the three huge backup generator buildings they have. So it's surprising that they're still down. Oh well...

  12. blueocean
    Posted 10 years ago #

    I used to be on Dreamhost, but I kept going down so I switched to another host (Netfirms). I'm sure every company has their fair share of issues, but I haven't had any problems since I switched (9 months now).

  13. jalenack
    Posted 10 years ago #

    Well let's not turn this into another "who has what hosting" thread.

    My site's back up. Weee!

  14. blueocean
    Posted 10 years ago #

    Ok, well, since you brought it up...who has what hosting?

  15. scruffy
    Posted 10 years ago #

    It looks like my site is back up, but the database is still down. I get my WP "Error establishing a database connection" error. Interestingly, though, the dreamhost.com domain itself is still down which means I can't get to my control panel, either.

    This was (is) far from just DH, BTW. MySpace was down, Yahoo went down briefly, and Media Temple is still down.

    Trust me, having to go through a large host location and start resetting stuff one thing at a time after a complete, and unexpected power failure, is not an easy or quick task. things do *not* just start working again when the power comes back up. You have to take into consideration all of the different servers running all the different services (pop, smtp, http, ftp, dns, ssl...). Getting them all back up and talking to each other (and praying every step of the way that all the data is still there) is a miserable undertaking.

    I feel more sorry for the folks at DH, MT, and anyone else effected by this than I do for myself because my podcast isn't going to be heard today.

    Speaking of that... I think this may all be my fault. I posted m latest podcast, like, 10 minutes before this happened :)

  16. blueocean
    Posted 10 years ago #

    Well...like someone else above mentioned...it's a little hard to feel 'sorry' for any host that goes down for a measly power outage after DirectNIC's heroic efforts.

    What's your podcast, by the way?

  17. Makki
    Posted 10 years ago #

    yep from Dreamhost emergency status site......

    Serious electricity problems in the Los Angeles area took our entire network off-line for a few hours. Service is currently being restored.

    Power to our data center has now been restored and our network is back up. Our servers and equipment are being powered on gradually to avoid potentially damaging power spikes. Barring any complications, your websites and email will be fully operational again very soon.
    Last Updated: Monday, 12-Sep-2005 16:44:33 PDT

    My site seems to be back up just waiting for email servers :)

  18. justex07
    Posted 10 years ago #

    And now a pretty extensive message from Dreamhost:

    On Monday, September 12 the greater Los Angeles area experienced a major power outage
    affecting large sections of the city, including our main data center. The power outage began
    shortly before 1pm PST and continued until about 4:30pm PST. Our data center is equipped
    with a redundant backup power system with both battery UPS systems and diesel generators,
    but the backup failed and our entire data center was powered down.

    We have previously covered much of this information on our official weblog (http://
    blog.dreamhost.com/) but many of you have not seen that information so we will summarize
    the events here.

    When the grid power to our building was cut, the UPS system kicked in and kept everything in
    the building up and running. The five generators also fired up and began providing power.
    The building needs four generators to operate at full power so the system is designed to
    tolerate a single failure. Unfortunately, two of the five generators failed within minutes of each
    other. We receive our power from the building housing our data center and they also manage
    the redundant power system. We do not know the exact reason for the generator failures at
    this time. We have received some vague explanations that we have not found to be satisfactory.
    Regardless, the remaining three generators were not sufficient to meet the building's power
    needs and that caused the emergency electrical systems to transfer into a “load shedding
    mode� and the building’s UPS system to turn itself off, thus preventing permanent UPS and
    related equipment damage. That shut everything down, including emergency lighting, and the
    building was evacuated.

    About 15 minutes later, one of the generators was started up to power emergency lighting and
    a couple of our senior technicians made their way into the (still evacuated) building and down
    to our data center to assess the damage. Since the backup power had failed, our own data
    center power remained off until the main grid power came back. We then proceeded to slowly
    power up our equipment. Servers (and all computers) consume significantly more power when
    booting up than when up and running so there is some risk of overloading the power circuits if
    too many of them are flipped on at once. Keeping that in mind, we powered everything on as
    quickly as possible. At that time the majority of our services were fully back up and running
    but some services were still down and we began the process of systematically verifying all
    services and making any necessary repairs and adjustments. Whenever a large number of
    servers suddenly loses power a certain small percentage of them will not come back up properly
    and when you have several hundred servers it takes awhile to verify all of them.

    Once our own access to our servers was restored our staff continued working into the night to
    restore as much service as possible and to respond to as many of your support cases as
    possible. Some of our staff continued working all the way through the night and we were able
    to restore almost everything that first night.

    Tuesday (September 13) started off early with all of us addressing the residual issues. At
    around noon that day one of our core routers experienced an internal failure stemming from
    damage previously sustained during the power outage. Our routers handle all of the Internet
    traffic coming in and out of our network and they are set up in a redundant way to minimize
    network disruption when a failure does occur. In this case, the main cpu of the router (called
    the 'supervisor') died and the secondary one took over. Everything continued working almost as
    it should have, but there is a remaining router issue that we are still working with Cisco support
    on. That issue is responsible for the slower than normal performance of our network and it will
    be resolved absolutely as soon as possible.

    During this outage, our off-network Emergency Status Page (http://status.dreamhost.com/)
    proved to be an invaluable resource for disseminating information among our customers. That
    status page remained up throughout the power outage and was updated regularly as we
    received new information. Unfortunately, not everyone knows about it and we will be working
    to improve that situation in the coming days. Those bloggers among you that did know to
    check the status page were extra helpful in passing along the information to other
    dreamhosters who were still in the dark. Thank you to everyone who helped out with that!

    This announcement will be followed by another explaining what went wrong with our processes
    and what we plan to do to address them. That will come in the next few days.

    We will be continuing to provide more detailed information on our official weblog found here:

    Also, everyone who has not bookmarked our Emergency Status Page should do so now. That
    page is found here:

    We will be improving on the basic page we have there to provide as useful of an avenue of
    information as possible.

    If you have any additional questions about this outage, please let us know. We will be happy to
    address all of your questions or concerns.

    The Un-Happy DreamHost Powerless Team

Topic Closed

This topic has been closed to new replies.

About this Topic