Work Has Been Hell
February 21, 2006 | In: Work
Last week was absolute hell!
We had a major outage in our production environment a couple of weekends ago, resulting in major data loss. Through a series of unfortunate events, our SAN lost power, resulting in data corruption in our 2 TB database.
Due to unforeseen circumstances, we weren’t able to revert to a recent snapshot of our database, but instead one taken during the previous day, in the morning, resulting in a data loss of about 36 hours. Fortunately, we’ve been able to reconstruct a lot of it, but we have some angry customers and we’ll certainly loose business because of this disaster.
I’m thankful for not being anywhere near the frying pan on this one, but I dread the thought of us loosing a team member. While the offending parties may deservingly be terminated for what happened, I don’t want to see anyone loose their job and I definitely don’t want to be forced to pick up the resulting slack.
The company was very responsive to the stress level amongst the employees. Food was provided by either ordering in for us or allowing us to get, for free, up to two items from the chocolate shop on the first floor. They also had a massage therapist in the building helping out.
We’re certainly over the hump, in my opinion, but I don’t think we’ll see the fall-out from this crisis for some time to come, unfortunately.
