Week 7 Clueless

As a supervisor we rotate weekends and a couple of years ago during my weekend (of course), the data center had to be shut down to prevent it from overheating and destroying all of our servers and other equipment located in there. I work in the call center so all day long, we get phone calls and we have to use our online applications to handle each customer. Well that day it all happened like a domino effect, my phone turned off and the internet cut off. I stood up and like lemmings everyone; one by one started popping their heads up telling me their systems stopped working. The call center started to get warm and nothing was working. When this happened I started calling around, I started calling the data center and leaving messages for them to find out if something had happened. After about 40 minutes, I finally got word that the AC stopped working and that the data center had to be shut down. At this point, we are not able to do any work, receive any calls or do anything and the room started heating up. We eventually had to get commercial fans and place them in the call center to hopefully start cooling things down. After spending all day on calls with everyone it was found that some fuse busted and that’s what caused the issue. By the end of the day, they were finally able to transfer everything to our data center in Ohio. It took about 3 days to get a piece to fix the problem and get the AC going and things switched back to our location. I was blown when this happened, a company that has a 24-hour call center and is a leader in banking technology would have had a backup to a backup to ensure that things like this don’t happen and at the very least doesn’t last all day. Thankfully, it was the weekend and we didn’t get a lot of calls. If this had happened during the week, it would have been devastating because of a number of calls we get. Having a better-equipped team at our location would have made a difference, having someone that can make that call to switch everything over to another location would also be a good thing to have. Eventually, they had one of our IT groups in another country keep an eye on things as well. Having systems in place that can monitor our systems would be something that is needed, this way they can see stuff go down and what is going down. Plus with the addition of another group watching our systems they can fix things quicker. Because being down for so long is not acceptable to me, I can’t imagine what it was like for my bosses boss.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s