Lessons Learned in a Prolonged Power Outage

Started by Raptor, July 21, 2021, 04:24:50 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Raptor

The building in which I am involved had a multi day power outage. The cause has yet to be adequately explained but we have been told an animal caused the 48 hour power outage. This article says a bird was the cause. More on that later. :rolleyes1:
https://www.wdsu.com/article/more-than-8000-people-without-power-in-multiple-neighborhoods-across-new-orleans-1626799330/37080272

The power went out around 9am Monday morning and did not get restored until 4pm Tuesday. The building is 23 stories and has around 1,000 people working in it (at least pre-COVID post COVID ~ 500 a day).

The building is well prepared for a power outage with 2 large diesel generators and a diesel powered sprinkler pump. These run emergency lighting systems, Alarm and monitoring systems as well as CCTV, building management offices for management and engineering staff.

The building is setup with another Katrina in mind with a lot of supplies and redundancy in critical systems. So I was more annoyed when I heard about the outage than concerned. It turns out we did a lot of things right but we also discovered some oversights that we thought we had corrected.

The generators are designed to run critical systems like the elevators and emergency lighting but they are not big enough to operate the 3 HVAC systems or the tenant equipment needs we would need 2 - 2 mw generators to handle that load.

Still power outages like this unrelated to storms are rare and during hurricanes we close the building and the tenants work elsewhere.

The good planning.
Generators.
We replaced all 4 of the transfer switches in 2020 during the lull. They were 25 years old and one had failed in 2019. These worked as planned transferring the load to the emergency generator. It is always exciting when a transfer switch for a 500 KW generator switches over. A warning horn in the generator room goes off and 10 seconds later the generator starts and gets up to speed, the analog system waits 3 minutes for the generator function or fail and then starts the transfer in sequence. There are 4 units and there is a lag of 2 minutes for each transfer switch to come on line to allow the surge of the electric motors being powered to subside. We have several banks of elevators and each elevator has a 25 horse power motor and these have a huge power surge when they start. That is why there is a lag. All elevators are recalled to the ground floor and the doors open, at which time all but one per bank shuts down leaving the doors open. This process is always nerve racking since if all of the elevators are in motion along with all of the other systems powered by the generator we are close to 90% of the generator capacity.  Once the elevators are shut down, the generator has a lot of reserve capacity. If there is a failure of any system people may get stuck in the elevator which will require our staff to deal with that issue directly. So was a great relief to see the elevator status showing all but one elevator per bank as recalled/standby status. 

Communication:
Our tenant notification system uses text messages, automated telephone announcements as well as PA announcements. We were able to notify all tenants about any issues and get responses back from them. This worked well.
We use company radios with repeaters on several floors. The repeaters are connected to the generator circuits. We also use cell phones for comms. These were not affected.

Diesel fuel. We had our fuel bunkers full but still managed to put a dent in them. The good news is that full load run test we generally schedule will not be needed this year and that will offset the cost of the replacement fuel.

The lessons learned.


It became very obvious that 10 cameras were not powered by the generator. That was easy to tell; they had black screens on the monitors. This type of event is the only way to find this type of failure, so no points off.

It was also obvious that while there was emergency outlets running gear in several areas that these areas did not have any ceiling lighting likewise connected. It was obvious by the darkness. This is also a no points off item since these areas do have a light on 24/7 but they were simply not tied into an emergency circuit.

The security staff normally operate in areas with emergency lighting but there are flashlights for their use. They are kept in a locked desk drawer to make sure they do not walk off. The guard on duty was not aware of the flashlights. Out of sight out of mind. Fortunately they were not needed but this is a training issue. It sound strange but a glass office building can be very dark even in the daylight away from the windows.

Several of the office staff did not have lights over their desk but did have outlets to run their computers. The plan was to buy lamps for the desks to provide them with lighting. That was overlooked and forgotten. This is a lack of follow up by management.

The extension cords that we bought to provide power from outlets powered by the generators for overlooked items were present and put to use but they were not taped over to prevent a trip hazard. We used these to power the refrigerator, microwave, coffee maker  and copy machine. These were to provide for overlooked items. +1 for thinking of extension cords and CB protected power strips, but they are deployed for items that we will need to function in a prolonged recovery like Katrina's aftermath.  This will have to be addressed.

The final item was a surprise. After 24 hours the telephone system and internet (we use VOIP) lines shut down. They were running on their own battery back up in the building. We have 5 internet providers working in our building They are all responsible for their equipment however we supply the power. They are all in one equipment room with emergency power but with the increase in suppliers the existing emergency circuit does not have enough capacity for all of them. So the company we use was plugged into a regular circuit.
We could have operated normally under these conditions if we had simply swapped plugs. It should be noted that virtually all of our systems are cloud based and as such our server handles mainly the building HSE, HVAC and elevator functions which cannot be accessed remotely. That said no internet means the office (as opposed to the engineering ) systems are off line. This is a new critical function that was shown to us by this power outage.

Information distribution. I cannot say enough negative about our power supplier's information flow system. We are a major customer in the area and have a direct contact. We were told that power would be restored by 11am Monday and passed that along, at 12:30 pm after phoning several times we were told 4pm Monday. Then it became 6pm Monday, then it became 6am the next day then 4pm and then again pushed back to 4:30pm only the have the power come on at ~ 4pm Tuesday.  The information flow as terrible. Granted there was not anything we could do we are there 24/7/365. However had we been provided with the information that it would require 24 hours that would have allowed our tenants to make other plans instead of waiting around to see what would happen.

Clearly that was either a very big bird or a very unusual hit. IMO it was fluke that damaged difficult to replace gear and the delay was due to actually finding the pieces/parts. However that is only my SWAG.

So in short no harm no foul but rather a good chance to look for problems in planning and address them.





 



 
Folks you are on your own...Plan and act accordingly!

I will never claim to have all the answers. Depending upon the subject; I am also aware that I may not have all the questions much less the answers. As a result I am always willing to listen to others and work with them to arrive at the right answers to the applicable questions.

woodsghost

As always, excellent write up and excellent information.

Thank you.

RoneKiln

That's a real good after action summary.

I have a vague memory of long ago  a squirrel getting into the wrong thing and sparking a ridiculous series of failures in the power system that jacked up a large area near me. The power company might have kept getting surprised to discover what they thought was a problem was the symptom until they chased it all the way back to a bird.

Long ago a minor power issue near me was mistaken as something else by a controller 100ish miles away and they flipped switches they shouldn't have. Every light bulb, surge protector, and the microwave in my home blew to pieces quite spectacularly. Everything else was on surge protectors and survived. Many of our breakers needed replacing as well. Most of our neighborhood faired much worse. Power company gave odd answers to questions for quite a while cause they didn't want to admit it was caused by they're error. They tried to blame it all on the minor issue the controller wrongly responded to.
"Seriously the most dangerous thing you are likely to do is to put salt on a Big Mac right before you eat it and to climb into your car."
--Raptor

sheddi

Great write-up, thank you. While my domestic power outage arrrangements are on a considerably smaller scale you make some good points that I need to consider!

You mention that you "put a dent" in your diesel bunkers. Are you happy to share the intended duration of your system? A week, a month, longer?

When I was a child (I would guess 10 years old. or so) we had an 8-day winter power outage which I remember being a bit of an adventure (my parents' home had solid fuel heating so the lack of power was an inconvenience rather than life-threatening). Since then the UK's national power distribution network has been privatised and as part of that there's now an obligation on the supplier to reinstate power within 12 hours (or pay out to affected consumers).

This became relevant once when, mid-December, there was an underground fault on the supply to our neighborhood of ~500 people. After 11.5 hours the power co hooked up a large diesel genset to our substation and we were all fed by that for about a month (including the entire holiday period) while they found the fault, excavated the site and repaired it.

Raptor

#4
Quote from: sheddi on July 22, 2021, 02:09:14 AM
Great write-up, thank you. While my domestic power outage arrrangements are on a considerably smaller scale you make some good points that I need to consider!
This is a commercial building so the setup must be robust, not because of local rules & regulations, but because our insurers require it. 


Quote from: sheddi on July 22, 2021, 02:09:14 AM
You mention that you "put a dent" in your diesel bunkers. Are you happy to share the intended duration of your system? A week, a month, longer?


We have an unusual set up with 2 generators. We inherited a 250Kw generator from a tenant who departed. We have tied that into the emergency power net work so that if the larger 500 kw generator fails we can use this in its place albeit at ~1/2 the capacity. Once the elevators are recalled and at standby or locked out this smaller generator is more than adequate for our needs. It can in theory also supply enough power by staging the transfer switch shift to allow the elevators to recall to the ground floor.  There are 4 transfer switches involved and with the new ones we have "smart" load shedding capability that will prevent the next one from shifting to the generator if the load exceeds a fixed capacity. In theory it should work better.

After Katrina we shut down the large generator and ran on this smaller generator to save fuel. We ultimately got a 1 mw trailer mounted generator which could run the smallest night HVAC unit and run the 30 hp air handling fans on each floor. That sucker needed a fuel truck delivery every two days.

The reason for this explanation is we are limited to storing no more than 10-55 gallon drums of diesel per generator room. They are in different parts of the building so instead of just 10 we get a total of 20. Each has a base tank that is sized to allow 8 hours of run time. The engines are automatically exercised weekly and they are rarely filled to the brim to avoid spillage. So when this started it is unlike that a full 8 hours of fuel was in the base tank.   

We went through 6 drums of diesel (330 gallons) the large generator is a GM 12-71 and screaming jimmys are not known for their fuel thrift or quiet operation. The smaller generator engine is a Cummins and it burns about 1/2 of the larger generator. 

The reason for this explanation is that the run time is variable based upon which generator is used. We may have as little as ~105 hours of run time if only the large generator is used or as much as ~210 hours if only the smaller one is used. Our key metric is keeping 20-55 gallon drums of diesel hand.

BTW the engineering crew hates to move the drums (in fairness they each weigh about 450 lbs full) so a realistic run scenario is to use the large generator until it becomes clear that it will soon be out of fuel. This would be ~50 hours of run time leaving ~5 hours in reserve so as not to have to bleed the injectors when it runs out of fuel. Then switch to the smaller generator and run it until its 10 drums were expended which is ~ 90 hours for a total of ~140 hours. 

This is a very long way of saying our key metric is observing an artificial limitation of 20-55 gallon drums of diesel hand.  :words:

Folks you are on your own...Plan and act accordingly!

I will never claim to have all the answers. Depending upon the subject; I am also aware that I may not have all the questions much less the answers. As a result I am always willing to listen to others and work with them to arrive at the right answers to the applicable questions.

EBuff75

If I remember the conversation with our security & compliance manager correctly, the building my company is located in has about 2 days supply of fuel for the generator.  Not sure on the size of the generator, but it's enough to power two elevators (the building is only three stories, plus a basement parking garage / utility space), limited lighting throughout the building, our server rooms, and I believe that one outlet at each desk (about 250-300 cubes) is powered.  Having backup power was part of the reason that we stayed in our existing building when we recently expanded (we now occupy about 60,000 sq ft of office space there).  We could've moved to the newer building next door, but they weren't as amenable to our backup power needs, because it would have required them to significantly upgrade their system.

Last month there was a multi-day power outage at work and the security manager was telling me that the primary fuel company flaked out on us for refueling.  We went to the alternate, but they were running late and we were within an hour or so of generator shutdown before they finally arrived.  The facilities people were about to start making trips to a gas station with fuel cans to try to keep things operating when the truck showed up.  Since this isn't the first time we've had a close-call with the refueling, it really seems like something we should be looking into improving...
Information - it's all a battle for information. You have to know what's happening if you're going to do anything about it. - Tom Clancy, Patriot Games

Raptor

Quote from: EBuff75 on July 22, 2021, 06:33:53 PM
Since this isn't the first time we've had a close-call with the refueling, it really seems like something we should be looking into improving...

Opinion & Personal Preference, not fact follows:

Everyone tends to focus on the size of the generator in KW. I put forth that the absolute most important thing about a generator is fuel. It makes no difference if you talkin about a diesel, gasoline, NG, propane. wind or even solar. Refueling (or recharging) is most critical thing to work out before you make any decision about size. Now granted size also has a huge impact on fuel choice especially in commercial applications. However until you figure out how you will refuel the critter you can make a decent choice.

In the application I described a generator is mainly expected to power only the HSE systems for a few hours mainly to permit the orderly and safe evacuation of the occupants.

The application I described it is a royal PITA to get diesel fuel to the generator. You cannot back a fuel truck into the lobby and there is no exterior access. So that means wheeling 55 drums in and out ~ 500lbs when they are full but even empty they are ~30 lbs. Then you have transfer the fuel into the base tank.

I have developed a preference for NG pipeline fed generators where there is NG provided by the utility. You do of course rely on the utility company and give up that aspect of self reliance that a diesel or gas generator provides. Still inventory control is conducted with the turn of a valve.

In an ideal world you could have both and run on the NG generator until/unless the provider fails and then switch to the liquid fueled generator.

Still maintaining an inventory of diesel fuel is a huge PITA. I noted the "arbitrary" limit of 10 drums but realistically that is all I would likely stock anyway. Even with fuel stabilizer and annual fuel polishing the fuel starts to degrade and after 3 years I cycle it out at great expense. It is like ammo...if you need it you cannot have too much ... and if you no do not need it you already have too much.

I generally have to cycle out 2 to 4  drums a year (when they hit 3 years of age) to the recycler for 50% of the then cost of replacement fuel and then buy fresh diesel to replace it.

Before you buy a generator give long and hard thought to how you will keep it fueled up and running.

Folks you are on your own...Plan and act accordingly!

I will never claim to have all the answers. Depending upon the subject; I am also aware that I may not have all the questions much less the answers. As a result I am always willing to listen to others and work with them to arrive at the right answers to the applicable questions.

EBuff75

Quote from: Raptor on July 23, 2021, 03:44:06 PM
I have developed a preference for NG pipeline fed generators where there is NG provided by the utility. You do of course rely on the utility company and give up that aspect of self reliance that a diesel or gas generator provides.
That was what my parents did with their "whole house" generator - got one which used the NG line as a fuel source.  My dad started with a portable, gas-fueled generator, but decided that it was too much work.  The major downside for the whole house setup was the cost for the initial installation (generator, automatic switch, splitting out the critical loads to a separate panel, and the labor for everything). They were immensely happy with it afterwards though.  Whenever the power would go out, there would be a few seconds of outage as it started up, but then they were back up and running. 

At their house, the basement would flood every time the power went out, due to the house being in a swampy area.  The sump pump was the #1 item they wanted to have in the critical loads, along with the freezer, fridge, furnace, and a few of the lights/outlets in the house.  Heat was more important in the winter than a/c in the summer, so I'm not sure if the a/c was included or not.  They've moved since then and my dad is considering doing the same for their new house.  I tried to convince them to go with solar / battery system, but that's more than he wants to spend (even if net metering could help pay for some of the cost).
Information - it's all a battle for information. You have to know what's happening if you're going to do anything about it. - Tom Clancy, Patriot Games

boskone

Quote from: Raptor on July 23, 2021, 03:44:06 PM
I generally have to cycle out 2 to 4  drums a year (when they hit 3 years of age) to the recycler for 50% of the then cost of replacement fuel and then buy fresh diesel to replace it.
Out of curiosity, are y'all load testing your generator periodically?  That's kind of a necessary part of generator maintenance, and should burn enough fuel to help with diesel rotation.

We did a partial load test monthly, and a full load test quarterly.  I'll note that in our case "partial" and "full" were holdovers from our old generator, and in both cases we were running a full building load; the difference was time, with the partial test being an hour and the full test being 4 hours.  It also acted as an opportunity to verify UPSes and whatnot, though individual groups were responsible for that.

We also had some NG generators at another site (until a fuel gelling issue, which is something to be cognizant of), and they ran some kind of hybrid grid supply-local storage system.  ...which, predictably, was kind of a pain that never worked quite as well as it theoretically should have.

(Humorous anecdote: During Sandy the NG grid supply failed, and the local tanks were gelling.  We called out generator maintenance guy for suggestions, and his recommendation was...drumroll please...to start a fire under the fuel tank.  I understand the theory, but  :eek1:)

Raptor

Yes we do a load test when the utility does an annual switching gear inspection. The utility shuts down the power for about 30 minutes for the inspection. It is not a full load since it done on a weekend when not many people are in the building. That said it is a test of system.
As for the advice about building a fire under a fuel tank. Well
I cannot imagine what could go wrong with that. :clownshoes:
Folks you are on your own...Plan and act accordingly!

I will never claim to have all the answers. Depending upon the subject; I am also aware that I may not have all the questions much less the answers. As a result I am always willing to listen to others and work with them to arrive at the right answers to the applicable questions.

EBuff75

Quote from: boskone on July 24, 2021, 02:41:55 PM
(Humorous anecdote: During Sandy the NG grid supply failed, and the local tanks were gelling.  We called out generator maintenance guy for suggestions, and his recommendation was...drumroll please...to start a fire under the fuel tank.  I understand the theory, but  :eek1: )
Reminds me of a story that a friend of mine who works part-time with the local government told me recently:
 
A few years back, the power went out during the winter months, and the police department wasn't able to get their generator started.  When they checked to see what the problem was, they found that the battery warmers hadn't been properly reinstalled after a service call. Instead, they were laying on the floor next to the batteries which had frozen without the warmers in place.  Since it's a big diesel generator, they didn't have anything which could be used to jump-start it and without the generator, they were completely dead in the water - no phones, radio/dispatch, lights, heating, etc..

Fortunately, someone remembered that Detroit Diesel isn't very far away and that they would likely have the right equipment to jump-start a large diesel engine.  The police managed to get in touch with someone at DD who confirmed that, yes, they did have a service vehicle which could manage and yes, they would be happy to send it over to assist.  The PD gladly provided a police escort for the vehicle, which arrived a few minutes later and had no problem getting the generator started. 

Needless to say the contractor who had been responsible for servicing the generator was NOT happy with their tech, as it meant that they had to go back to all of his recent jobs to ensure that he hadn't botched anything else.
Information - it's all a battle for information. You have to know what's happening if you're going to do anything about it. - Tom Clancy, Patriot Games

aikorob

Quote from: Raptor on July 23, 2021, 03:44:06 PM

Everyone tends to focus on the size of the generator in KW. I put forth that the absolute most important thing about a generator is fuel...............

.......................Before you buy a generator give long and hard thought to how you will keep it fueled up and running.

about 20ish years ago, remnants of a hurricane came through ATL area. I was outside metro, but close enough to receive FM.
Power was out for 10 days in some areas. Along about the 5th day or so, this particular station's GM came on the air, to make an appeal for diesel. They were down to 10-12 hours of run time on their generator, and had run through their list of local suppliers. Pretty much name your price if you could bring 500 gallons
I  hate to advocate drugs, alcohol, violence, or insanity to anyone, but they've always worked for me.

MPMalloy

Most excellent thread!

A few days ago, power went out here, not due to weather.  The cause is still unknown.  I immediately called the utility and reported the outage (automated system) & spoke with management.  I received a text back from the utility within a few minutes & they stayed in contact like that throughout the outage..  The UPS that I have my CPAP plugged into reported 1 hr of run time.  I addressed this by replacing it with a larger model that I already had.  The main UPS (APC Pro BX 1500 w/External Battery pack)  Continues to function as designed, but is showing it's age.  I immediately shut it down because I didn't now how long power would be out.

Outages are concerning to me. The Kitts really didn't know what to think & it was interesting to watch them.  Luckily, it was not out long, although I could have taken us to the car & used it for shelter & AC if I had to.

Once again, my headlamps rule the darkness!

Raptor

Quote from: MPMalloy on August 01, 2021, 06:35:23 PM

Once again, my headlamps rule the darkness!

Head lamps inside are perfect for many many tasks including simply walking around.
Folks you are on your own...Plan and act accordingly!

I will never claim to have all the answers. Depending upon the subject; I am also aware that I may not have all the questions much less the answers. As a result I am always willing to listen to others and work with them to arrive at the right answers to the applicable questions.

MPMalloy

Quote from: Raptor on August 01, 2021, 08:20:33 PM
Quote from: MPMalloy on August 01, 2021, 06:35:23 PMOnce again, my headlamps rule the darkness!
Head lamps inside are perfect for many many tasks including simply walking around.
The rank up there with microwave popcorn & drop-free mags :D

Raptor

Quote from: MPMalloy on August 01, 2021, 11:26:11 PM
Quote from: Raptor on August 01, 2021, 08:20:33 PM
Quote from: MPMalloy on August 01, 2021, 06:35:23 PMOnce again, my headlamps rule the darkness!
Head lamps inside are perfect for many many tasks including simply walking around.
The rank up there with microwave popcorn & drop-free mags :D

I bought these and they are great because they use a single AA battery and has a flood light that makes navigating inside a lot easier for me. With 2 of these and 8 AA batteries you have about 4 nights of light for 2 people or 8 nights for one and a spare. All for less than $20.
  https://smile.amazon.com/gp/product/B07GQZ33C2/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1
Folks you are on your own...Plan and act accordingly!

I will never claim to have all the answers. Depending upon the subject; I am also aware that I may not have all the questions much less the answers. As a result I am always willing to listen to others and work with them to arrive at the right answers to the applicable questions.

TACAIR

In a wide area disaster, a major issue with fuel suppliers is the lack of staff. 

As a Certified Recovery Planner, the most persistent problem within my management chain (a major telco)   was the mistaken belief that all staff people will come to work...    Not the case.  Many will stay home to protect their home/ family/ assets.

Vendors/subcontractors - like a fuel supplier / genset maintenance faces the same issue set..    A Plan B and C is a good start, but when it really hit the fan - like Katrina, all bets  are off.   A major cell phone provider kept getting their fuel trucks hijacked  by 'the authorities' and soon quite trying to restore cell service. 

Prepare for the worst and hope for the best...
I'd much rather be a disappointed pessimist than a horrified optimist....

My fiction work is fond here:
https://www.amazon.com/D-K-Richardson/e/B005JT4QP2/ref=dp_byline_cont_ebooks_1

Raptor

Quote from: TACAIR on August 18, 2021, 05:46:49 PM

Vendors/subcontractors - like a fuel supplier / genset maintenance faces the same issue set..    A Plan B and C is a good start, but when it really hit the fan - like Katrina, all bets  are off.   A major cell phone provider kept getting their fuel trucks hijacked  by 'the authorities' and soon quite trying to restore cell service. 

I could find diesel after about a week but I had to go get myself just for that reason. I had access to literally millions of gallons of diesel in a fuel jobber's supply tank on the MS river but I had to retrieve it myself 110 gallons (2-55 gallon drums) at a time.

Quote from: TACAIR on August 18, 2021, 05:46:49 PM
Prepare for the worst and hope for the best...

Remember in a SHTF situation you are on your own...plan accordingly.
Folks you are on your own...Plan and act accordingly!

I will never claim to have all the answers. Depending upon the subject; I am also aware that I may not have all the questions much less the answers. As a result I am always willing to listen to others and work with them to arrive at the right answers to the applicable questions.

Catshooter

Raptor,

Your experiences mirror mine exactly.  You must be a genius!

Fuel management is of critical importance and easily overlooked.  I am an electrician and once ran a job where we installed three 3,000,000 watt gen sets.  Each sat on it's own 1,500 gallon diesel tank.  Now that can sound like a good amount of fuel, but at full load each generator ate 110 gallons per hour.

The physical arrangements were such that a man could get a 55 gallon drum on a hand truck in to re-fuel.  Just.  Of course a fuel truck with his hose would have no problem.  As long as one was available and could access the site.

Since this was a internet sever farm, full load was a normal thing.  Less than a 100% electrical supply was unacceptable and incredibly expensive.

Good luck on your next excitment!

SMF spam blocked by CleanTalk