TV News February 2018

WHEAT:NEWS TV  FEBRUARY 2018  Volume 5, Number 2

5 Disaster Proof Tips for Your Network

WheatNewsFeb2018 Disaster

Here at the Wheat factory in New Bern, North Carolina, we know a thing or two about disaster proofing systems, both as an IP audio network manufacturer and as a long-time resident in hurricane country. During a recent hurricane encounter, the following made our top-five list of simple things you can do to disaster proof your network.

1. Use separate electrical circuits. You put in redundant power supplies, redundant computers, redundant everything but you’re running all of that off the same electrical circuit? No. Just no. Separate your operation into zones, and run each zone off a separate electrical circuit so if one gets overloaded, the others will keep things humming along. 

2. Add network switches. You can probably get away with one network switch for all your studios if yours is a small operation, but don’t do it. It’s far better to add edge switches to each studio or studio group than to take the chance that that one switch won’t fail and take the entire network down with it. Your network should have a main core Ethernet switch, and at least one edge switch. It’s also a good idea to set up redundant master switches and to cross couple studios to edge switches. In the event of an emergency, you can combine the automation outputs into one feed and route that down the network and out to air. And here’s something else to keep in mind: if it comes down to it, you can always run audio through one I/O BLADE – it has silence sensing, GPIO and mixing inside.

3. Get backup audio to the transmitter. No doubt, your transmitter site has backup power. But don’t forget the backup audio loop at the transmitter. This is important should you lose power in the control room or if the link goes out; you will still be able to get an emergency broadcast out. This can be a regular recorder at the transmitter site rigged to turn on in an emergency, or you can add this contingency as part of your I/O BLADE-3 at the transmitter. (BLADE-3 access units come with optional embedded audio storage and playback. Should the transmitter site lose connection with the studio, a BLADE-3 at the transmitter site can play back audio; audio playback can be automatically triggered by silence detection within the BLADE-3 or through an LIO triggered from a console, button or a script running within the WheatNet-IP system.)

4. Consider two-for-ones. If possible, get gear that can do more than one thing so you’ll have several layers of redundancy to fall back on in an emergency. We’re firm believers in this, as you’ll discover throughout our product lines.

5. Have an exit plan. If you think you have to be glued to the studio during a flood, fire, tornado or other disaster, think again. Any quality IP audio network built for broadcasting should let you move station operations to higher (or lower) ground. In the case of WheatNet-IP, we have remote software that lets you load a virtual console onto your laptop for continued broadcasting from just about anywhere.

 

IP QA

Q: What does an IP audio network do that is different than, say, a regular network?

A: Enterprise networks were not intended for real-time audio or video delivery. The biggest issue is that they distribute packets in a non-deterministic manner, which can lead to dropped packets or jitter as the traffic increases on the network. IP audio networks built for broadcast, like WheatNet-IP, talk both IP and audio, so they prioritize traffic to reduce packet dropouts and other issues that can affect the quality of audio. Because they are broadcast specific, IP audio networks also serve the purpose of controlling and managing audio devices and mixes, as well as bring audio into the network from mics and other sources.

It’s all explained in our free ebook. Download your copy of IP Audio for TV Production and Beyond

ATL Atlanta. What A Disaster.

AtlantaDarkThenCellPhoneFlashlightsBy Scott Johnson

Can your facility continue to operate even if a critical component fails? The parties responsible for the power systems at Hartsfield-Jackson Atlanta International Airport probably thought they had every scenario covered and that no single-point failure would take them out of business. Close, but no cigar. That became clear late last year when a sudden power outage brought the busy airport to a standstill, grounding more than 1,000 flights and stranding thousands of passengers. There are some important lessons broadcasters can take away from the events that unfolded at ATL.

On the afternoon of December 17, 2017, a piece of underground high-voltage switchgear belonging to public utility Georgia Power failed in a spectacular way. At least it would have been spectacular had there been anyone there to see it. The equipment, located in an underground vault, developed an arc and caught fire. Power throughout the airport’s terminals, gates, and transportation facilities progressively failed as the fault spread.

Ordinarily, a power outage at the airport is no big deal. The FAA control tower, navigational aids, and certain other systems have their own generators or backup power. And the airport itself maintains very large diesel generator plants, designed to automatically pick up the load should there be a failure of the off-site utility power. Having spent a lot of time around that airport, I’ve seen these plants and remarked to myself on the apparent safety they represented.

But as passengers waited in the terminals, nothing seemed to happen. The lights didn’t come back on, but the PA system did remain operational. Unfortunately, it offered no information other than the same recorded, generic messages over and over. Airport and airline employees, like the passengers who sought them out, seemed to be in the dark.

 

Click To Continue

Why weren’t the diesels, designed for just such a contingency, picking up the load and getting the airport back in business? The reason is simple, but maddening. When the airport was originally laid out, power entered the central complex via high voltage feeders running through an underground tunnel. And later, when the large auxiliary power plants were built, the lines connecting them to the complex were also routed through that same tunnel. And both sets of lines ran adjacent to and connected with the vault containing Georgia Power’s switchgear. The lines were within a few feet of each other and within a scant dozen feet from the failed equipment, in a very confined space.

Darkness falls across the area shortly after 5 p.m. this time of year, and as even the sunlight from the windows faded, passengers found themselves wandering through the terminals in pitch darkness as the battery-powered emergency lights ran down. Escalators, which normally just become stairs when the power fails, were closed off by airport police for fear of people falling in the darkness. The airport’s dedicated underground train, which transports passengers between terminals, simply stopped and had to be evacuated. People upstairs were stuck there.

The fire and arc blast from one failed piece of equipment had damaged the primary feed lines AND the lines from the redundant power system. One big bang disabled everything. Of course, emergency crews responding to the situation had their own problems to contend with. Airport fire crews knew that before even thinking about restoring power, the fire had to be extinguished. This proved difficult, as the fire filled the entire tunnel with thick, toxic fumes and smoke. Some of it even seeped into parts of Terminal D, frightening passengers and aggravating the conditions of passengers with breathing difficulties. Firefighters battled through the tunnel wearing respirators and within hours, had managed to put out the flames. Until they were sure things were cooled down and free of toxic gases, Georgia Power crews could not even access the failed equipment or begin to assess the damage.

The airport, known to the airlines and FAA as ATL, is the busiest airport in the world. As the headquarters and largest hub for Delta airlines as well as a hub for several other carriers, it’s a key link in the country’s air travel industry. As an old saying goes, when you die, whether you’re going to heaven or hell, you’ll have to change planes in Atlanta. With the airport’s passenger-handling facilities literally dead and dark, there was little choice but to sever that link. The airport closed to traffic. Outgoing flights were put on “ground stop” by the FAA, meaning they would not be allowed to take off. Neither could they easily return to a gate. So they parked and sat there, full of irate, sweaty, hungry, thirsty passengers. Incoming flights suffered the same fate if they were not diverted to other airports in time. Soon, the usual steady roar of turbofan engines became an ominous silence as the runways handled no planes and the ramps and taxiways became crowded with parked aircraft.

This wasn’t just any Sunday afternoon. At the peak of holiday travel season, the airport was absolutely packed with passengers. And not just typical passengers; according to one airline official, the holidays mean that the very young as well as the very old are among the traveling public in much larger numbers than usual. And with no air handling systems and precious little food or drink available, these folks suffered more than their share of inconvenience, if not downright danger, particularly in areas where smoke was a problem.

A full investigation will take some time, but I think we can now see the lesson in this: in order for a system to be truly redundant, all of the parts of it that could suffer any credible failure should be both functionally and physically separate. The airport had multiple diesel backup generator plants, two power feeds, two sets of switchgear, and two control systems. But somehow, either in the original design or during some subsequent upgrade, nobody looked at the one-line diagram and said, “What happens if this piece faults in a way that makes it blow up and catch fire?” That fire caused what’s called a single-point failure — where one fault in one place takes out the whole shooting match.

So as you periodically review and test your station or facility’s engineered redundancy (you DO do that, right?) remember to take this sort of unexpected interaction into account. Because in this case, someone missed a big one, and close only counts in the game of horseshoes.

 

Scott Johnson is the systems engineer and webmaster for Wheatstone.

 

DOWNLOAD YOUR FREE
AUDIO OVER IP E-BOOK

IP TV EBOOK COVER

Putting together a new studio? Updating an existing studio? 

We've put together this IP Audio for TV Production and Beyond e-book with fresh info and some of the articles that we've authored for our website, white papers, and news that dives into some of the cool stuff you can do with a modern AoIP network like Wheatstone's WheatNet-IP. And it's FREE to download!

Just click here or on the cover.

Wheatstone TV Consoles Video Tour

Phil Owens gives a tour of each of Wheatstone's TV audio consoles/control surfaces, explaining the differences between them to help give you a better idea of what you'll need to know.

Stay up to date on the world of broadcast radio / television.
Click here to subscribe to our monthly newsletter.

Got feedback or questions? Click my name below to send us an e-mail. You can also use the links at the top or bottom of the page to follow us on popular social networking sites and the tabs will take you to our most often visited pages.

-- Scott Johnson, Editor

Site Navigations