If your company utilized a BIA (Business Impact Analysis) prior to a disaster, then the weeks following the disaster will actually assess the goals and objectives addressed by the BIA. The disaster has given your company an opportunity to test your Disaster Recovery plan, your Continuity Plan and allows you to identify what went ‘right’ and what went ‘wrong‘.
If your company hasn’t performed a Business Impact Analysis or doesn‘t have a Disaster Recovery plan in place, then you might find this article useful in its examination of the voice communications and general lessons learned from Hurricane Katrina.
Like many companies, the company I contract with has a Disaster Recovery plan. I’m going to approach this topic from the point-of-view of assuming your PBX/Telecommunications department has embraced a BIA or has non-tested Disaster Recovery Plan n place.
This article will be broken into several topics, with each topic providing a transitional phase to the next. The topics are;
Contingency Planning and Management – Disaster Recovery.
How to identify and rate risks.
Succession Planning.
ORA – Obvious Relationship Awareness.
An Overview: Contingency Planning and Management – “Disaster Recovery”. Voice Communication and General Lessons From Hurricane Katrina
Disaster Recovery can be defined as the identification of critical core component business functions and the processes designed to protect and/or recovery those functions in the event of a disaster. The strategy is to be prepared with a Disaster Recovery plan based on facts, not assumptions or opinions.
Disaster Recovery should certainly be addressed by your company. If your company isn’t prepared for a disaster and is the victim of a disaster, then the recovery process is going to be difficult, precious man-hours/money will be wasted and reactionary rather than proactive planning will be the norm.
I will not go over the BIA process or the actual preparation of a Disaster Recovery plan, as there are plenty of resources on the internet and books available on the subject. Links are available at the end of this article. I will however, discuss some items and real-life experiences that might prove useful when developing your BIA and Disaster Recovery.
Identifying Risks:
Identifying risk is a three is step process; Identify the risk, rate the risk and developing a constructive and practical action plan for the risk(s). I suggest taking the traditional risk identification process a step further and identify your risk(s) for each month of the year.
Step 1: Identification Phase.
Identify your risk. Risk are everywhere and abundant- consider the following:
A. Is your business in a flood zone?
B. Is your business in a tornado zone?
C. Is your business in a hurricane zone?
D. Is your business in an earthquake zone?
E. Is your business susceptible to forest fires?
F. Is your business susceptible to industrial fires?
G. Is your business near a rail station, airport or ship port?
H. Is your business near a government facility?
I. Is your business heavily dependent on outside/external companies?
J. Is your business limited by inbound/outbound roads/bridges?
Risks (A) through (E) are considered natural disasters and may effect entire communities.
Risk (F) may subject your company to mass evacuations and/or temporary shutdowns due to industrial fires or accidents.
Risk (G) may subject your company to mass evacuations and/or temporary shutdowns due to terrorist activities or Hazardous Material spills.
Risk (H) may subject your company to mass evacuations and/or temporary shutdowns due to terrorist activities or threats.
Risk (I) may subject your company to external financial and/or process problems. Imagine if one of your important suppliers suddenly claims bankruptcy or is the target of a hostile takeover or merger /acquisition by one of your competitors.
Risk (J) may subject your company to loss revenue if a main road/highway or bridge is damaged. Imagine if you company is on a peninsula and the main bridge or causeway is impassable.
Step 2. Rating the risk:
Use this Risk Assessment Card to identify risks and rate each one.
Risk Rating* for Each Month
Risk J F M A M J J A S O N D Total
Flooding
Tornados
Hurricanes
Earthquakes
Forest Fires
Industrial Fires/Accidents
Terrorist/Hazardous Materials
Government Imposed Shutdowns
External Processes
Roadways
* – Risks are rated by the probability of the risk occurring and it’s impact on the business. Probability of risk is 1=Low (none or very small chance of the risk occurring), 2=Medium (risk may occur), 3=High. (risk will most likely occur or is imminent) .
*- Impact on Business is a rating of loss of revenue and recovery costs. 1=Low (Loss of revenue and recovery cost is minimal ), 2=Medium (Loss of revenue and/or recovery cost are significant), 3=High (Loss of revenue and recovery cost are significant).
Have each member of your telecommunications department complete the Risk Assessment card and use it as a basis for a brainstorming session. Remember that rigidity can be counter-productive, be open to your employees opinions.
As an example, if your company is on the Gulf Coast, it will be subjected to Hurricane Season. A risk assessment for a Gulf Coast company may look like this:
Risk Rating* for Each Month
Risk J F M A M J J A S O N D Total
Flooding 1 1 1 1 3 3 3 3 3 3 3 1 26
Tornados 0 0 0 1 2 2 2 2 2 2 1 0 14
Hurricanes 0 0 0 0 0 6 6 6 6 6 6 0 36
Earthquakes 0 0 0 0 0 0 0 0 0 0 0 0 0
Forest Fires 0 0 0 0 0 0 0 0 0 0 0 0 0
Industrial Fires/Accidents 0 0 0 0 0 0 0 0 0 0 0 0 0
Terrorist/Hazardous Materials 2 2 2 2 2 2 2 2 2 2 2 2 24
Government Imposed Shutdowns 2 2 2 2 2 2 2 2 2 2 2 2 24
External Processes 0 0 0 0 0 0 0 0 0 0 0 0 0
Roadways 2 2 2 2 2 2 2 2 2 2 2 2 24
Flooding – The company is on the water but is physically elevated. History has shown that the area floods, but is usually caused by storm surge. This is why the risk rating increases during the Hurricane months.
Tornados – Tornados are known to accompany hurricanes and tropical depressions.
Hurricanes – During the months of April through November, the company is highly susceptible to hurricanes which can result in significant loss of revenue and recovery cost.
Earthquakes – The company isn’t in a known earthquake zone. This risk can be ignored.
Forest Fires – The company isn’t near a forest. This risk can be ignored.
Industrial/Accidents – The company isn’t near an industrial park. This risk can be ignored.
Terrorist/HazMat – The company isn’t a traditional target of terrorist activities and isn’t near a company that is a traditional target of terrorist activity. The company is near a railway and may be subject to evacuation caused by a HazMat spill.
Government Shutdowns – The company is near a military base and may be subject to shutdowns or curfews.
External Processes – The company manages many of its processes internally.
Roadways – The company is isolated between two bridges and a main highway. It is possible that an accident may render the bridge impassable, severely limiting employee and client travel.
Thus, according to the Risk Assessment card, the company should focus on Hurricane Disasters (36) before Flooding (26) or Tornado (14) disasters and the company can basically ignore earthquakes and forest fires.
You may also see a trend in May that suggest the company should be;
1. Concerned with Flooding and Tornados before Hurricanes, and
2. The company should be preparing themselves for Hurricane Season.
In addition, the company should be concerned and knowledgeable of alternate roadways and possibly lobbying state/federal agencies for improved access.
Step 3: Developing Constructive and practical action plans.
According to the assessment, hurricanes are the primary threat to the company. The company could use the assessment to establish a Hurricane Preparedness Program;
Promote Understanding of Hurricanes and Their Effects,
Work better to define Hurricane Risks to company functions,
Improve Hurricane-resistant and protection measures,
Encourage the use of Hurricane-Safe Policies and Planning.
Also, the company should also be prepared for flooding and tornados.
Succession Planning:
Succession is defined as:
(1.) series in time: a sequence of people or things coming one after the other in time
(2.) following: the following of one thing after another
(3.) taking up of title or position: the assumption of a position or title, the right to take it up, or the order in which it is taken up .
Succession planning, as used in this article and in the traditional business sense, is simply; identifying ‘who does what’ and ‘who can do it’ if that person isn’t available. You may recognize this as ‘Continuity Planning’ and it is that, plus two additional invaluable considerations – what tools are needed and can it be done remotely?
Succession planning is more that just pasting your organizational chart into a table and labeling it “Succession Plan”. Just as the company needs to identify and rate risk, it will need to identify personnel critical to core components of a company and if those activities can be done remotely, what tools are needed and who is the most practical back-up person.
I’ll use your companies PBX Manager as an example;
Your PBX Manager does much more than manage the telecommunications technicians, prioritizing service calls/work order request and managing a operator staff. He/she is most likely responsible for all of your telecommunication service contracts, your cellular contracts, your pager contracts and may even be responsible for your mobile radios.
He/she will know what your current systems are capable of and possesses insight to internal processes that don’t convey well in a written continuity plan. True continuity isn’t a snapshot of the company at a specific time, but a viable, ever-changing state of business circumstances that only your PBX Manager knows.
As an example, your PBX Manager may be the only person who knows that your pager bill was floating around in an inner-office envelope and is thirty days past due just before the company fell victim to the disaster.
This isn’t to say that Continuity planning should be ignored – quite the opposite. Those processes that can be mapped should be in written/printed format and easily portable. Succession planning is just an important add-on to this procedure, especially for those risk that have some forewarning.
In the risk assessment above, the company identified hurricanes as a primary risk to the company. Fortunately, hurricanes come with some warning. In this instance, a Succession Plan is an excellent way to mitigate the risk to the continuity of the companies voice communications.
Not only do hurricanes come with some forewarning, they are usually accompanied by mandatory evacuations. Lets assume that the company knows that a hurricane is heading their way. The company has prepared the property according to the Hurricane Preparedness program, has educated its employees and heeded the mandatory evacuations. What happens next?
In many cases, ‘What happens next?’ is a play-it-by-ear process, but with a Succession Plan in place – “What happens next – .” is an actionable item, a process – not a question. It’s a prepared process not a reactionary plan.
Suggestions to improve your Succession Plan.
Step 1: Follow the traditional guideline in creating a Succession Plan and Continuity Plan.
Step 2: Have a Succession meeting prior to the disaster, if possible.
Go over your latest Continuity plan and pen-correct those items that have changed.
Obtain the latest problems that haven’t been resolved.
Update important numbers and collect where all team members will be staying,
Step 3: Continuity planning + = An Improved Succession Plan.
Your PBX Manager should be leaving the area to conduct business from a pre-paid, pre-planned location. He/she should know where all the other critical members on the company will be located at (this should be pre-planned as well).
The location can be a hotel room safe from the hurricane or a sister property not in the danger area, or an Emergency Operations Center not in the danger area,
Your PBX Manager should have all the documents of the Continuity Plan,
Your PBX Manager should have a fax machine or access to a fax machine,
Your PBX Manager should forward all calls to his/her office phone to the remote locations number,
Your PBX Manager should be prepared to operate from the temporary location for at least several weeks.
How is this different that a normal Succession Plan? The answer: the pre-paid and pre-planned qualifiers. For those who’ve attempted an late-hour evacuation of an area, you know how hard it is to find a hotel room. You will spend valuable time calling area hotels that can be spent finalizing last minute company business and preparing your personal property.
When hurricane Katrina landed, we had key/critical personnel all over the United States and had to wait for them to check-in. It would’ve made the recovery process simpler had they all be in a central, pre-paid, pre-planned location.
Voice Communication Lessons learned from Hurricane Katrina.
Lesson 1: Your Hot Site/Emergency Operations Center.
The company I contract with has a designated EOC, with a small NorStar system for its voice communications. Two days after Hurricane Katrina – the company knew that the damage to the primary building was extensive and decided to upgrade to a BCM-400, an ISDN-PRI and a 100 DID Number Range at the EOC. The equipment and facilities were ordered from our Regional Bell and the company waited.
The Lesson: I knew that once an area is declared a natural disaster, the areas recovery is controlled by FEMA and Homeland Security. What I did not know is that once Homeland Security takes over that the Regional Bell, Regional Power and other public services follow priorities set by Homeland Security – thus we waited for several weeks before our phone service was turned up.
Lesson 2: Your Succession Plan should have a ‘return-rate-of-staffing‘.
In addition to the tips I offered above with respect to Succession Planning, you should also consider ‘Uncertainty’. Uncertainty is a terrible feeling. Do I still have a job? How did my home make out? What happens next? I’m not a social scientist, but I observed and had many of these feelings. What I noticed is people want some sense of normalcy – and being that we spend so much time at work – people want to return to work.
This is a good article on the Human Factor. ([http://www.ccep.ca/news0205.html])
The Lesson: Your EOC isn’t designed to support non-critical personnel. If personnel who don’t necessarily have to be at ground-zero start returning to work, it places a huge burden on the companies limited resources at the EOC, both data and voice. This burden, plus the fact that you may now need to order additional equipment and/or services to support the added staff – is once again subject to the priorities set forth by Homeland Security.
Return-rate-of-staffing is just a term I’m using to suggest that personnel shouldn’t return to the company just because there is a desk available. The return-rate should be controlled by;
A) There is a desk available,
B) It has a working phone/fax machine,
C) It has a working networked computer,
D) This job is best done from this location.
If there aren’t return-rate-of-staffing qualifiers, then people will return to a desk and start asking for ;
A) A working phone/fax machine,
B) A working networked computer.
This of course, places a burden on your technicians, your systems and the supply line.
Obvious Relationship Awareness (ORA):
ORA is a term I created in my fictional book, “Chaos Theorem” and it is meant to be a play on the word aura. ORA, “Obvious Relationship Awareness”, basically means ‘everything is related and the relational paths are connectors’. Everything is related in the big scheme of things and the connectors are often taken for granted.
The Lesson: Take account of those processes in your company that you can’t control. For example; the fact that we ordered an ISDN-PRI but had to wait until our Regional Bell fulfilled it’s responsibilities to Homeland Security.
Some other examples of ORA:
Take note of your supply lines feeding the company – In the days that followed Hurricane Katrina, the supply lines for any commodity (gas, food, ice, water, electronics, etc) were extremely limited. Shelves sat empty at Best Buy, WalMart, and Circuit City. I lost my home PC and laptop in the storm and had to buy a very expensive laptop (one of the only models left on the shelf) in order to program the new BCM-400 that was purchased.
Map your connectors – know what businesses your company relies heavily on and ‘connect’ beyond them. In effect, know your sources – sources. After a disaster, the company your company relies on might be out-of-business.
Expect water and sewage services to be interrupted. Do you have enough water at your EOC to support your staff? Also consider food and waste disposal.
Consider support materials that you would not normally consider. For example; diesel fuel to run generators and is the supply chain in-tact? Generators go through fuel a lot faster than you think. Is someone on the staff knowledgeable in maintaining diesel generators? Who is responsible for ensuring that it’s operational and has fuel on a 24/7 basis?
Uninterruptible Power Supplies (UPS) – are usually mounted in the lower portion of your communication racks and are an integral part of your voice/data systems. Are they susceptible to flooding? If the UPS’s have to be replaced do you have the manpower? (Some UPS systems are extremely heavy). And, if they can’t be replaced do you have enough multi-outlet extension cords to plug in all the switches and routers?
And finally, will your Regional Bell half-tap existing T1’s to an alternate location?
Article by Charles Carter
Website links for, BIA, Business Continuity and Disaster Planning:
Succession Planning (http://www.bdc.ca/en/my_project/Projects/growth/succession_planning.htm?cookie%5Ftest=1)
Business Continuity ([http://www.thebci.org/London%20Firsts.PDF])
Workforce Planning ([http://www.hr.state.tx.us/workforce/04-704%20WorkforceGuide.PDF])
General Business Link (http://www.businesslink.gov.uk/bdotg/action/Title)
Disaster Recovery (http://www.disasterrecoveryworld.com/)
FEMA’s Document Library ([http://www.fema.gov/library/prepandprev.shtm])
National Response Plan (http://www.dhs.gov/dhspublic/interapp/editorial/editorial_0566.xml)
Article by Charles Carter
Pbxinfo.com
And
CS2 Communications, LLC
This article is available in PDF format in [http://www/pbxinfo.com] Downloads.
Note: This article contains tables. If you can not view the tables please download the PDF file from http://www.pbxinfo.com