Incident Response – Are we adequately prepared?
Recently, Heavy Reading made public their findings from a survey of mobile network operators, conducted this past August. (Mobile Network Outages & Service Degradations: A Heavy Reading Survey Analysis. October, 2013).
Though much attention was given to the headline-grabbing estimate of what these operators spend to rectify and restore network outages and degradations, one finding that I did not see in any headline should be of particular interest to Carrier Ethernet service providers, and their suppliers.
In announcing the release of the report, Heavy Reading’s lead author Patrick Donegan said the following:
“A lot of incidents are triggered – and the duration of incidents and impact on customers is often rendered unnecessarily long – as a consequence of inadequate training of operations personnel and inadequate communication between different teams, including teams representing the vendor and teams representing the operator."
Talk about stating the obvious! Most managers of service provider Network Operations and Service Assurance teams would merely smile at this “finding”. Yet, my experience monitoring network issues reported by customers, and the trends I have been witnessing over many years supporting both mobile and Carrier Ethernet service providers, suggest that at least two factors may be having a real impact on network availability.
- Service providers continue to put downward pressure on operational fixed costs. There are countless examples of service provider reductions in staff and consolidation of NOC/service assurance resources, resulting in fewer eyes, hands and minds in these critical functions. Too often, I hear this directly from our customers’ employees who are quick to share their frustrations with a member of the Overture Customer Support team.
- On-the-job training on new HW/SW platforms is becoming a thing of the past. My first-hand discussions with service provider executives show that, despite their desire for vendors to provide training to their teams, the time and resources allocated to do so continue to be constrained. Many managers are content for vendors to simply supply animated slideshows, yet the individuals responsible for ongoing operation of the network feel they have little support and no time for self-study.
The negative effects of these pressures may be best reflected in yet another finding, summarized in Heavy Reading’s announcement of their report, that “less than half (44 percent) consider their understanding of outages and degradations to be ‘good.’”
This was confirmed for me recently, in a meeting with executives and managers responsible for network service assurance at a Tier I business service provider. The management team expressed concern that they were currently reacting to outages too quickly without adequate understanding of potential root causes. Given their time and resource constraints, they had not been able to commit to the training needed for their operations teams to spot these causes proactively, before future outages occurred. In this particular case, Overture stepped in to fill the void. We responded by scheduling an in-depth review of the incident - in person over multiple days – with the appropriate subject matter experts (SME). The end result was a better informed incident response process, and a real-world demonstration of the power that comes from collaboration between service provider and vendor.
Overture continues to look for ways to enhance our customers’ experience with our solutions. If you have a story to share about your positive experience with a customer support organization – at Overture or elsewhere – I would love to hear from you. Or if you would like to discuss the benefits of a session between members of your team and an Overture SME, please drop me a note. Just send an email to firstname.lastname@example.org or call me.
About the Author
Keith Donahue is Overture's vice president of customer service. Before joining Overture, Keith led the network services business unit of Aviat networks, a provider of wireless network solutions, where he grew revenue and profitability for maintenance and support services and built Aviat’s global professional service and field service delivery capabilities. Before moving into services leadership, he led product line management for Aviat. Previously, Keith drove several new business and product programs for Harris Corporation. From 1997 to 2000, he was European Marketing Manager, based in the U.K. Keith has also held product management assignments at Premisys Communications (acquired by Zhone Technologies) and AiT (acquired by 3Com). He holds a Bachelor of Arts from McGill University, Montreal, Canada and a Master of Business Administration from University College, Dublin, Ireland.