What was the major news story of this Christmas holiday? Flight cancellations and the Southwest Airlines “meltdown.” Manufacturers should be looking at that story and asking themselves if they are next. For too many manufacturers the answer is yes. Let’s take a quick look at the Southwest story and one of the lessons manufacturers should be learning from the meltdown.
Horrible weather, at the wrong time, with a very lean flight schedule, is the core reason for flight cancellations across North America at the end of 2022. However, while other airlines were able to get back to a semblance of normalcy quickly, Southwest Airlines continued to have massive cancellations for several days after. Why? The New York Times and other media outlets reported that employees believe that the company failed to invest in the technical infrastructure of operations. The Times reported that one employee had even earlier created a protest sign that said: “Another victim of SWA’s outdated technology,” with a graphic showing a stuck software progress bar. The Times went on to explain that “technical debt” was common and that it was core to the meltdown.
Why bring this up in a blog post focused on industrial operations? Simple, because “technical debt” is so common in manufacturing.
LNS Research regularly hears that:
Key manufacturing systems are dependent on one or two employees that are on the verge of retiring, or even worse, just retired.
Equipment connectivity is dependent on old hardware that can only be maintained by cannibalizing other systems available only on eBay.
The company cannot address supply chain shortages effectively because systems are too rigid to enable agile manufacturing.
Custom legacy systems abound with no upgrade, maintenance, or replacement strategies in place.
DCS systems have aged and reached obsolescence in many process environments.
Manufacturing is a laggard in technology adoption and often the “poor stepsister” within corporations. Part of that is the fact that manufacturing is capital-intensive relative to other corporate departments. Part of that is a manufacturing culture that says, “if it isn’t broke, don’t touch it.” It is exacerbated by Lean advocates that can often view technology negatively. However, the reality is that manufacturers are often just like Southwest allegedly in that they have a massive technical debt putting their operations at risk.
The question of technical debt has become mainstream. How would your company stack up?
So, what should a CEO be doing? The answer is to ask the hard questions and develop strategies to address those questions. LNS Research believes there are four questions that CEOs (and CFOs if they are reporting to the street) should be digging into with their operational leadership teams.
Have you assessed the level of technical debt across operations? Do you know your vulnerabilities?
Reporting would indicate that Southwest employees were regularly communicating the risks associated with their systems. Manufacturing tends to be more hierarchical, culturally diverse, and less likely to communicate bad news upward. Plant maturity and equipment vary widely even in the most advanced manufacturers. Few, if any, CIOs (or even their designates) understand the details of systems installed in each and every plant. Heads of worldwide manufacturing are typically not digging into system health. Technical debt information typically does not flow up to a single point of collection in most manufacturers. CEOs need to ensure they understand the risks and vulnerabilities in today’s systems.
LNS Research believes the assessment of technical debt in manufacturing requires the engagement of IT, OT, and Operations (not just OT personnel). We would encourage CEOs to force a collaborative initiative of investigation across these three domains.
Have you quantified the potential risks associated with that technical debt? Will you be able to keep shipping products if systems fail?
Has your team assessed system vulnerabilities? Personnel dependencies (could a beer truck accident involving one employee bring down a manufacturing plant)? “Black Swan” events are supposed to happen sporadically, but they seem to be hitting us regularly since 9/11. Is your manufacturing network vulnerable to a “meltdown” if we encounter another market-changing event?
What is the risk of one of your corporation's PR people being forced to tell The New York Times what a Southwest employee had to say: “Our systems were overwhelmed by the scale of the disruption?” While the distributed nature of manufacturing and the variety of systems across plants inherently reduces those risks, they still need to be studied. Often products from one small plant can hold up shipments across a supply chain because of the single source dependencies. Is there a potential for a “meltdown” in your operations?
The evaluation of risk is not just a technology question. It is a question about the potential impacts of an external or technical event. Hence, we encourage Operations to be engaged in the assessments.
Does the scale of your technical debt constitute a “risk” that should be reported in your financial statements?
Being the cynic that I am, I would forecast that unhappy investors will be suing Southwest if they did not acknowledge their system, and therefore, operational vulnerabilities in their financial risks. Reporting indicates the company was aware of the issue if not the negative potential impacts of their technical debt. How will you respond on your next earnings call when asked about a “technical debt risk assessment” across manufacturing?
Do you have a real, actionable plan to fix the technical debt?
The reality of what we hear from manufacturing IT and OT teams is that manufacturers have not developed an actionable plan to fix their technical debt nor contingency plans to handle the consequences of system failure or external disruptions. If the risks are real in your organization, you may need to focus on these plans.
In fact, LNS Research believes the ongoing assessment of technical debt must become a standard component of the corporation’s overall operational risk management program.
The New York Times was not the only mainstream media outlet to identify “technical debt” as a key factor in Southwest’s meltdown. Manufacturers need to assess their risks and develop plans to mitigate those risks with all deliberate speed.
All entries in this Industrial Transformation blog represent the opinions of the authors based on their industry experience and their view of the information collected using the methods described in our Research Integrity. All product and company names are trademarks™ or registered® trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.
LNS Research Principal Analyst, Andrew Hughes, answers the 6 most asked questions from the webcast, Digital Twin: Discrete Manufacturers Making the...
LNS Research Analyst, Patrick Fetterman, answers the 5 most asked questions from the webcast, The Road From Metrics to Analytics for Mid-Size...
LNS Research Principal Analyst, Andrew Hughes, answers the 8 most asked questions from the Making MOM Choices in a Changing World webcast held on...