(Bloomberg)
-- It remains the mystery at the heart of Boeing Co.’s 737 Max crisis:
how a company renowned for meticulous design made seemingly basic
software mistakes leading to a pair of deadly crashes. Longtime Boeing
engineers say the effort was complicated by a push to outsource work to
lower-paid contractors.
The
Max software -- plagued by issues that could keep the planes grounded
months longer after U.S. regulators this week revealed a new flaw -- was
developed at a time Boeing was laying off experienced engineers and
pressing suppliers to cut costs.
Increasingly,
the iconic American planemaker and its subcontractors have relied on
temporary workers making as little as $9 an hour to develop and test
software, often from countries lacking a deep background in aerospace --
notably India.
In
offices across from Seattle’s Boeing Field, recent college graduates
employed by the Indian software developer HCL Technologies Ltd. occupied
several rows of desks, said Mark Rabin, a former Boeing software
engineer who worked in a flight-test group that supported the Max.
The
coders from HCL were typically designing to specifications set by
Boeing. Still, “it was controversial because it was far less efficient
than Boeing engineers just writing the code,” Rabin said. Frequently, he
recalled, “it took many rounds going back and forth because the code
was not done correctly.”
Boeing’s
cultivation of Indian companies appeared to pay other dividends. In
recent years, it has won several orders for Indian military and
commercial aircraft, such as a $22 billion one in January 2017 to supply
SpiceJet Ltd. That order included 100 737-Max 8 jets and represented
Boeing’s largest order ever from an Indian airline, a coup in a country
dominated by Airbus.
Based
on resumes posted on social media, HCL engineers helped develop and
test the Max’s flight-display software, while employees from another
Indian company, Cyient Ltd., handled software for flight-test equipment.
Costly Delay
In
one post, an HCL employee summarized his duties with a reference to the
now-infamous model, which started flight tests in January 2016:
“Provided quick workaround to resolve production issue which resulted in
not delaying flight test of 737-Max (delay in each flight test will
cost very big amount for Boeing).”
Boeing
said the company did not rely on engineers from HCL and Cyient for the
Maneuvering Characteristics Augmentation System, which has been linked
to the Lion Air crash last October and the Ethiopian Airlines disaster
in March. The Chicago-based planemaker also said it didn’t rely on
either firm for another software issue disclosed after the crashes: a
cockpit warning light that wasn’t working for most buyers.
“Boeing
has many decades of experience working with supplier/partners around
the world,” a company spokesman said. “Our primary focus is on always
ensuring that our products and services are safe, of the highest quality
and comply with all applicable regulations.”
In
a statement, HCL said it “has a strong and long-standing business
relationship with The Boeing Company, and we take pride in the work we
do for all our customers. However, HCL does not comment on specific work
we do for our customers. HCL is not associated with any ongoing issues
with 737 Max.”
Recent
simulator tests by the Federal Aviation Administration suggest the
software issues on Boeing’s best-selling model run deeper. The company’s
shares fell this week after the regulator found a further problem with a
computer chip that experienced a lag in emergency response when it was
overwhelmed with data.
Engineers
who worked on the Max, which Boeing began developing eight years ago to
match a rival Airbus SE plane, have complained of pressure from
managers to limit changes that might introduce extra time or cost.
“Boeing
was doing all kinds of things, everything you can imagine, to reduce
cost, including moving work from Puget Sound, because we’d become very
expensive here,” said Rick Ludtke, a former Boeing flight controls
engineer laid off in 2017. “All that’s very understandable if you think
of it from a business perspective. Slowly over time it appears that’s
eroded the ability for Puget Sound designers to design.”
Rabin,
the former software engineer, recalled one manager saying at an
all-hands meeting that Boeing didn’t need senior engineers because its
products were mature. “I was shocked that in a room full of a couple
hundred mostly senior engineers we were being told that we weren’t
needed,” said Rabin, who was laid off in 2015.
The
typical jetliner has millions of parts -- and millions of lines of code
-- and Boeing has long turned over large portions of the work to
suppliers who follow its detailed design blueprints.
Starting
with the 787 Dreamliner, launched in 2004, it sought to increase
profits by instead providing high-level specifications and then asking
suppliers to design more parts themselves. The thinking was “they’re the
experts, you see, and they will take care of all of this stuff for us,”
said Frank McCormick, a former Boeing flight-controls software engineer
who later worked as a consultant to regulators and manufacturers. “This
was just nonsense.”
Sales
are another reason to send the work overseas. In exchange for an $11
billion order in 2005 from Air India, Boeing promised to invest $1.7
billion in Indian companies. That was a boon for HCL and other software
developers from India, such as Cyient, whose engineers were widely used
in computer-services industries but not yet prominent in aerospace.
Rockwell
Collins, which makes cockpit electronics, had been among the first
aerospace companies to source significant work in India in 2000, when
HCL began testing software there for the Cedar Rapids, Iowa-based
company. By 2010, HCL employed more than 400 people at design,
development and verification centers for Rockwell Collins in Chennai and
Bangalore.
That
same year, Boeing opened what it called a “center of excellence” with
HCL in Chennai, saying the companies would partner “to create software
critical for flight test.” In 2011, Boeing named Cyient, then known as
Infotech, to a list of its “suppliers of the year” for design, stress
analysis and software engineering on the 787 and the 747-8 at another
center in Hyderabad.
The
Boeing rival also relies in part on offshore engineers. In addition to
supporting sales, the planemakers say global design teams add efficiency
as they work around the clock. But outsourcing has long been a sore
point for some Boeing engineers, who, in addition to fearing job losses
say it has led to communications issues and mistakes.
Moscow Mistakes
Boeing
has also expanded a design center in Moscow. At a meeting with a chief
787 engineer in 2008, one staffer complained about sending drawings back
to a team in Russia 18 times before they understood that the smoke
detectors needed to be connected to the electrical system, said Cynthia
Cole, a former Boeing engineer who headed the engineers’ union from 2006
to 2010.
“Engineering
started becoming a commodity,” said Vance Hilderman, who co-founded a
company called TekSci that supplied aerospace contract engineers and
began losing work to overseas competitors in the early 2000s.
U.S.-based
avionics companies in particular moved aggressively, shifting more than
30% of their software engineering offshore versus 10% for
European-based firms in recent years, said Hilderman, an avionics safety
consultant with three decades of experience whose recent clients
include most of the major Boeing suppliers.
With
a strong dollar, a big part of the attraction was price. Engineers in
India made around $5 an hour; it’s now $9 or $10, compared with $35 to
$40 for those in the U.S. on an H1B visa, he said. But he’d tell clients
the cheaper hourly wage equated to more like $80 because of the need
for supervision, and he said his firm won back some business to fix
mistakes.
HCL,
once known as Hindustan Computers, was founded in 1976 by billionaire
Shiv Nadar and now has more than $8.6 billion in annual sales. With
18,000 employees in the U.S. and 15,000 in Europe, HCL is a global
company and has deep expertise in computing, said Sukamal Banerjee, a
vice president. It has won business from Boeing on that basis, not on
price, he said: “We came from a strong R&D background.”
Still,
for the 787, HCL gave Boeing a remarkable price – free, according to
Sam Swaro, an associate vice president who pitched HCL’s services at a
San Diego conference sponsored by Avionics International magazine in
June. He said the company took no up-front payments on the 787 and only
started collecting payments based on sales years later, an “innovative
business model” he offered to extend to others in the industry.
The
787 entered service three years late and billions of dollars over
budget in 2011, in part because of confusion introduced by the
outsourcing strategy. Under Dennis Muilenburg, a longtime Boeing
engineer who became chief executive in 2015, the company has said that
it planned to bring more work back in-house for its newest planes.
Engineer Backwater
The
Max became Boeing’s top seller soon after it was offered in 2011. But
for ambitious engineers, it was something of a “backwater,” said Peter
Lemme, who designed the 767’s automated flight controls and is now a
consultant. The Max was an update of a 50-year-old design, and the
changes needed to be limited enough that Boeing could produce the new
planes like cookie cutters, with few changes for either the assembly
line or airlines. “As an engineer that’s not the greatest job,” he said.
Rockwell
Collins, now a unit of United Technologies Corp., won the Max contract
for cockpit displays, and it has relied in part on HCL engineers in
India, Iowa and the Seattle area. A United Technologies spokeswoman
didn’t respond to a request for comment.
Contract
engineers from Cyient helped test flight test equipment. Charles
LoveJoy, a former flight-test instrumentation design engineer at the
company, said engineers in the U.S. would review drawings done overnight
in India every morning at 7:30 a.m. “We did have our challenges with
the India team,” he said. “They met the requirements, per se, but you
could do it better.”
Multiple
investigations – including a Justice Department criminal probe – are
trying to unravel how and when critical decisions were made about the
Max’s software. During the crashes of Lion Air and Ethiopian Airlines
planes that killed 346 people, investigators suspect, the MCAS system
pushed the planes into uncontrollable dives because of bad data from a
single sensor.
That
design violated basic principles of redundancy for generations of
Boeing engineers, and the company apparently never tested to see how the
software would respond, Lemme said. “It was a stunning fail,” he said.
“A lot of people should have thought of this problem – not one person –
and asked about it.”
Boeing
also has disclosed that it learned soon after Max deliveries began in
2017 that a warning light that might have alerted crews to the issue
with the sensor wasn’t installed correctly in the flight-display
software. A Boeing statement in May, explaining why the company didn’t
inform regulators at the time, said engineers had determined it wasn’t a
safety issue.
“Senior company leadership,” the statement added, “was not involved in the review.”
No comments:
Post a Comment