A massive recall of Volkswagen’s diesel vehicles will begin in January, following the EPA’s accusations that the automaker used illegal software to cheat emissions tests. Volkswagen could face criminal prosecution and $18 billion in fines for deceiving regulators. VW’s CEO Matthias Müller, who has taken over after Martin Winterkorn resigned in late September, says that “very few employees were involved” in the scandal. Results of an internal investigation are expected to be released in several weeks.
Although it might only be caused by a few employees, the problem is widespread. The emissions issue affects as many as 11 million vehicles. Fixing the defect won’t be an easy task; the engine was used in several models with country-specific variations, so thousands of solutions will be needed. The good news is that most cars will be fixed with a relatively painless software, although others will need new fuel injection equipment and catalytic converters.
On Thursday, Michael Horn, CEO of Volkswagen of America, will testify before a congressional hearing about whether VW intentionally installed software that made diesel vehicles appear “cleaner” during emissions tests than they are in real life driving situations. No matter what the penalties, VW executives are preparing for the company to take a big financial hit. Müller spoke to 20,000 factory workers at the company’s main facility in Wolfsburg on Tuesday, warning them that all planned investments are under review. While VW was planning to invest billions of dollars in new factories as well as electric and connected vehicle technologies, Müller now says that “what isn’t essential will be scrapped or postponed.”
Source: Wall Street Journal |Photo by xddorox 
“Although it might only be caused by a few employees…”
Let’s stop to think how few it could have been.
It only takes one to make the code; as I understand it, it’s a pretty simple if-then statement. But there are probably dozens of embedded firmware engineers who work on aspects of that codebase and any one of them might have seen that line; we have to assume that they are competent enough to realize what it was doing… or, if they weren’t, we have to question their competence to be working on code that controls things like an automotive engine, breaks, steering, that sort of thing.
Then, we have to assume that this code underwent some sort of internal code review. I’d like to think that all code that’s controlling things like automotive engines, breaks, steering, etc. undergoes some sort of review, at least an internal review. Either this code was not reviewed, or the reviewers were either incompetent or were “in on it.” None of these three options, no review, an incompetent review, or wide-spread participation in the fraud, looks good for VW.
Finally, we have to assume — well, hope at this point anyway — that the code and the product underwent some internal “black box” SQA (software quality assurance) testing. For this sort of test, the quality assurance engineers are given a specification of what the code is supposed to do… and not do. They test the product’s functionally against the specification without looking at the code. This time, we have five options: A) there was no SQA inspection, B) the inspectors missed this obvious deviation in behavior, C) this behavior was part of the written specification for the product, D) the SQA inspection found the problem and reported it, but management chose to ignore that report, or E) the SQA inspectors were “in on it.” None of these options looks good for VW either.
So, back to our original question: how many was this “few” employees? At minimum, it was the original author, all of the other soft/firmware engineers with access to that codebase (probably at least a dozen), the code reviewers (one would like to think at least five), and the SQA engineers (again, one would like to think eight or ten). We’re talking about 20-25 people here… minimum. That’s not a “few” rogue employees.
But here’s another angle: VW’s diesel engines got mileage and performance that was well-above their competitors while still apparently passing the same emissions tests that frustrated the competitors’ performance and mileage. And VW did this without employing some of the complex, balky, and expensive emissions control hardware that weighted down the competitors’ cars… and their bills-of-materials… and their warranty cost. We have to assume that some of the competitors bought some of these VW cars and A) tested them very carefully in their own labs, and B) disassembled them and studied them to discover VW’s secrets.
Certainly competent — much less expert — technicians and engineers in a fully-equipped lab (and all of the big auto manufacturers have such labs and employ experts to staff them) would have discovered, as a university lab did, that these cars were behaving differently when in “test” mode vs. actual driving mode. It wouldn’t have taken much snooping to figure out that the engine control computer was behaving differently under test conditions than in real driving. But none of the other auto manufacturers said anything. It was a university lab that “dropped the dime” on VW.
It wouldn’t surprise me if, deep in the development offices at the other major diesel car manufacturers, the firmware engineers are scrambling to back similar “zaps” out of next year’s code.