Some programmer - or team of programmers - implemented this software. Sure, it may have been a PHB who came up with the idea and told them to code this into the algorithm - but it was programmers like you or I who actually cranked out the code that made this possible.
Can you imagine this happening with, say, an architect? When unsafe buildings fall down, it makes international news. By and large, professionals have a code of conduct which they must follow or there is a very real chance that they will lose their livelihood.
I'm a Member of the British Computing Society - it has a fairly simple code of conduct for members. http://www.bcs.org/category/6030
I can certainly see adding this functionality would probably be a breach of ...
> 1 a) have due regard for public health, privacy, security and wellbeing of others and the environment.
> 2 d) ensure that you have the knowledge and understanding of Legislation* and that you comply with such Legislation, in carrying out your professional responsibilities.
> 2 f) avoid injuring others, their property, reputation, or employment by false or malicious or negligent action or inaction.
> 3 e) NOT misrepresent or withhold information on the performance of products, systems or services (unless lawfully bound by a duty of confidentiality not to disclose such information), or take advantage of the lack of relevant knowledge or inexperience of others.
But, here's the kicker - if I were kicked out of the BCS for adding this code, nothing would happen to me. Employers don't care about professional bodies - except in terms of certification and, possibly, indemnity.
I'm quite happy being a member of a Trade Union, because I believe it offers me the best protection against a malicious employer - I wonder how long before more codes start joining professional bodies to help protect themselves from being asked to act counter to their best interests?
I'm a member of the ACM (Association of Computing Machinery) and every year I read over the IEEE-ACM Software Engineering Code Of Ethics: https://www.acm.org/about/se-code
Due to manager and business pressure to make more money, multiple sections are subverted and there's absolutely no consequence to them. The only consequence they understand is if programmers start to refuse jobs or tasks or quit based on violations of the code.
This is sometimes why I would favour certification or more responsibility placed on the programmer. Some programs should not be written or if they are written they should be held to a very high standard. We wouldn't have issues with high frequency trading if it cost a lot to provide insurance for that kind of development. Sure simple websites don't really need high standards, but as soon as you're past 1 million in revenue you're dealing with real money, real data and any bugs could have serious consequences.
On the other hand, I like being able to write little scripts that say "hey if something breaks, it ain't my fault".
In the same vein is why I advocate for "Software Engineering" being brought into the realm of other engineers and having the status protected so people who call themselves engineers are held to the ethics and governing regulations that all engineers are.
Holding the P.Eng. designation myself I would be liable to lose my license had I published this software, I really hope we can reach a point in our profession where a higher standard of care to public and environnment is not only expected, but enforced.
I see this a lot. And I generally agree. However...
For software it's hard to see where to draw the line. For other engineering disciplines this can be pretty easy. I can release a little mechanical toy and not be an engineer. The public won't try to use it to drive around town.
But if I make a pet software project millions of people might decide to use it to "drive around town", and some portion will hurt themselves.
I think that if you build stuff for certain things (such as 1 ton objects zooming around at 70+ miles per hour), it's obvious. But what about financial software? What about social networks (privacy issues)?
When I first started programming I built a D&D utility for a friend. He passed it to his friends. Soon around 10 of them used it. If it had a serious bug, should I have been prosecuted? Should I have been unable to build it for him in the first place?
If you say all of this activity should be restricted, how do you get from here to there? Some of the best "software engineers" in the world aren't technically software engineers.
Software you write for fun or to pass around to friends would (should) have the standard disclaimer of implied warranty of suitability.
Software developed to control an automotive emissions system by a manufacturer and in the context of being used for that purpose would not be able to escape that.
This is a very common counter argument to software engineering as a discipline.
> But if I make a pet software project millions of people might decide to use it to "drive around town", and some portion will hurt themselves.
That is a legitimately unique aspect of software and deserves consideration.
> I think that if you build stuff for certain things (such as 1 ton objects zooming around at 70+ miles per hour), it's obvious. But what about financial software? What about social networks (privacy issues)?
Likely some portions of the system matter and some don't. Which ones likely require engineering judgement. For physical products it's pretty easy to make that call for a paper weight and for an airplane, but all the products in between need that decision. We are generally building software systems that are about as complex as a Black & Decker power drill. When designing something like that you need to make damn well sure you are not electrocuting the user, but if the gears break and it doesn't function you aren't hurting anyone. Therefore most drills are 'double insulated', but there is no backup for the actual function. In software we should make that same call. What is the consequence of failure? If it's small then we live with it. If it's not, then we should build a fail safe / backup system.
I'm super in favor of software engineering licensing and all that it entails, but I do think large swathes of software are completely immune because failure has no consequences. Your D&D utility is an example. Most open source licenses rightly have a clause that should protect them from civil lawsuits, much less prosecution.
> Likely some portions of the system matter and some don't. ....... but I do think large swathes of software are completely immune because failure has no consequences
This is where software seems unique - this seems false to me. A common failure mode of software is code execution, which has nearly unlimited consequences.
E.g. my D&D utility could read and write files. But it might have a bug in the file reading code (Edit: or anywhere in the code), which someone could exploit with a properly crafted file to gain access to the user's computer. But you said this utility was immune to failure. It seems clear to me that it isn't.
There is no physical object analogy here that makes sense to me. It would be like - if you used your black and decker drill on a piece of wood someone gave you, it could electrocute you if there were a flaw in the drill bit. Or it could turn off your refrigerator. Or it could play a song to you. This makes no sense, but this is the world software lives in.
Edit: or do you mean no physical consequences, so only software controlling physical devices require the proper licensing?
I was thinking more physical consequences. There's no way that a program running on my desktop can hurt me other than privacy / security / availability of my data. But I'm not going to the hospital because your D&D utility was written in a weekend and crashes sometimes.
But you do raise a point about the real costs of mistakes even in trivial systems. Arguably that makes the point that operating systems should have stronger sandboxes between apps. An operating system is effectively infrastructure, you probably want higher quality standards there.
I tend to think that our society should move in the other direction. I see the need for college degrees and certifications as a form of caste system which limits upward mobility and increases cost of living. I would like to see education and certification added to the list of qualities on which you're not allowed to discriminate (like race/creed/etc already are).
I think the VW case can be partially solved by handing out indictments to everyone involved, CEO down to engineer. Working in a corporation does not excuse criminal behavior.
Similarly, I don't think making Financial Engineer a thing would do anything to head off the next financial collapse and reign in excessive risk taking on the part of the banks.
Education only resembles a caste system because its availability is heritable. The quality of public education depends on the income characteristics of the geographical location you're born into; the quality of higher education depends on the disposable income of your parents (except at the most elite schools with full need-based scholarships for middle-class students).
I'd rather see us abandon the bizarre notion that education is in any way a parental or local responsibility. It should be federally controlled with uniform quality for every student and differentiations that make sense (on natural talent, effort, and specializations chosen) rather than those based on parentage.
Good education gives you a better understanding of what you're doing and the context of the world, people, and systems you're doing it to. Of course it makes a difference, and it should be available to everyone.
The difference between someone with a college degree and without is the work they put in. That seems like a pretty reasonable thing to base a hiring decision on. Your suggestion is like saying 'you shouldn't be able to discriminate based on work experience'. The difference is college is 4 years of reading books and taking tests vs 4 years of writing code.
You can argue which one is more valuable / relevant / useful, but they're just different kinds of experience. Not allowing that in a hiring decision is flat out unworkable.
Scott Alexander (of slatestarcodex) once wrote about medical ethics[1]:
> Listen up, this will be on the test. You need to help patients, not hurt them. You need to be responsible. Try to avoid being irresponsible. Try to act professionally, and to be a positive asset to society. Don't abuse nurses. If someone asks you do to something unethical, tell them you won't. Did I mention you need to help patients, and not hurt them?
> If reading that paragraph seemed like a waste of your time, imagine how I feel after attending the three hour lecture version.
Engineering ethics would be a similar waste of time. Good people don't need courses to know that cheating emissions tests is wrong. Bad people won't be swayed by a three hour lecture.
Here's an idea that would be more effective at reducing future harmful software: Run git/svn/hg blame to see who wrote the emissions-dodging code. Then arrest those people and charge them with crimes.
I would love to teach a software ethics course, and it wouldn't be me standing up saying "be good" over and over. We would study contemporary ethics questions:
- what should you do if your government demands you secretly backdoor a specific user?
- what if your easiest recruiting channels yield a higher proportion of white males than other, costlier recruiting channels do?
- if your startup is struggling to even survive, should you invest time in infrastructure that protects users' privacy from your employees?
- how do you find nonprofits and companies that are aligned with your values and need your help?
Those are just four questions off the top of my head. I think there are plenty of thugs to talk about to fill a semester, let alone an hour.
> - what should you do if your government demands you secretly backdoor a specific user?
They've got a warrant or a court order? OK.
> - what if your easiest recruiting channels yield a higher proportion of white males than other, costlier recruiting channels do?
If you're a good person, you try to correct for that. If you're a money-grubbing sociopath, you try to correct for that. This may sound unusual, but think about it from the sociopath's point of view: If other companies are neglecting some potential candidates, you can hire those people for less pay. Recruiting is cheap compared to salaries, so it's worth the upfront cost. In the long run, even if everyone is a money-grubbing sociopath, all demographics end up being paid the same.
> - if your startup is struggling to even survive, should you invest time in infrastructure that protects users' privacy from your employees?
Probably not. A dead company can't defend users' privacy very well. Also, assets (such as user data) tend to get sold off when a company shuts down. Best is the enemy of good, and all that.
> - how do you find nonprofits and companies that are aligned with your values and need your help?
Givewell.[1]
These are not hard questions to answer. More importantly: Your hypothetical course, even if taken by every software engineer, would not have prevented the Volkswagen emissions cheating. That was obviously wrong, but the people involved didn't care.
If you already have enough ethics, you can work out on your own what would be ethical for any given situation. You don't need the class. It may save you a bit of time in reflective contemplation, but it's not going to give you anything you didn't already have.
If you don't have enough ethics to figure it out yourself, the class is just teaching you how to pretend to have ethics. And like jazz, you will fail just as soon as you are called upon to improvise.
To me, there is an ethical "operating system" which is what you are describing, very low level principles that you apply.
But your ability to be ethical in a specific situation depends on your database of "ethical considerations" about possible decisions you might make that will have serious ethical ramifications and how those things are wired up. The purpose of a course is to build up that database.
You might have a deep and earnest desire to be kind to someone, but without a cultural framework for understanding their feelings, for example, you could be completely unable to do that.
That's sounds like an argument against "The problem here is people not knowing what's ethical." And accomplishes it, because the problem pretty clearly isn't that.
But the point still stands, that there should be some coordination among programmers to ensure that e.g. when someone is asked to do this, they have a group to fall back on, who's willing to help him/her out and boycott this employer.
If you don't have the spine to say "no, that's wrong, I won't do it" as a software engineer then I'm not sure a group is going to help you out.
Software engineers who are any good at all can find work easily. They are among the last people who should feel held hostage by a job. If your personal ethics are not enough, I don't see how a group will help.
Sure, if you're a teenager or 23 year old who shares an apartment and has no dependents and no debt.
Try that attitude on for size if you have a family that depends on your income, or you live in a city that isn't a tech hotspot, or you or your spouse or children have medical issues and depend on your medical insurance. You can't pay the rent or mortgage with ethics. There's lots of towns even in America where there are only a handful of high tech employers so no, you don't have alternatives. Unless you consider moving, which can easily cost $15k in moving costs and tens to hundreds of thousands of dollars in housing costs to be free. Not to mention the disruption to friends, family, and social networks caused by moving. Or care arrangements -- grandparents babysitting is close to free; daycare can easily be $400/week/child (or $600/week/child for under 2 y/o) in sf.
Yes (as would you if you bothered to look), and plenty more offer relocation bonuses of eg $5k which (1) will be taxed, and (2) are wildly insufficient unless you're basically living on a futon. For a family that needs a trip to house hunt and has furniture for 3-4 people, an actual relocation can very easily cost $15k ($3k airfare/hotels/meals/babysitting for 3-4 days in the new city to house or apartment hunt; $10k move for 3 bedroom house + car; apartment lease termination / house selling costs; other services shutdown plus startup in the new city).
> Sure, it may have been a PHB who came up with the idea and told them to code this into the algorithm - but it was programmers like you or I who actually cranked out the code that made this possible.
So i think there's been a lot of assumption that the software engineers made this code explicitly for production. For all we know they were making what they thought was code for specific in-house experiments and a subset or different set knowingly kept it for production.
That's possible, but I find it to be a stretch. Unless one group were told "write some code to meet this emission standard" and another were told to "write some code to detect when emmission testing is happening" that could be the case. But somewhere along the line someone was given the task to combine those and "disable the emission controls unless testing is happening."
In other scenarios, yeah it's perfectly possible for an developer to complete a task that seems innocent enough and unbeknownst to him will later become a part of something nefarious.
Actually that was the purpose stated; they developed it for testing in-house and it makes sense to do so. However, at some point there must have been someone who was a front-line employee who could have blown the whistle on this kind of disgusting ethics violation.
The engine control software in question was produced by Bosch. They say they warned Volkswagen that the feature in question was for development only and not to be used in production.
It seems unlikely that Bosch would implemented an algorithm to detect an emissions test and alter the engine behavior. It seems much more likely that Bosch implemented diagnostic features to force the engine into a particular operating mode (e.g. Enable/Disable EGR) and that Volkswagen activated those features when it detected certain qualifying criteria (i.e. emissions test conditions).
Unless there's direct evidence implicating Bosch, then blaming them for the cheating is like blaming a bullet manufacturer when you get caught shooting people.
Was it a stern warning, loaded with gravitas, or was it accompanied with a wink that passed unnoticed by the corporate attorneys?
Bosch might have sold them the pistol, but VW is still the one that shot itself in the foot with it. But was the pistol pre-loaded with toe-seeking bullets, or did VW carefully aim at its own extremity before firing?
There is still some question in my mind as to how complete a defeat device package Bosch shipped. If there were several separate functions like LowestEmissions() and LowestFuelConsumption() and LowestMaintenance() and BestPerformance() and LowAmbientTemp() and HighAmbientTemp(), that's quite a bit different from DetectEmissionsTest() or OperateAsDefeatDevice(). Who was it that wrote the code to switch modes when an emissions test was detected?
From what I understand of the situation, It's entirely likely that the devs got a rule table from the engine and emissions guys and couldn't actually tell if the metrics were good or sketchy as hell.
It sounds like the main problem is with high fidelity O2 sensors at a cheap enough price point that these lookup tables can be banished. We legislated people carrying a block of platinum around in their car. Maybe $200 O2 sensors is the next step if the manufacturers can't be trusted.
There's a whole business structure that drove and enabled this decision, it wasn't just a bunch of engineers who had a bright idea one day. (Not meant to be exculpatory for the people who worked on this.) But you never hear anyone going around calling for businesspeople to conduct themselves ethically.
Well said! I did think about the programmer or programmers when this story first broke. I am not shocked that some executive would come up with the idea...but I am shocked that someone would actually code it. You have to wonder how many people knew and said nothing.
I could imagine a scenario where this started out as something less evil. Seems to me that cars will perform differently in a lab than on the road; off the top of my head:
- Stationary cars have different air-intake characteristics
- The temperature of the fuel and air is likely different
- The humidity of the air might be different
- The fuel in the tank is stationary not sloshing around
There may well be valid reasons for the software to vary engine parameters test mode; maybe to compensate for the above.
Then sometime later someone said "This NOx Reduction module is killing fuel efficiency on the open road, how about we turn it off"
The net result is the same but motives and intentions of the individuals are important.
What steps do people take to comply with #2d ? Does the British Computing Society maintain a service which keeps people up to date with legislation and relevant court rulings and where you can ask hypothetical questions of actual lawyers?
I do receive emails from them talking about policy changes - and the BCS's position on government consultations.
They also have a number of reports & white papers which members can read.
Members also have access to a legal helpline. Don't think I can link to it, but it says...
> As part of the ongoing commitment from BCS to invest in services and benefits, the BCS has launched a new and improved Legal Advisory Service comprising two new online legal information websites and a dedicated telephone helpline for both Business and Personal legal issues (restrictions apply*). These services are totally FREE and are provided by Law Express, the UK's leading specialist provider of legal advice.
I've not had cause to use it, so I can't comment on how well it works in practice.
This one does seem quite onerous: how do you even know what legislation applies to the software you're working on, to even ask the right questions? There are already specialists in legislation (we call them lawyers), so why should the programmers take on this responsibility?
I agree with the spirit of the rule, but I don't think it makes sense as it's basically asking all programmers to become lawyers. It should be more along the lines of "comply with legislation you are (made) aware of" and require some due diligence like "solicit the services of a lawyer if there is a reasonable belief there is legislation that could apply".
That's part of your professional responsibility. How do you know not to store passwords in plain-text? How do you know that you have to make your software accessible to people with visual impairments?
I honestly don't think you need a legal degree to know that "dramatically alter how the pollution controls works when in a test environment" is likely to be extremely dodgy!
Now, maybe these programmers did flag this as a problem. And, maybe VW's high-priced lawyers said it was OK. In which case, responsibility is probably shifted.
I'm not saying every programmer should be a lawyer. But you can't say "how was I supposed to know that running a red light while drunk is illegal?! I'm just a delivery driver - not a lawyer."
Here's the far more interesting question about this situation: What if nobody at Volkswagen really even knew this was happening? In other words: systems are complex, and it is not beyond the realm of possibility that this cheating* could have arisen purely as an emergent property of a set of otherwise innocuous changes, and then stuck around through something like environmental fitness, as it was functionally useful for the organization as a whole.
If the Volkswagen case seems too clear cut for that, then think about PageRank, or Facebook's software that decides what to show to whom. Are we all so sure that every engineer who works with this code really knows, ex ante, all of the effects their changes might have down the line?
There are a lot of demands for criminal liability in this thread, but I'd suggest we proceed carefully. While it certainly looks suspicious, there are a lot of ways that weird behavior can creep into software that don't involve malicious intent.
Look at it this way: could Volkswagen engineers conceivably have written code that caused the system to fail all emissions tests all the time? (Yes.) Could they have done so without realizing that they'd made an error? (Sure.) Would we all assume that that bad code was deliberately introduced? I'm not so sure.
* Or maybe some other, hypothetical cheating. Sure, in this particular case, maybe a particular software engineer or set of engineers knew exactly what they were doing when they wrote the code that enabled the cheating. But the thing I'm interested in is what happens when that's not the case, and how close that day is.
This probably isn't what happened, but I think in this context we need to remember that it is perfectly benign and normal to have software in the ECU for detecting an emissions or motor test.
Modern cars have a lot of sensors and environmental inputs. A motor or emissions test then is quite abnormal in that some sensors will report standstill while you are going full throttle. The ECU needs to make sure it doesn't detect this as an abnormal system state and brake, shut the engine or otherwise perform corrective action that could put the test system or personnel in danger.
> Voting machines could appear to work perfectly -- except during the first Tuesday of November, when it undetectably switches a few percent of votes from one party's candidates to another's.
Apart from difficulties having manufacturers publish their code I think another problem will be proving that some (open source) software is actually loaded in some piece of hardware and nothing else or extra. Most products on the market can be taken, teared down and tested for as long as needed. With voting machines the time to proof some software was active during elections and nothing else is the time you have before you officially publish the results. This is an extremely short window and impossible for a large part of the public to verify.
Forcing VW to release source code would not have prevented this. They could have simply released a different set code from what is actually running on the cars.
This is an extremely hard problem to work around. They could let you dump the binaries of the software running on an individual car, and then you could compile the source code and compare the resulting binaries, but how do you know the car isn't feeding you a fake binary dump? It seems like a catch-22: I can't think of any way around this problem short of tearing the car apart, cutting all the chips open, and physically verifying them under electron microscopes.
On the other hand, if emissions testing would actually test what's coming out of the tailpipe under normal driving conditions, then that would seem pretty foolproof.
I agree that actually measuring the real emission is a good idea, but there is a way of checking that the binary and the source does not match, at least in theory.
The way is to require reproducible builds. If the source is built with a specific compiler with known source and known parameters, and the source of the system in question is available, then running the compiler will produce the exact same binary every time if the compiler is sane. If all of this is published, then it is possible to check that the binary in the cars firmware and the binary produced by the published source match bit by bit. This isn't just some sort of academic exercise, but something that have been demonstrated to work in practice. In fact debian try to make this work for the whole distribution. This may not be totally practical yet, but this kind of checking can be done if the vendors are required to make reproducible builds and the regulators bother to check. All of this are big ifs, but at least in theory it is possible.
is there a way to guarantee the binary we get from the control system in the vehicle is the binary that is running on that vehicle? That was the concern NickM had for this style of verification.
Yes, exactly. Reproducible builds can catch accidental differences, but discrepancies introduced by a malicious actor can still easily be hidden if they have complete control over the software and hardware.
You're assuming the car doesn't also cheat on the "make install" test. How do we know the car isn't programmed to secretly leave the cheating software intact?
Whatever extra layers of software verification we attempt to throw at the problem can be defeated by extra layers of deception from the manufacturer.
What if increasing government requirements are just not achievable? The laws are written by politicians, not engineers. If car companies are being asked to meet unreasonable performance metrics, what other choice do they have?
Cars have gotten a lot more efficient, I can get a 1.5L engine that puts out over 150hp. A 2L 4 cylinder engine can put out over 200hp!
These are huge savings compared to what used to just a decade ago, but they are still apparently not good enough to meet government requirements.
I understand that part of VW's cheating was a cost saving measure, but with all the talk about every car manufacturer doing it, one has to one, if 10 different people all independently come up with the same solution, maybe there is a problem?
All of the regulations CAN be met... but it is increasingly difficult to meet emissions, MPG, and Safety regulations, AND satisfy customers.
Safety equipment increases weight, which lowers MPG.
More efficient engines are generally smaller and less powerful, which does not satisfy customers.
In the VW case, reducing NO2 emissions lowered MPG due to the burn-off process.
One change to the fuel economy law that is sorely needed is changing the standard from MPG to Gallons per Hundred Miles. Differences between GPHM measurements are easier to understand [1].
> More efficient engines are generally smaller and less powerful, which does not satisfy customers.
They are also less safe for freeway driving.
On a recent vacation, I recently visited Albuquerque NM, driving around there requires a fast powerful engine to get up to speed ASAP, or accidents will happen.
In comparison, when I visited Boston, I rented a Fiat 500 and loved it, it was perfect for zipping around a those old narrow winding city roads, in and out of traffic, and finding parking. Heck I didn't pay for parking in Boston once while I was there!
Freeway driving is the norm for most of America though. A tiny engine won't due, it will however get someone ran over.
Don't there already exist several brands and models of cars that meet the standards these companies say they can't possibly meet? (I mean the Chevy Volt, Toyota Prius, and Tesla models probably meet emissions standards, right?) It seems more likely that all these companies found it more cost effective to cheat than to innovate than that it isn't possible to innovate.
The problem with making it look like an accident is that, to really do that, the testing mode would sometimes have to be "accidentally" enabled while really driving. Which to a customer would just seem like the car occasionally and unpredictably performs very poorly, which is not exactly going to inspire confidence in the brand.
And if the testing mode never comes on during normal driving, well, that's not going to look very much like an accident, is it.
Worth knowing that this attitude is a bit of an about-face for Schneier (albeit one that happened many years ago). For instance, this is what Schneier had to say about full disclosure software security in the early 90s:
That was the first hit I found for what I remembered as a strain of Schneier comments about researchers that pissed me off (I'm friendly with some of the old eEye crew, is part of it). Here's a clearer-cut one:
We shouldn't lose sight of who is really to blame for this problem. It's not the system administrators who didn't install the patch in time, or the firewall and IDS vendors whose products didn't catch the problem. It's the authors of the worm and its variants, eEye for publicizing the vulnerability, and especially Microsoft for selling a product with this security problem. You can argue that eEye did the right thing by publicizing this vulnerability, but I personally am getting a little tired of them adding weapons to hackers' arsenals. I support full disclosure and believe that it has done a lot to improve security, but eEye is going too far. As for Microsoft, you can argue that the marketplace won't pay for secure and reliable software, but the fact remains that this is a software problem. If software companies were held liable for systematic problems in its products, just like other industries (remember Firestone tires), we'd see a whole lot less of this kind of thing.
Full disclosure of found software vulnerabilities and mandating that companies provide access to source code are two entirely different things.
One component of Schneier's recommendation is protecting researchers who practice full disclosure ("...attempt to muzzle security researchers who find problems"), but that's not really the core of his message in this post.
"Both transparency and oversight are being threatened in the software world. Companies routinely fight making their code public and attempt to muzzle security researchers who find problems, citing the proprietary nature of the software. It's a fair complaint, but the public interests of accuracy and safety need to trump business interests."
No one, I hope, thinks it's that simple. Businesses cannot be expected to put all their source code on github. Instead, this needs to follow the route that all other regulation goes. Instituting private but 3rd party review of the source code and testing, for which the manufacturer needs to pay a fee to support. I'm not saying this will catch 100% of the issues, but it's a lot better then what we have now and much more likely to work for businesses.
That's not obvious at all. Cars can make all of their control software open source with little impact on profitability. Who buys a car based on the software? Isn't the sole reason its kept secret, is to hid how incompetently its been written?
Electrical devices have UL certification. Underwriter Labs is an independent, private company (nonprofit until very recently) that evaluates products and provides a seal of approval.
This model could be applied to software as well. An independent organization like UL could evaluate the source code, talk to the engineers, and grant (or deny) certification. The source code would still be under NDA or appropriate privacy, but consumers and governments would have more to go on than just some company's word that they're not making anything dangerous or illegal.
I wonder if this can already be done without waiting on government oversight. Essentially outsourced code review with liability protection up to a certain amount. All too often I see code reviews/projects that occur in an internal, self-reinforcing echo chamber. Even a tier of outsourced code review without liability protection but instead an accompanying seal of approval would make me feel better as an executive.
this practice (of determining that the software is being tested, and thus altering its behaviour in favourable ways) has been present, off and on, in the anti-virus industry, for years.
> But transparency doesn't magically reduce cheating or improve software quality, as anyone who uses open-source software knows. It's only the first step. The code must be analyzed. And because software is so complicated, that analysis can't be limited to a once-every-few-years government test. We need private analysis as well.
I'm skeptical of whether VW would have been caught any sooner, or would have changed their behavior, if they were forced to release source code; "and then analyze" is far easier said than done, especially with generated code (which is common in the automobile industry). I fear that if anything, forcing VW to release source code would have simply resulted in uselessly obfuscated "generated" code.
I'm skeptical of the proposition that taxpayers should take on the cost of analyzing reams of generated code without any context or documentation.
And finally, I'm skeptical that these calls for public access to source code are politically feasible, fair, or wise. The amout of intellectual capital that's spent on ECU design is absolutely massive. I don't see anyone in the tech industry calling on Congress to force Google or Microsoft to open source core components or reveal their software to regulators, even though vulnerabilities in their software could easily ruin or end lives.
It might make more sense to mandate that comapnies provide verifiable evidence that their safety-critical or regulation-relevant systems are properly designed, with a variety of avenues to compliance.
Releasing source code to the public and paying for at least one private analysis (to be selected by government regulators) would be one way of achieving this. This would probably be the easiest option for IoT companies (e.g. run-of-the-mill smart lightbulb manufacturers) whose source code doesn't contain any particularly valuable IP. And this would also force companies to pay up when they release hopelessly obfuscated code.
But this also opens the opportunity for other paths to compliance which, if designed properly, could address the safety concerns of the public as well as the fairness/property rights concerns of private entities. For example, one alternative path for companies whose IP concerns are legitimate could be use of formal methods. The regulation/safety specifications could be open to the public for criticism, and would be far more readable than a dump of generated code. And a few regualtors could double check that a trusted formal methods tool verifies that the specifications hold for the software running on the car, at minimal cost to both the car company or the general public.
then I think they would know they were being watched and would act less horribly, at the very least. It's the same reason why you almost never see backdoors in free software. Just knowing that you could be watched makes it a lot harder to be sneaky, even if nobody is actually watching. Same principle as having placebo cameras that are actually non-functioning.
But forcing Microsoft to reveal their crown jewels (they sell software) vs car companies reveal how an internal controller works (they sell cars), is not comparable in any way. Its disingenuous to imply that.
> But forcing Microsoft to reveal their crown jewels (they sell software) vs car companies reveal how an internal controller works (they sell cars), is not comparable in any way. Its disingenuous to imply that.
This makes two assumptions that I disagree with.
The first assumption is that software isn't a core component of cars. I think this is already not true. And to the extent that it is true, it won't be in 2-5 years. Software is at the core of emerging differentiations, such as self/assisted-driving features.
The second assumption is that that software doesn't or can't reveal sensitive information about (other) core components of cars. I think you'd be surprised at how much you can deduce about a physical system from its control software.
Finally, there's no clear bright line on which companies should or should not get to protect their IP. For Microsoft it's open-and-shut, but e.g. Google doesn't sell its search engine software. And what about IoT companies? "We're a software company that sells IoT appliances, not a lightbulb/car/robot/etc. company". So the only way to write such a regulation would be to write a regulation for the auto industry -- which is insufficient for the same reasons that Schneier talked about IoT and car companies in the same breath.
There's nothing about the physical system you can't learn by buying the car, and looking. So no secrets there to give away.
The controller we've been talking about control efficient engine operation, and conformance to existing federal standards. Not auto-driving cars (yet). There are powerful reasons to force them to be open, and only 2nd-order reasons to keep them secret.
In future I'd expect auto-driving cars would absolutely be completely open. There are even stronger reasons that emission-control issues, by far. Not running over kids for example. So we're likely to see a huge move in that direction.
As for a bright line, how about: if I can breath what you emit, or get run over by your software, then it belongs in the open domain? Pretty clear to me.
You make all kind of assumptions about the sophistication behind and engine ECU. It's one of the most complex ECU's in the car.
Just as car companies may just sell cars, Dell or Apple just sell computers. Why not provide the source code which comes with a new Dell/Apple laptop? In the end, how can a car maker, using Windows/Apple SW to build the SW for the cars, can trust them to not mess up? They are part of the tool chain for making cars.
For a car maker to provide source code, they would have to get permission from a lot of people.
In this case, from Bosch. Bosch would require permission from their own OS supplier (probably Etas' RTA-OSEK/RTA-OS) and from other 3rd party suppliers. To be clear, these technologies are not directly available to the car maker. They never get complete source code from suppliers (unless some special agreement, but I doubt it). They can audit the SW/process though, which implies some level of access.
The engine ECU may not contain the new hype controls of self driving, but it contains a lot of know how, inventions, innovations. It's very nice to have a smooth running car, but it takes a lot of tech to make it run like this.
Apparently not. Look at Toyota's ECU when they finally were forced to reveal it - total crap. Disambiguating claims of company secrets, from the reality of coverups, is the whole point. You gotta point to one that has any real valuable secret before claiming they'll lose anything by it.
And no they don't have to reveal the source to the OS, that's a strawman.
You're massivley underestimating the impact that open sourcing would have because you're assuming that these software quality metrics whose utility is substantiated with correlative evidence will have any causal significance post-open-sourcing regime.
If you tell companies they have to open source, you can be sure they'll be MISRA-C compliant. That doesn't mean their code is any less shit. Things like MISRA-C are useful in large part because where they are violated, you can be sure there was sloppy engineering. Giving people incentives to game standards whose claim to significance is largely correlative only guarantees that those standards become meaningless. In order for open sourcing to be useful, you need to analyze the code against some form of specification that concisely captures what that code is supposed to be doing, not just the absence of certain low-level errors or coding patterns highly indicative of certain low-level errors.
> And no they don't have to reveal the source to the OS, that's a strawman.
Why? A major bug in the OS would almost certainly make it impossible to provide any guarantees about the behavior of the ECU. And that OS is probably used in a lot of other safety-critical settings.
Sorry, but I still completely fail to see how your position isn't simply "primarily software companies deserve IP protections / primarily car companies don't".
> Disambiguating claims of company secrets, from the reality of coverups, is the whole point.
You're massively underestimating that amount of money car companies invest in both R&D and in engineering of software (Hell, car companies definitely spend more on CS R&D than the average software company, even if you restrict that to software companies working in safety-critical industries). This flippant attitude toward million/billlion dollar software investments based upon what companies happened to write that software is not going to fly outside of tech bubbles.
The audacity of claiming to "search those who are responsible", the sheer hypocrisy of pretending that not every little requirement is fully traced in multiple tools and databases alongside the information who in the chain are the stakeholders and who has written the test specification and who has released the component shows that this group of confirmed whore mongers [0] will get away with everything here in Germany.
> Computer-security experts believe that intelligence agencies have been doing this sort of thing for years, both with the consent of the software developers and surreptitiously.
What ever happened with that thing a few years back where some in the OpenBSD community were claiming the FBI was attempting to insert a backdoor?[1][2] I was always surprised with how little media attention that seemed to get.
Can you imagine this happening with, say, an architect? When unsafe buildings fall down, it makes international news. By and large, professionals have a code of conduct which they must follow or there is a very real chance that they will lose their livelihood.
I'm a Member of the British Computing Society - it has a fairly simple code of conduct for members. http://www.bcs.org/category/6030
I can certainly see adding this functionality would probably be a breach of ...
> 1 a) have due regard for public health, privacy, security and wellbeing of others and the environment.
> 2 d) ensure that you have the knowledge and understanding of Legislation* and that you comply with such Legislation, in carrying out your professional responsibilities.
> 2 f) avoid injuring others, their property, reputation, or employment by false or malicious or negligent action or inaction.
> 3 e) NOT misrepresent or withhold information on the performance of products, systems or services (unless lawfully bound by a duty of confidentiality not to disclose such information), or take advantage of the lack of relevant knowledge or inexperience of others.
But, here's the kicker - if I were kicked out of the BCS for adding this code, nothing would happen to me. Employers don't care about professional bodies - except in terms of certification and, possibly, indemnity.
I'm quite happy being a member of a Trade Union, because I believe it offers me the best protection against a malicious employer - I wonder how long before more codes start joining professional bodies to help protect themselves from being asked to act counter to their best interests?