Safety-Critical Software Reliability
Commenters debate the high standards and liability required for software in life-critical systems like medical devices, aviation, and nuclear controls where bugs can kill people, contrasting it with lax practices in non-critical software and citing examples like Therac-25.
Activity Over Time
Top Contributors
Keywords
Sample Comments
Welcome to the real world, where software can get people killed.
We trust software to fly our planes and run our nuclear power stations... So what's the problem?
I think the reason why first two examples can say that they are safe is that in case of failure, people die. And thus there was a regulation introduced. People generally care when they can die.In case of software, rather rarely people die and when software is being done on something human critical (a friend of mine works on pacemakers), there a great deal of care being done.So basically it's a wide range from DIY using a cheap wrench to person rotating a torque calibrated wrench on a
Software is eating the world, right? There's already a lot of code that could kill people. Or work on bridges.
I think there's a software engineering bias. Unless you work on critical systems you can generally "move fast and break things." But in the rest of the engineering world this kills people and is often illegal (e.g. aircraft). You're legally liable.We can even notice this with the difference in software on more critical systems vs non-critical. Compute systems on cars are not the same compute systems in your computer, and are way under powered in comparison. These systems d
Kudos, but is there a flip side of the coin, where the software you write can cost someone life (if you screw something up/introduce a bug)?
"It's not like I write software for a nuclear reactor, making a mistake will not kill anybody".
Software is built like crap because when it fails it doesn't kill anyone. Except the software that does kill people when it fails, and that's engineered properly -- not perfectly, but in line with other things like airliners and elevators.
It's not about complexity, sophistication or quality - it's about liability when shit hits the fan and your bugs kill someone.
With that "works 90% of time" idea, please don't ever involve yourself in software for anything serious: air traffic control, self-driving cars, autopilots, nuclear reactor control, insulin pumps, defibrillators, pacemakers, spacecraft attitude control, automated train control, the network stack of a popular OS, a mainstream web browser, a Bitcoin client, the trading software of a major exchange, ICANN's database, certificate signing, ICBM early warning system, cancer irradia