On March 21, 1986, a medical nightmare unfolded in the city of Tyler, Texas. A male cancer patient received radiotherapy treatment for a small tumour on his back from a new type of radiotherapy machine, the Therac-25.
The patient had been prescribed a small dose of 180 rads. When it was turned on, the Therac-25 began buzzing loudly, the patient felt a horrific flash of pain and the skin on his back began to sizzle.
He had received a massive overdose of radiation. Instead of 180 rads, he had received tens of thousands of rads. He was immediately given treatment for radiation sickness, but it was too late; the patient passed away five months later.
The Therac-25 incident is sometimes used as a cautionary tale of how poorly designed computerised medical devices can cause catastrophic consequences. This case prompted the development of tighter controls over medical devices.
But how did things go wrong, as we currently witness the emergence of an AI revolution in medicine? Does the Therac-25 tragedy have anything to teach us about the hidden dangers of medical AI?
What was the Therac-25?
The Therac-25 was a radiotherapy system consisting of a bed and a beam emitter, which was placed over the site of a tumour. It was manufactured in a partnership between Atomic Energy of Canada Limited (which, these days, is involved in managing radioactive waste and decommissioning) and French engineering company, CGR.
The machine could be set to emit either a beam of electrons, for shallow tumours, or a more penetrating beam of gamma rays, for deeper tumours. Crucially, for the electron beam, magnets would be placed between the emitter and the patient, to spread out the beam over the body. For the gamma-ray beam, a metal plate would be placed between the emitter and the body to convert the electrons into more gamma rays.
Where did the Therac-25 go wrong?
Central to the story of Therac-25 is software bugs. Importantly, the Therac-25 was designed to be solely controlled by software. This was different from earlier versions of the machine (the Therac-6 and the Therac-20).
Despite the differences between the earlier machines and the Therac-25, some of the software developed for the earlier versions had been migrated to the Therac-25. These bugs were not so important for the earlier versions because they had built hardware safety controls. But with the Therac-25 they could be catastrophic.
The bugs meant that, for example, if the radiographer changed the beam type from x-ray to electron beam within eight seconds, the machine would give the dose but display a message that the dose had not been given. This could cause the radiographer to give repeated doses in succession while wondering why the machine was not giving doses at all.
The Therac-25 as a ‘black box’ problem
As ever, the truth is rarely plain and never simple. There were several factors leading to the Therac-25 tragedy; it was not just about software bugs. These factors included inadequate testing of the machine prior to release and the unwillingness of AECL to accept the possibility that the machine could have faults. Fundamentally though, this case can be also seen as a ‘black box’ type problem.
A ‘black box’ is a system that allows you to see the input and output but gives you no idea about the processes at work in between. Its use is a matter of faith for the operator. The early versions of the Therac were not black boxes. The software was merely a convenience, and the machine was under the control of technicians and radiographers. There were hardware safety controls in place that were easy to understand.
For example, there was a fuse that blew if someone tried to activate a dangerous mode of operation, such as a high energy beam without the magnets or the metal plate in place. The technicians and radiographers were aware if the machine had been configured wrongly because they were responsible for setting it up.
But with the Therac-25, these hardware safety measures had been removed. The machine was essentially controlled by the software, which was totally inaccessible (and indeed was never released). This new version had a quicker setup, allowing more treatments to be done. But the working principles of the machine were now invisible to operators and lurking dangers were impossible to spot.
It should be added that this black box problem was also deepened by AECL’s assurances that overdoses were impossible, and by the system’s error messages, which were hard to understand.
For the full story, look out for the July-August edition of Omnia Health Magazine.
Felix Beacher is the Head of Omdia’s Healthcare Technology team. He is currently designing Omdia’s forthcoming intelligence service on medical AI.
