When the poison is the antidote: risky disaster research
A recent report by Lipsitch and Galvani warns that some virus experiments risk unleashing global pandemic. In particular, there are the controversial “gain of function” experiments seeking to test how likely bird flu is to go from a form that cannot be transmitted between humans to a form that can – by trying to create such a form. But one can also consider geoengineering experiments: while current experiments are very small-scale and might at most have local effects, any serious attempt to test climate engineering will have to influence the climate measurably, worldwide. When is it acceptable to do research that threatens to cause the disaster it seeks to limit?
The report notes that although the virus experiments occur in biosafety level 3 or 3+ facilities, accidents happen at a worrying rate: roughly 2 per 1,000 laboratory-years. This means that a research program running for a decade at 10 laboratories has nearly 20% risk of resulting in at least one laboratory-acquired infection. The chance of this event leading to an outbreak has been estimated to at least 10% for influenza.
The number of fatalities of an outbreak would be a skew-distributed random number: most likely small, but with a heavy tail potentially running into tens of million dead. Even if the outbreak just corresponds to the normal flu mortality (around 2 per million) that means 280 people globally would die in expectation due to the research program. However, the research would also – in expectation – reduce flu mortality by some fraction. If that fraction is larger than 2% then the net effect would be an overall improvement despite the risk. This seems fine by the Nuremberg Code: “the degree of risk to be taken should never exceed that determined by the humanitarian importance of the problem to be solved by the experiment.”
Except that this is not how we normally reason about risky research. First, there is the long tail problem: if there is a big but unlikely outbreak, then that could easily swamp the number of saved lives in normal years. We might not care about the expected degree of normal risk, but the expected degree of disastrous events. Second, the intentional risking of other’s lives – especially innocents with no say in the experiment – might be problematic from a moral perspective. Third, there is somebody to blame and a policy that can be accepted or not.
The first problem becomes especially acute for existential risks threatening the survival of humanity. However, I think most of us have an intuition that a big disaster is worse than the same number of victims spread out in time and space. The reason is that a correlated disaster has numerous other bad effects: societies are stretched to the breaking points, institutions may crumble, tragedies compound each other. If we accept this, then reducing the probability of extreme tail risks becomes more important than reducing median number of victims. Research or policies that trade huge disasters for more numerous but small tragedies might be the right thing to do.
The second problem is one of knowingly risking lives of innocent people. This may be unavoidable: introducing a new medication could plausibly harm some users that would otherwise have been fine, even if the medication on its own works well (consider giving aspirin to middle-aged people to reduce cardiovascular disease). In the medication case this is somewhat unproblematic because the aim is the benefit of the users: things are worse if the potential harm is imposed on an external group. However, for research with global applicability like reducing pandemic risk or helping the climate, there might not be an external group. Even people who do not know they are benefited by improved medicine or a managed climate would benefit from it. So research that is likely to help everybody in the large eventually might avoid the second problem.
There is still a stakeholder problem: if somebody does something affecting me, do I not have a say about it, even if it is halfway around the world? How much can be left to the researchers or local agencies?
Fouchier and Kawaoka criticized the report, claiming “their work had full ethical, safety and security approval, with the risks and benefits taken into account”. This might be true, but other researchers question whether these approvals are correct: there are subtle issues here about whether the flu community is overly optimistic about their research potential and safety record, or whether other parts of the oversight system and the research critics actually understand the problem right – and how to accurately tell who has the right kind of expertise.
However, the flu research appears to happen within a framework aiming for accountability. It is not perfect, but it can be inspected and if causing trouble controlled or held accountable. The wildcat iron seeding experiment in the Pacific was not done within any framework and seems hard to hold clearly accountable. At the very least, experiments with large-scale effects need to have proportional accountability.
Summing up, it seems that risky experiments can be justified if they look likely to reduce overall risk (especially extreme tail risk), their benefits would accrue to all who are also subjected to risk, and the experiments can be adequately monitored and kept proportionally accountable.
It is interesting to consider these factors for other activities, such as global surveillance. It might be that breeding new pathogens is more ethically justified than NSA espionage.