Skip to content

What Got Us Here Won’t Get Us There: Failure Modes on the Way to Global Cooperation

By Joao Fabiano and Diego Caleiro (UC Berkeley, Biological Anthropology)

From single-celled to pluricellular to multicellular organisms or from hunter-gatherers to the EU, the history NASA Flickrof evolutionary forces that resulted in human society is a history where cooperation has emerged at increasingly large scales. The major life transitions and, once human, the major cultural transitions have rearranged the fitness landscape of evolving entities in ways that increased the size of the largest existing coalitions. Notwithstanding, it seems that freewheeling evolution will not lead to satisfactory levels of global human cooperation in time to prevent severe risks. Nor it will lead to the preservation of human values in the long run; humans, human values, and human cooperation are in no way the end-point of evolutionary processes. One proposed solution for the lack of global cooperation is increasing our cooperativeness with the use of enhancement technologies. One proposed solution for the value preservation problem would be placing permanent constraints on these contingent multi-level evolutionary forces necessary for the continuity of human values; placing such constraints would also require a strong global cooperative structure.

We will gather a list of possible paths towards massive global cooperation and discuss how all of them fail; thereby drawing attention to the obstacles to achieving such state. By massive global cooperation we mean one where there are no strong sub-groups aggressively competing between themselves or with the global group. Arguably, this level of cooperation would both (1) drastically decrease risks that arise due to our inability for large-scale cooperation (e.g. the Ultimate Harm), and (2) would guarantee our values are preserved even if evolutionary unfit. We will leave aside the question of whether this solution is required and whether there are other solutions for these problems.

Group condensation

Smaller groups tend to form semi-spontaneously inside a larger scale group, becoming stronger over time and compromising the larger cooperational structure. For instance, suppose we ban most forms of in-group favouritism and existing sub-groups, but allow monogamy to continue as a dominating mating strategy. Over time, in-group favouritism condensates over the nuclear family, and it gradually spreads across society. This process is partially how some classical civilizations were born. The nuclear family is the solid particle that, over time, condensates the volitile cooperation structure into forming sub-groups.

But suppose that, using some moral biochemical manipulation or whatever, we make it so that theNature___Clouds_Thick_clouds_over_the_earth_093737_ prior probability of any individual cooperating with any other is the same. Even then, stochastic variations allow reciprocal binary relationships to evolve, and asset specificity and the like will naturally develop. Regardless of the prior probability, other individuals will find more efficient to cooperate within the dyad due to reciprocity and asset specificity. Sub-groups emerge all over again. To avoid the formation of sub-groups, it seems there has to be active altruistic punishment for in-group favouritism. Any instance of it seems to form a nuclei over which progressively larger and stronger groups will condensate; in the same way atmospheric vapour will condensate into cloulds in the presence of micro water droplets. If massive global cooperation is the result we ought to pursue, then the formation of tight-knit, fraternal, norm-oriented, cooperative and efficient groups could be seen as, in this context, a form organized crime, i.e., systematized and coordinated cheating inside a larger cooperative structure.

Now suppose we ban long-term reciprocity altogether. Nonetheless, if we let leadership skills vary, then naturally individuals will outsource decisions to one another, and this will let groups condensate around natural managers. Therefore, we would have to abolish leadership inequality – and servantship inequality for analogous reasons. It seems that if we were to mirror an ant-colony we would have to be as socially featherbrained as they are. No long-term reciprocity, no long-term mating, no asset specificity, no leadership, no servantship and so on. On the extreme, this would be the abolishment of individual agents, we would have turned into one single superorganism. But then doom-prevention is starting to look more like doom than it should. This solution ultimately fails because it produces the very scenario massive global cooperation was intended to avoid.

Altruism requires bloodshed 

In our path towards increasing human cooperation we might dismiss the fact altruism needs some form of aggression to survive. Eliminating in-group favouritism (parochialism) has to be done thoroughly, because groups without it have no out-group aggression, no defence mechanism and thus become vulnerable. Groups with parochial altruists will always have an advantage over groups with tolerant altruists, thus any remaining group with parochial altruists can immediately take over. No matter how cooperative a group is, if there is no level of parochialism whatsoever, then there will be no incentive to protect the group against aggressors from other smaller groups that do contain parochial altruists. When a tolerant altruistic group encounters a parochial altruistic group, it loses every time, no matter how powerful. This fact is the reason cooperation without aggressiveness never evolved in the first place.

Marlie British (Pinterest)Another path – in line with an alternative explanation for the emergence of altruism – would be to allow strong altruistic punishment irrespective of group membership. In order to counterbalance the parochial-altruists’ danger mentioned above, there would have to be egalitarian measures. There will be no in-group favouritism, but non-parochial altruists would get less punishment iff there is already a high probability outgroups-parochial-altruists will punish them. Call this strong egalitarian altruistic punishment.

Here is a failure mode for both these paths. Even though there is no credible account of the evolution of altruism where it becomes stable without any form of aggression, an increasingly cooperative and altruistic society will see aggression as a primitive monstrosity. Aggression gets less and less frequent, and the idea of evoking any of the paths above seems abhorrent to this evolved cooperative society. Refusing to take the available options, altruists go extinct for their extremism.

Solution. Strong reputational positive feedback with egalitarian corrections. There is no punishment for cheaters or out-groups, but altruists are rewarded and non-parochial altruists with a high probability of being punished by outgroups-parochial-altruists are rewarded more. However, this is all very complex and abstract as far as massive global cooperation proposals go. It might as well be that these affirmative justice measures infer too many costs on non-parochial altruists. It would be as if the UK and US gave extra resources to France during WWII instead of directly fighting the Nazis. No idea if it would have worked. We suspect it would have failed from the start because it is unconvincing. If we wish to look for simpler, already evolved solutions, then of the two hypothesized evolutionary pressures for cooperation, i.e. aggression towards out-groups and aggression towards cheaters, the latter seems the most compatible with massive global cooperation.

Systemic risks

routesSome obstacles arise not in the path towards massive global cooperation but in creating an end-result that is too vulnerable to systematic risks brought by increasedrisk correlation and dependency across society. Increased global cooperation means increased risk correlation on a global scale, increasing the likelihood of existential catastrophe. Our current clusters of increased relative correlation (nations, etc.) protect us from existential threats. There are few systemic risks that would take out all nations together, but if we become only one nation we will lose variability. Moreover, if all individuals within nations become unspecific, we become the bleak repetition of a single immune system variant. Whatever kills one individual will be likely to kill all of them. Whatever threatens any group will threaten all groups.

Solution. Each individual could be packed with an emergency survival strategy. Just like our reptilian brain kicks in when we sense danger, our in-group favouritism and specificity would kick in when we sense existential catastrophe. Like an ant colony that produces more queens every time it senses danger. Society would become some metamorphic superorganism with local fluctuations that would rapidly turn into divergent groups when existential risk increases, only to merge again whenever the risk lowers.

Problem. This strategy looks more like science fiction than a massive global cooperation solution. Perhaps it should be a conditional. Whenever we increase individual specificity or dilute groups into a single humanity, we should also assure that we preserve the ability to differentiate again when needed.


None of the presented solutions is satisfactory. If we currently have reasons to believe massive global cooperation is required, then we should find more feasible and secure ways of producing it.

Share on

7 Comment on this post

  1. Good essay. I don’t think it makes sense to try to build such stable cooperation, because it would certainly be a tyranny worse than the alternative. But nice food for thought.

  2. Suppose there are lots and lots of groups. Some consist of parochial altruists; others consist of tolerant altruists. The tolerant altruist groups will probably find it easier to combine with each other and cooperate, because they genuinely like their allies. So they can form an unbeatable giant tolerant mega-group!

    Happily, the world is not a one-on-one tolerant vs. parochial game.

  3. Why should anyone care about human values “in the long run” if we are potentially talking about, say, 100,000 years in the future? Bostrom needs to stop trying to bully people with this twaddle.

    1. I agree with you that it is not a good idea, because “human values” are often horrible and the future is very uncertain anyway.

      But I think it is unfair to accuse Bostrom of “bullying”; he is wrong, but he is not aggressive. Take a look at politics and the machiavellian game of law-making and breaking, and you will find real bullying.

      1. He’s becoming a priest of a new AI religion. Whether Oxford wants to associate itself with that is up to them, but folks should see this for what it is.

        1. I have a low opinion of people who call everything they disagree with a “religion”. I think it’s rather dull.

          1. Two similiarities with (millenarian) religion:

            1) Robocalypse scenarios immanentize the eschaton.
            2) The implications of unknown future science are inaccessible to empirical inquiry. That’s why it’s unknown.

Comments are closed.