Rational Rationality

Game theorists love studying rational agents, yet humans are obviously not rational agents.¹ [¹ No, not even rational agents with very complex utility functions.] Why study rational agents?

In a slogan: it's rational to be rational.

Even in a world with irrational agents, it is individually advantageous to be rational. It never hurts to play the rational strategy—that defines the rational strategy. So it is individually advantageous to become a rational actor. The only equilibrium is for everyone to move toward being a rational agent.² [² Of course, being a rational agent is hard, which is why our world isn't like this. Also, there may be good reasons to not be known as a rational agent, so you may want to strategically act irrationally.]

Yet…

Internet rationalists like to say that rationality is winning, but despite the above I think that's incorrect.

That argument has strength in a one-person game: to win the game is to win the game. But in multi-person games it is not correct. Winning often means other people acting the way you want them to.

Clearly it is not always better when everyone is rational. In a world where prisoners’ dilemmas are constantly being played,³ [³ Non-repeated prisoners’ dilemmas, that is.] everyone would be better off if everyone were irrationally cooperating. However, even in this world there is reason for rationality to take root and drive the world to ruin. Though society is worse off when everyone is rational, each individual gets a better deal by becoming rational: either they start defecting on cooperators, or they stop cooperating with defectors.

In multi-person games, rationality is more the deplorable consequence of winning.⁴ [⁴ I do not understand why internet rationalists sometimes claim to cooperate in prisoner's dilemmas. They have some other, informal notion of rationality, I guess. Rational agents emphatically do not cooperate in one-shot prisoner's dilemmas.] This points the need to something beyond rationality; let's say, rationality for a multi-party world.

I suppose it'd be right to call this thing beyond rationality as morality. It should bear two signs: to be moral must be rational; and, it must be moral. Ideally it should also be itself grounded in rationality—there should be a rational element in determining what is moral. But I suppose that is my own esthetic sense talking.

I’m not aware of any moral theory with game-theoretic roots, but if you know of any, please send it along.