How genetics and social games drive evolution of mating systems in mammals

Traditional explanations for why some animals are monogamous and others are promiscuous or polygamous have focused on how the distribution and defensibility of resources (such as food, nest sites, or mates) determine whether, for example, one male can attract and defend multiple females. 

A new model for the evolution of mating systems focuses instead on social interactions driven by genetically determined behaviors, and how competition among different behavioral strategies plays out, regardless of external factors such as defensible resources. In this model, social interactions can drive evolutionary transitions from one mating system to another, and can even drive a population to split into two separate species with different mating systems.

The model is based on three fundamental behavioral strategies: aggression, cooperation, and deception. The conflict between competitive and cooperative social behaviors drives the evolution of the mating systems. In a paper published December 18 in American Naturalist (online ahead of print publication in the February issue), researchers compared the predictions generated by this model with published data on the mating behavior of 288 species of rodents.

"By and large, everything in our predictions seems to be borne out in rodents," said first author Barry Sinervo, professor of ecology and evolutionary biology at the University of California, Santa Cruz. "Our model is a universal equation of sorts for mating systems."

The evolutionary story that emerges from the study goes something like this: An ancestral population of rodents is promiscuous in its mating behavior. Genetic variation within the population results in individuals with distinctive behaviors. Some males are highly aggressive, defend large territories, and mate with as many females as they can. Others are not territorial, but sneak onto the territories of other males for surreptitious mating opportunities. And some are monogamous and defend small territories, cooperating with neighboring males at territorial boundaries.

These three types can coexist, but any imbalance in the relative advantages of different strategies can lead to the elimination of some behaviors and an evolutionary transition to a species that is, for example, entirely monogamous or entirely polygamous. The cooperative behavior of monogamous males, for example, can include paternal care for the young and the ability recognize and affiliate with other cooperative males, making them stronger in the competition with other strategies.

"They are able to find each other and form colonies, and the bigger the colonies get the stronger they are against the barbarians at the gate. Then they split off from the rest of the population as a separate monogamous species," Sinervo said.

This may sound like little more than storytelling, but in fact it emerges from a set of mathematical equations based on game theory and population genetics, and it is supported by extensive research in animal behavior and genetics.

The new paper builds on Sinervo's decades-long research on mating behaviors in California's side-blotched lizards. He showed that three throat colors correspond with different behaviors in the male lizards: blue-throated monogamous males form partnerships and cooperate to protect their territories and their mates; orange-throated males are highly aggressive and usurp territories and mates from other lizards; and yellow-throated males sneak into the territories of other males to mate.

The competition between these strategies takes the form of a rock-paper-scissors game in which orange aggressors defeat blue cooperators, which defeat yellow sneakers, which defeat orange aggressors. Thus, no single type can dominate the population, and the abundance of each rises and falls in cycles. In 2007, Sinervo and his collaborators discovered the same dynamic in the distantly related European common lizard.

"That was when I started thinking that the same thing could be happening in mammals," Sinervo said.

In the new paper, Sinervo and two of his longtime collaborators--Alexis Chaine at the National Center for Scientific Research (CNRS) in Moulis, France, and Donald Miles at CNRS and Ohio University--generalized the rock-paper-scissors system and extended it to include additional behaviors such as paternal care for offspring (linked to monogamy). They focused on male strategies to simplify the analysis. Sinervo has documented corresponding female strategies in side-blotched lizards and is currently working to incorporate female strategies into the general model. 

The three male behavioral strategies represented in the model are: - Polygyny, characterized by aggression to maintain large territories overlapping with several females, but without paternal care for the offspring, as seen in polygamous mating systems where one male mates with multiple females; - Monogamy, involving lower aggression and smaller territories, with cooperation at territorial boundaries and investment in paternal care; and - Sneak, a non-territorial strategy with no paternal care, resulting in sneaking behavior in otherwise territorial systems.

Using a computer to run a mathematical model of these strategies, the researchers simulated the evolution of mating systems over 1,000 generations, varying the strength of different parameters in each simulation. At the start of the simulations, the genes that determine the different strategies were assumed to be equally abundant in the population.

The results of the simulations revealed four evolutionarily stable outcomes determined by the interactions and payoffs (in terms of reproductive success) of the different behavioral strategies. Which stable outcome emerges depends on how much of an advantage each behavior provides. 

One of the key factors influencing the effectiveness of a given strategy is a male's ability to recognize which behavioral group other males belong to and choose a neighborhood to settle in where his own strategy will have a competitive advantage. Cooperative, monogamous males need to recognize and affiliate with other cooperative males, whereas aggressive, polygynous males want to avoid other aggressive males and find cooperative males whose territories they can take over.

"It all depends on how good you are at finding the right neighborhood, or how good you are at cooperation and paternal care. By varying these parameters in the model, we were able to find the four different evolutionarily stable states," Sinervo said.

One stable outcome is the rock-paper-scissors dynamic documented in lizards, with the coexistence of all three male strategies. Another stable outcome is the coexistence of polygyny and sneak.

There are two stable outcomes in which only one strategy survives, either polygyny or monogamy. A mix of polygyny and monogamy is rare and unstable, eventually leading to a pure system of one or the other.

Turning to the empirical data, the researchers found evidence in studies of rodent behavior and territoriality of the mating systems and behavioral strategies described in the model. There is even a type of mole rat found in southern Africa that exhibits the rock-paper-scissors combo of all three male strategies that Sinervo discovered in lizards. He noted that, whereas mutual recognition of male strategies is based on throat colors in the lizards, in mammals it is more likely to be mediated by smells. "It's there, but we don't see it. We only saw it in lizards because of their bright colors," he said.

The researchers analyzed the phylogenetic tree of rodents (representing the evolutionary relationships among rodent species) and found the same patterns they had seen in the simulations. Species at the base of the phylogenetic tree, closer to the common ancestor of all rodents, tend to be promiscuous, with multiple mating strategies. Polygyny and monogamy very rarely occur together, but they frequently appear in sister species, suggesting they diverged from an ancestral population of mixed strategies. 

The model showed that evolutionary transitions in mating systems are largely driven by increases in the benefits of monogamous behaviors. In rodents, monogamy is the most common evolutionary transition from a promiscuous ancestor, and more rodents are monogamous than polygynous. In the simulations, pure polygyny is a relatively uncommon outcome. "Polygyny is readily invaded by the sneak strategy," Sinervo explained.

Paternal care for the offspring is found in all monogamous species, supporting a key assumption linking paternal care to the evolution of monogamy.

"Promiscuity is very common, and can involve two or three different strategies. But the neat thing is that cooperation and monogamy are far more common than anyone realized," Sinervo said. "The frequency of monogamy in rodents is about 26 percent, much higher than for mammals in general and similar to primates."

The model assumes that these behavioral strategies are genetically based. Evidence in support of this includes research on the role of the hormone vasopressin (and the related hormone oxytocin) in complex social behaviors in numerous species, including rodents and humans. In the monogamous prairie voles, for example, vasopressin has been linked to pair bonding, mate guarding, and paternal care. In some rodent lineages, evolutionary transitions between monogamy and polygyny have been linked to a mutation in a vasopressin receptor gene. 

The effects of the genes underlying monogamous behaviors may even drive the evolution of more advanced forms of sociality. Highly social species of rodents--such as mole rats, some of which live in colonies in which only one pair reproduces--originate from monogamous lineages.

Sinervo and his coauthors are not claiming that resources and other external ecological factors have no role in the evolution of mating systems. But the genetic model gives predictions that are consistent with the rodent data and can explain cases where a species' mating system does not match its resource ecology.

The authors also acknowledged that animal behavior can be very flexible and is not entirely determined by genetics. This is especially true in humans, whose behavior is so strongly influenced by cultural and environmental factors. In terms of mating systems, our species can be described as promiscuous, but with very high rates of monogamy. Sinervo said he sees a connection between monogamy and the deeply cooperative social behaviors that are at the core of the human condition.

"We can see analogues for human behavior in other animals, but there's really nothing else like humans," Sinervo said. "There are 'kneejerk' behavioral impulses in us that are not far from rodents, but our cultural and social complexity makes us very different from most mammals."