Hamilton's rule and kin competition in a finite kin population

Kin selection means that individuals can increase their own inclusive fitness through displaying more altruistically toward their relatives. So, Hamilton's rule says kin selection will work if the coefficient of relatedness exceeds the cost-to-benefit ratio of the altruistic act. However, some studies have shown that the kin competition due to the altruism among relatives can reduce, and even totally negate, the kin-selected benefits of altruism toward relatives. In order to understand how the evolution of cooperation is influenced by both kin selection and kin competition under a general theoretical framework, we here consider the evolutionary dynamics of cooperation in a finite kin population, where kin competition is incorporated into a simple Prisoner's Dilemma game between relatives. Differently from the previous studies, we emphasize that the difference between the effects of mutually and unilaterally altruistic acts on kin competition may play an important role for the evolution of cooperation. The main results not only show the conditions that Hamilton's rule still works under the kin competition but also reveal the evolutionary biological mechanism driving the evolution of cooperation in a finite kin population.