Vocal Complexity Constrains the Dear Enemy Effect: A Comparative Study of Coal Tits and Green-Backed Tits

The "dear enemy effect," wherein territorial animals exhibit reduced aggression toward familiar neighbors compared to strangers, is a widespread strategy to minimize energy expenditure on territory defense. However, whether and how this behavioral capacity varies across with differing vocal complexity remains poorly unclear. We investigated neighbor-stranger discrimination (NSD) in two sympatric tit species that exhibit a stark contrast in song repertoire complexity: coal tits (Periparus ater) and green-backed tits (Parus monticolus). Acoustic analysis revealed that coal tits possessed a large population-level song-type diversity (19 distinct song types) and, crucially, a significantly larger individual syllable repertoire size compared to green-backed tits (5 song types). Playback experiments showed that coal tits exhibited a robust "dear enemy" effect, responding to strangers with significantly closer approach distance and higher flight frequencies near the nest. In contrast, green-backed tits showed uniformly low and undifferentiated responses toward both playbacks of familiar neighbors and strangers, indicating a lack of discrimination. This interspecific divergence was underpinned differences in individual repertoire size and population-level acoustic diversity, with green-backed tits exhibiting higher vocal similarity among individuals. These results demonstrate that the capacity for fine-scale NSD is not universal and suggest that constrained vocal systems-characterized by minimal individual repertoires and high acoustic similarity among individuals-may limit the potential for vocal individual recognition, thereby favoring alternative territorial strategies.