Ultrahigh mobility and Rashba spin splitting in Sb-substituted bismuth telluride and bismuth selenide

Topological insulators (TIs) such as Sb-doped Bi₂Te₃ and Bi₂Se₃ exhibit promising phenomena for advanced spintronics. While previous studies explored isolated doping levels; a systematic understanding of how Sb concentration influences topological behavior, Rashba-type spin splitting, and surface state formation is lacking. Here, we use density functional theory to investigate the structural, electronic, topological and transport properties of (Bi_1-xSb_x)₂Te₃ and (Bi_1-xSb_x)₂Se₃ thin films across 0 ≤ x ≤ 1. We identify pronounced Rashba spin splitting in Bi₂Te₃ at x = 0.5, 0.6, and 0.9 with in-plane helical spin textures. We identified the orbital origins of topological surface states and demonstrate that band inversion persists across the Sb doping range. At x = 0.2, 0.4, and 0.8, calculated surface electron mobilities are consistent with experiments and increase an order of magnitude over Sb₂Te₃, with minimal impact on bulk mobilities. These insights advance our understanding of TIs for spintronic and quantum device applications.