Supporting Adolescents With HIV in South Africa Through an Adherence-Supporting App: Mixed Methods Beta-Testing Study

Novel smartphone app-delivered interventions have the potential to improve HIV treatment adherence among adolescents with HIV, although such interventions are limited. Our team has developed Masakhane Siphucule Impilo Yethu (MASI; Xhosa for "Let's empower each other and improve our health"), a smartphone app-delivered intervention to improve treatment adherence among adolescents with HIV in South Africa. MASI was adapted to the South African cultural context using the HealthMpowerment platform, an evidence-based digital health intervention developed for and with youth in the United States.

We conducted this beta-testing study to (1) explore the initial usability of MASI, (2) examine engagement and experiences using MASI features, and (3) inform refinements to the app and intervention implementation plan prior to a subsequent pilot randomized controlled trial (RCT).

This study was conducted from August 2021 to December 2021 in Cape Town, South Africa. Beta-testing participants received access to MASI for 3 weeks. A mixed methods approach was used, with brief questionnaires and semistructured in-depth interviews conducted prior to app installation and after 1 week to 2 weeks of app testing. Engagement with MASI was measured through analysis of back-end app paradata, and follow-up in-depth interview guides were tailored to each participant based on their app use.

Participants in the beta-testing study (6 male participants, 6 female participants; ages 16-19 years) collectively spent 4.3 hours in MASI, averaging 21.4 minutes per participant over the 3-week period (range 1-51.8 minutes). Participants logged into MASI an average of 24.1 (range 10-75) times during the study period. The mean System Usability Scale score was 69.5 (SD 18), which is considered slightly above average for digital health apps. Thematic analysis of qualitative results revealed generally positive experiences across MASI features, although opportunities to refine the app and intervention delivery were identified.

Initial usability of MASI was high, and participants described having a generally positive experience across MASI features. Systematically analyzing paradata and using the interview findings to explore participant experiences allowed us to gain richer insights into patterns of participant engagement, enabling our team to further enhance MASI. The results from this study led to a few technological refinements to improve the user experience. Enhancements were also made to the intervention implementation plan in preparation for a pilot RCT. Lessons learned from the conduct of this beta-testing study may inform the development, implementation, and evaluation of similar app-delivered interventions in the future.