Multiple and mixed methods in formative evaluation: Is more better? Reflections from a South African study

Area of Science:

• Public Health
• Health Services Research
• Program Evaluation

Background:

• Formative evaluations explore intervention implementation to understand program function.
• Critical assessments of methods used in formative evaluations are scarce, particularly regarding objective achievement.
• This paper details a formative evaluation of a South African community-based lay health worker program for TB and HIV/AIDS clients.

Purpose of the Study:

• To describe a formative evaluation of a community-based lay health worker program.
• To assess the methods used in relation to evaluation objectives.
• To offer suggestions for optimizing multiple, mixed-methods in formative evaluations of complex health interventions.

Main Methods:

• Employed qualitative methods: interviews, focus groups, observations, and diary keeping.
• Utilized quantitative methods: time-and-motion study and client surveys.
• Authors conceptualized, conducted, and iteratively assessed evaluation methods and results.

Main Results:

• The multiple, mixed-methods approach yielded insights beyond traditional single-method evaluations.
• Strengths included detailed program understanding and triangulation of diverse stakeholder perspectives.
• Careful planning and implementation are crucial, as multiple methods can strain resources.

Conclusions:

• Multiple qualitative and quantitative methods offer advantages for formative evaluations of complex interventions.
• The value lies in method-question alignment and scientific integrity, not the number of methods.
• Optimizing mixed-methods requires careful consideration of evaluation objectives and resources.