Utility and practice of systems-based evaluation: a case study of health-service expansion in Lusaka, Zambia

Systems-based evaluation (SBE) methods apply systems concepts to planning and evaluation to elaborate and assess the interconnections and interdependencies of real-life factors, including interrelationships, perspectives, and boundaries, in a process of intervention towards action. Thus far, the utility of SBE approaches has not been rigorously studied for the purpose of providing practical recommendations for evaluators or evaluation managers. This research investigated a case where system dynamics (SD) modelling was applied to evaluate the human capacity impacts of the antiretroviral therapy (ART) expansion strategy in Lusaka, Zambia in 2008. From the perspective of an evaluation manager overseeing the implementation of SD by an experienced modeller, the research: 1) investigated the utility of the approach in that setting and 2) identified implications, principles and methods for future application of SBE. Commonly referenced SBE approaches were reviewed from the literature to categorize key features. An interpretive research approach using critical-hermeneutics philosophy guided the analysis and interpretation of data transcripts and the research journal for the Zambia case. Through observation and in-depth interviews with ten stakeholders before and after development and dissemination of the SD model, and analysis against key features from the literature, a set of critical aspects of utility emerged. While faced with real-world implementation challenges and consistent with the literature, the research affirmed that SBE approaches can incorporate the examination of multiple factors of a system and facilitate investigation of these system factors in a way that brings new insights among stakeholders. In the Zambia case, the SD diagram inspired new and expanded insights on the ART care-and-support system structure but did not influence sweeping strategic changes to the national strategy. Examples from the literature suggest that further facilitation of SD could have enabled such influence. Research concludes that optimal utility of SBE lies on a critical path that begins with engaging stakeholders to share and align their views on a representation of the system and progresses to their reinterpretations of that system (in terms of interrelationships, perspectives and boundaries), ultimately moving towards formal, coordinated action that results in transformative change. For practice, evaluators or managers must balance two purposes in managing for utility of SBE approaches: producing a viable representation of systems and facilitating transformative change appropriately with, for, and by system stakeholders.