Sherr et al. BMC Health Services Research 2017, 17(Suppl 3):827
DOI 10.1186/s12913-017-2658-5
RESEARCH Open Access
Measuring health systems strength and its
impact: experiences from the African
Health Initiative
Kenneth Sherr1,2*, Quinhas Fernandes3, Almamy M. Kanté4, Ayaga Bawah5, Jeanine Condo6, Wilbroad Mutale7
and the AHI PHIT Partnership Collaborative
Abstract
Background: Health systems are essential platforms for accessible, quality health services, and population health
improvements. Global health initiatives have dramatically increased health resources; however, funding to strengthen
health systems has not increased commensurately, partially due to concerns about health system complexity and
evidence gaps demonstrating health outcome improvements. In 2009, the African Health Initiative of the Doris Duke
Charitable Foundation began supporting Population Health Implementation and Training Partnership projects in five
sub-Saharan African countries (Ghana, Mozambique, Rwanda, Tanzania, and Zambia) to catalyze significant advances in
strengthening health systems. This manuscript reflects on the experience of establishing an evaluation framework to
measure health systems strength, and associate measures with health outcomes, as part of this Initiative.
Methods: Using the World Health Organization’s health systems building block framework, the Partnerships present
novel approaches to measure health systems building blocks and summarize data across and within building blocks to
facilitate analytic procedures. Three Partnerships developed summary measures spanning the building blocks using
principal component analysis (Ghana and Tanzania) or the balanced scorecard (Zambia). Other Partnerships developed
summary measures to simplify multiple indicators within individual building blocks, including health information systems
(Mozambique), and service delivery (Rwanda). At the end of the project intervention period, one to two key informants
from each Partnership’s leadership team were asked to list – in rank order – the importance of the six building blocks in
relation to their intervention.
Results: Though there were differences across Partnerships, service delivery and information systems were reported to
be the most common focus of interventions, followed by health workforce and leadership and governance. Medical
products, vaccines and technologies, and health financing, were the building blocks reported to be of lower focus.
Conclusion: The African Health Initiative experience furthers the science of evaluation for health systems strengthening,
highlighting areas for further methodological development – including the development of valid, feasible measures
sensitive to interventions in multiple contexts (particularly in leadership and governance) and describing interactions
across building blocks; in developing summary statistics to facilitate testing intervention effects on health systems and
associations with health status; and designing appropriate analytic models for complex, multi-level open health systems.
Keywords: Health system strengthening, Metrics, African Health Initiative, Ghana, Mozambique, Tanzania, Rwanda,
Zambia
* Correspondence: ksherr@uw.edu
1Department of Global Health, University of Washington, 1959 NE Pacific St,
Seattle, WA, USA
2Health Alliance International, Seattle, WA, USA
Full list of author information is available at the end of the article
© The Author(s). 2017 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and
reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to
the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver
(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Sherr et al. BMC Health Services Research 2017, 17(Suppl 3):827 Page 30 of 94
Background feasibility; and 4) consistency with global standards.
For over a decade, there have been calls to invest in low Though Partnerships were aware of the limitations of the
and middle-income country health systems to ensure building blocks framework – in particular with regards to
stable platforms are in place to maximize evidence- capturing interactions across blocks, and the missing
based health interventions through their delivery at scale element of ‘people’ – it was ultimately decided to
[1, 2]. Underpinning the need to strengthen fragile, operationalize an established framework rather than adapt
resource-constrained health systems is the recognition or develop a framework solely for the Initiative. Note that
that weak health systems impede attainment of global outcome and impact indicators, as well as contextual
and national targets [3], and are insufficiently resilient to factors, were defined by the same Data Collaborative and
prepare for – and respond to – crises [4]. Despite this Partnership representatives to map against the AHI
recognition and the rapid resource expansion from conceptual framework, though using an expanded set of
global health initiatives, investments in health systems reference frameworks guidance documents [12].
have decreased relative to the overall funding envelope Despite guidelines recommending the use of standard-
[5]. Stagnation in resource expansion through these ini- ized indicators – including measurement strategies to
tiatives raises further concerns that health system invest- support monitoring and evaluation of health systems
ments will only decrease, with prioritization of targeted, strengthening interventions [14, 15] – gaps remain in
disease-specific efforts [6]. A lack of shared understand- ensuring that measures are valid, sensitive to health sys-
ing of what constitutes ‘health systems strengthening,’ tems interventions, and readily available at the adminis-
the potentially high cost of comprehensive health sys- trative level that health systems strengthening efforts
tems interventions, and a weak evidence-base linking target [16, 17]. The state of knowledge on measuring
population-level health benefits with health systems health impacts and outcomes, however, is further ad-
strengthening strategies undermines broad investments vanced. For example, from the recent publication of 100
in this area. core indicators by the WHO – 77 of the measures fall
To address this evidence gap and catalyze investments into the impact (29) and outcome (48) categories, and
in health systems, the Doris Duke Charitable Foundation only 22 are specific to health system strengthening (in-
(DDCF) launched the African Health Initiative (AHI), puts and processes (12)), or proximally related to these
which supported Population Health Implementation and inputs and processes (10) [14]. Furthermore, of the
Training in five diverse sub-Saharan African countries ‘health system’ indicators, 42% map against the service
(Ghana, Mozambique, Rwanda, Tanzania and Zambia). delivery building block; 26% health financing; 12% health
Since 2009 (when support for Partnership implementa- information; 8% health workforce; 4% medical products,
tion was initiated), these Partnerships have implemented vaccines and technology; and none in leadership and
distinct interventions designed to strengthen health sys- governance. The lack of scientifically valid metrics
tems in their respective contexts, and measured the im- reflecting health systems functioning across all of the
pact of these investments on health system functioning building blocks impedes efforts to monitor and evaluate
and health outcomes [7–11]. interventions designed to strengthen health systems, and
To foster cross-site learning and produce results that build an evidence-base supporting health systems
may be generalizable to other low and middle-income strengthening to improve health outcomes.
countries (LMICs), DDCF instituted a Data Collaborative Attempts to evaluate health systems strengthening in-
to work with the Partnerships to develop a conceptual terventions – in terms of their effects on health systems
evaluative framework, with core and common metrics and health status – are nascent, and lack robust,
mapping against this framework (including inputs/pro- standardized methodologies for assessing complex inter-
cesses, outputs, outcomes and impact) to be gathered ventions implemented at a sub-national scale [16].
across Partnerships [12]. As part of the consultative Quantitative evaluations have largely focused on the
process, Partnerships used the World Health Organiza- impact of health systems on population health measures,
tion’s (WHO) health systems framework (comprised of six including the impact of individual health system compo-
building blocks of service delivery; health workforce; nents (e.g. financing or health workforce) on health
information systems; medical products, vaccines & status across multiple countries [18, 19]; a partial list of
technologies; health financing; and leadership & govern- health system components on health status across
ance) to identify core and common metrics on project in- countries [20, 21]; or a partial list of health system com-
puts, processes and outputs [13]. Candidate core and ponents on health status over time at a national or sub-
common input, process and output metrics were reviewed national level [22–24]. Published literature also includes
and selected by the Data Collaborative and Partnerships qualitative endeavors assessing the perceived impact of
based on 1) validity; 2) relevance and sensitivity to individ- health systems strengthening approaches broadly at a
ual Partnership aims and processes; 3) measurement national level [25], or at the micro-level (sub-national or
Sherr et al. BMC Health Services Research 2017, 17(Suppl 3):827 Page 31 of 94
on individual building block components) [26]. Though were asked to list – in rank order – the importance of
important for generating evidence on the role of health the six building blocks in relation to their intervention
systems components as determinants of population (Table 1). Though there were differences across Partner-
health status, current research does not adequately cap- ships, service delivery and information systems were
ture the complex, inter-connected relationships between reported to be the most common focus of interventions,
health system building blocks, and the setting in which followed by health workforce and leadership and govern-
they are situated [27]. Realist evaluation [28] and ap- ance. Medical products, vaccines and technologies, and
proaches based on complexity theory [29] may have the health financing, were the building blocks reported to be
advantage of addressing health systems interdependence of lower focus.
and implementation context, though their ability to lead To understand differences in data availability and situ-
to generalizable knowledge on health systems strength- ate selected measures of health system strength and its
ening interventions is unclear. impact within each project, the same key informants
The purpose of this manuscript is to describe ap- were requested to list – in rank order – the administra-
proaches from the five PHIT Partnerships to 1) measure tive level of the health system prioritized for Partnership
health systems strength, and 2) demonstrate its impact interventions (Table 2). Notably, two Partnerships
on the delivery of health services and population health. (Ghana and Tanzania) emphasized the community level
Through surfacing common and distinct experiences in their intervention design; another two (Rwanda and
from the Partnerships, we highlight the complexity in Zambia) emphasized the health facility level; and one
measuring health systems and its impact on health out- (Mozambique) emphasized the district level. Only one
comes and impact, and discuss opportunities and prior- Partnership intervened at the provincial (Mozambique)
ity areas for the future. By reporting on the Partnerships’ and national (Rwanda) levels.
experience with operationalizing measures of health sys-
tems strengthening, and analytic approaches to link Results
these inputs and processes with improved health services Partnership measures and data collection approaches
and population health, this article will be of interest to Because all Partnerships were asked to include under-
those engaged in designing and implementing complex five mortality (5q0) as the primary study outcome, and
interventions to improve the delivery of primary health all agreed on core outcome and impact measures early
care – including ministries of health, researchers, imple- in the Initiative, outcome and impact measures varied
menters, policymakers and funders. little across countries. However, given the diverse imple-
mentation settings and intervention designs, there are
Methods notable differences in Partnership measures of health
Partnership approaches to strengthening health systems system strength (Table 3) and data collection approaches
By design, the five Partnerships are responsive to specific (Table 4). For three building blocks (health financing;
needs in their country contexts (intervention descrip- medical products, vaccines and technologies; and service
tions have been previously published) [7–11]. Though delivery), there were commonalities in measures across
there may be commonalities in how Partnerships countries, which is likely due to both more established
strengthen individual health system building blocks, de- measures that are feasible to routinely collect, as well as
scribing differences in the health system focus of each agreement on core metrics in these areas early in the
Partnership clarifies reasons for the variations of mea- Initiative. However, there were differences in Partnership
sures of health system strengthening gathered across measures across the remaining building blocks of
countries to best assess their respective interventions. At leadership and governance, health workforce, and health
the end of the project intervention period, one to two information systems, reflecting the lack of established
key informants from each Partnership’s leadership team metrics (e.g. leadership and governance), differing levels
Table 1 PHIT Partnership countries’ ranking of intervention emphasis by health system building block
Service Information Health Leadership & Medical products, Health
delivery systems workforce governance vaccines & technologies financing
Ghana 1 4 5 3 6 2
Mozambique 3 2 5 1 6 4
Rwanda 1 2 3 4 5 6
Tanzania 2 5 1 3 4 6
Zambia 3 2 1 4 5 6
Median 2 2 3 3 5 6
Sherr et al. BMC Health Services Research 2017, 17(Suppl 3):827 Page 32 of 94
Table 2 PHIT Partnership countries’ ranking of administrative as core to the Initiative. Mozambique is also implement-
unit of emphasis by health systems building block ing a costly endline population-based survey, as there is
Community Health District & Province National no national survey that includes core initiative metrics
facility sub-district timed with the end of the intervention. Three countries
Ghana 1 2 3 0 0 (Ghana, Mozambique and Tanzania) carried out time-
Mozambique 0 2 1 3 0 motion studies to quantify human resource use patterns,
Rwanda 3 1 2 NA 4 including wait and consult times. Partnerships
Tanzania 1 3 2 0 0 highlighted the resource requirements to collect these
data, and though useful in understanding staffing pat-
Zambia 2 1 3 0 0
terns, there were questions about the sensitivity of time-
Median 1.5 2 2 3 4 motion measures to program interventions in settings
NA not applicable, 0 = not included in Partnership design, and not included in with severe personnel shortages. There were additional
calculation of median
concerns about the consistently high level of reported
patient satisfaction, which is of limited use for informing
of importance given Partnership interventions (e.g. targeted action or for Partnership evaluations.
health information), and different approaches of Partner-
ships themselves (e.g. health workforce). Summarizing health systems strength
All countries relied on a mix of sources for health sys- Given the complexity of health systems, approaches to
tems data, including facility surveys, population-based measure health system strength must be multi-faceted
surveys, internal monitoring systems, and government and include multiple indicators across the six building
health management information systems (HMIS). Simi- blocks, which presents a challenge in succinctly sum-
larities across Partnerships included a reliance on facility marizing health system strength. There are two principal
surveys to populate service readiness data (including needs for summary measures reflecting health systems
quality of care and patient satisfaction), and continuous strength, including that 1) they enable rapid monitoring
stock of essential supplies and commodities. The fre- of health system capacity for targeted action by minis-
quency of facility assessments, sampling approach, and tries of health, and 2) reducing the hundreds of health
inclusion of comparison areas differed by country con- systems indicators into a limited set of metrics is re-
text, and because of the stepped wedge design, only quired to quantify both the effect of interventions on
Zambia carried out facility surveys in all intervention health systems strength, and between health systems
and comparison areas. All Partnerships noted the value strength and measures of health service delivery and
of facility assessments as a principal source of health population health.
systems data; however, respondents voiced concern The five Partnerships all employed techniques to
about the validity of the findings from facility assess- summarize health systems data, though approaches dif-
ments, as well as the large quantity of data that are not fered across countries (see Fig. 1). Using principal com-
readily summarized for further analysis. The use of rou- ponents analysis on national health facility surveys, two
tine data was found to be efficient, and leveraged health Partnerships (Ghana and Tanzania) constructed compos-
information system improvement activities. ite indices that aimed to provide robust measures of
All countries relied on population-based surveys for health system capabilities [30, 31]. A limitation of these
outcomes and impact data, and to estimate service data, however, is that the surveys were not carried out to
utilization for programs most relevant to their theory of the dispensary level (the level of focus in both Partner-
change. Two partnerships relied on existing population- ships). A third Partnership (Zambia) adapted the WHO
based surveys as their primary data source (e.g. Demo- balanced scorecard [15] to summarize data from health
graphic and Health Surveys), which gained efficiencies, facility surveys (implemented as part of the Partnership’s
and in the case of Rwanda, was bolstered through over- evaluation plan) into 19 measures that crossed seven
sampling. Reliance on national surveys did pose chal- health system domains [32, 33]. Efforts were made to
lenges, however, in terms of 1) having limited flexibility summarize data within building blocks. In Mozambique,
in modules included in the surveys, 2) the relatively lim- where improving data quality was a priority, a summary
ited power of national community surveys considering measure was developed that collapsed the dimensions of
sub-national (and at times sub-provincial) intervention data availability and concordance using four indicators
and comparison areas, and 3) having no control over the from facility reports over 12 months into one facility-
timing of surveys. As a result, certain measures (e.g. hav- level proportion [34, 35]. In Rwanda, the Partnership de-
ing four or more antenatal care visits during the last veloped a composite measure of service quality as part
pregnancy, which was not included in a Multi-Indicator of their quality improvement approach (specifically to tar-
Cluster Survey in Mozambique) could not be included get facility-level improvement efforts). The Rwanda
Sherr et al. BMC Health Services Research 2017, 17(Suppl 3):827 Page 33 of 94
Table 3 PHIT Partnership countries’ health systems strengthening measures by health systems building blocka
Ghana Mozambique Rwanda Tanzania Zambia
Service Delivery Indicators • Service utilization & quality • Service utilization for • Service utilization for • Service utilization for • Service utilization for selected
for selected programs selected programs selected programs selected programs
• Emergency referrals • Patient satisfaction • Facility capacity programs • Patient satisfaction with PHC
• Cause of maternal deaths with PHC services • Quality of care: Appropriate • Time use of CHWs services
• Average wait/consult diagnosis and treatment • Quality of IMCI • Quality of care: readiness and
time for PHC services quality audit
Sources -HMIS -Facility surveys -Facility surveys -Health & Demographic -Facility surveys
-Project information system -HMIS -HMIS Surveillance System -Community surveys
-Mortality audits -Community surveys -Community surveys -Facility surveys
-Community surveys -Project information system
-Community surveys
Information Systems Indicators Not measured • Facility & district data quality • Community & facility • % monthly CHW reports • Timely health facility and CHW
data quality submitted reporting
Sources -Data quality audits -Data quality audits - Project information system -Project information system
Health Indicators • CHW coverage (stratified by • Health worker per capita • Health worker presence • CHW presence during • Health worker motivation
Workforce community health nurse and by cadre • Retention supervision • Health workers trained in
midwife) • Frequency of supervision • Competency, knowledge • CHWs selected/trained/ previous 12 months
• # and % of CHWs trained in visits by district & skills training results/retained
IMCI & supervision frequency • Supervisors trained & working
• Frequency of supervision visit
to CHWs
Sources -Community survey -Project information system -Facility surveys -Project information system -Facility surveys
-Project information system -Project information system
Leadership & Indicators • Perceptions of governance • Data utilization • Data utilization Not measured • Evidence-based planning
Governance and leadership changes • % of district management • Appropriate use of resources • Level of corruption
teams fully staffed
• Turnover of district and
provincial management teams
Sources -Process evaluation -Special study -Special study -Facility surveys
-Project information system -Policy analysis
Medical Products, Indicators • Continuous stocks of essential • Continuous stocks of • Continuous stocks of essential • Continuous stocks of essential • Continuous stocks of essential
Vaccines & commodities (tracer medicines and essential commodities commodities (tracer medicines commodities (tracer medicines commodities (tracer medicines
Technologies equipment) (tracer medicines and and equipment) and equipment) and equipment)
equipment)
Sources -Project information system -Facility surveys -Facility surveys -CHW stock card review -Facility surveys
-Community survey
-Facility surveys
Health Financing Indicators • Total costs in intervention area • Total costs in • Total costs in intervention area • Total costs in intervention area • Total costs in intervention area
• Incremental project cost intervention area • Cost by source • Incremental project cost • Incremental project cost
• Incremental project cost • Incremental project cost
Sources -Project information system -Ministry of Health reports -Ministry of Health & local -Project information system -Facility surveys
-Project information system government reports
-Project information system
aFurther details on the following metrics are available as a web annex to the following publication(Bryce et al., [12]): Medical products, vaccines & technologies; CHW Community Health Worker,
HMIS Health Management Information System, PHC Primary Health Care, IMCI Integrated Management of Childhood Illnesses
Sherr et al. BMC Health Services Research 2017, 17(Suppl 3):827 Page 34 of 94
Table 4 Data collection strategy by PHIT Partnership country: Frequency and Sampling
Ghana Mozambique Rwanda Tanzania Zambia
Facility-based Frequency • CHW time-motion • Patient exit and time-motion • Health facility • CHW time-motion survey: Baseline • Health facility survey:
data collection survey: Baseline survey: Baseline and endline survey: Quarterly • Health facility survey: Baseline Baseline, midline, and
• Health facility survey: • Health facility survey: Annual and midline endline
Baseline and midline service provision assessment
and data quality audit
Sampling Time-motion survey: Patient exit & time-motion All 22 facilities in Time-motion survey: Intervention areas All intervention and
Intervention areas survey: Intervention and intervention area Facility survey: Sample of 107 (baseline) comparison areasa
Facility survey: Intervention comparison areas and 141 (midline) facilities in intervention
and comparison areas Facility survey: 27 purposively & comparison areas
selected facilities, intervention
area
Special studies Frequency Qualitative process Qualitative process Qualitative process Qualitative process evaluation Qualitative process
evaluation evaluation: endline evaluation evaluation
Policy analysis
Sampling Key informant interviews Key informant interviews with Key informant interviews Key informant interviews & focus Key informant interviews
& focus group discussions purposively selected stakeholders with purposively selected group discussions with community, with facility and district
with district and national in intervention area local and national health facility and district health leaders leaders
health leaders leaders
Community Surveys Frequency Community survey: Community survey: Baseline, Community survey: Baseline Health and Demographic Community survey:
/ Demographic Baseline & endline midline (2 surveys), & endline & endline Surveillance System: Semi-annual Approximately annual
Surveillance System Demographic Surveillance
system (Navrongo):
Semi-annual
Sampling Intervention and Intervention and Intervention and Intervention and comparison areas Intervention and
comparison areas comparison areas comparison areas comparison areas
Health Frequency Monthly facility reports Monthly facility reports Monthly facility reports Monthly CHW activity reports Monthly facility reports
Management
Information System
Sampling Full capture for all Full capture for all facilities Full capture for all facilities Full capture for all CHWs in Full capture for all
intervention and in intervention area in intervention area intervention area facilities in intervention
control facilities and comparison areas
Project Frequency Ongoing training, Ongoing training, supervision Ongoing training, Ongoing training, supervision Ongoing training,
Monitoring System supervision, program program and financial reports supervision program and program and financial reports supervision program
and financial reports Ongoing contextual financial reports Ongoing contextual data tracking and financial reports
Ongoing contextual data tracking Ongoing contextual data Ongoing contextual
data tracking tracking data tracking
Sampling Routine activity reports: Routine activity reports: Routine activity reports: Routine activity reports: Routine activity reports:
Intervention area Intervention area Intervention area Intervention area Intervention and comparison
Contextual data: Contextual data: Intervention Contextual data: Contextual data: Intervention areas
Intervention and and comparison areas Intervention and and comparison areas Contextual data: Intervention
comparison areas comparison areas and comparison areas
aNote: As Zambia used a stepped-wedge design, intervention and comparison facilities vary over time
Sherr et al. BMC Health Services Research 2017, 17(Suppl 3):827 Page 35 of 94
Fig. 1 Novel summary measures of health systems strength by PHIT Partnership
Partnership also developed a micro-level composite indi- including the presence of other initiatives – to enable at-
cator for neonatal health screening that summarized per- tribution of effect to Partnership interventions. Second,
formance at the facility level for further targeted action. Partnerships noted difficulties in teasing apart the relative
Only one measure collected across all countries cut across contributions of different components of the health
all building blocks – the total cost of health services, and system on overall impact, given the interdependence
the incremental contribution of each PHIT Partnership. across building blocks, and that the relative contribution
of different components is likely unequal (with ‘dose’ vary-
Approaches to associate health system strengthening ing by Partnership design, and over time). Inherent in this
with outcome and impact measures challenge is recognition that critical attributes of a health
It is beyond the scope of this article to describe the system – such as trust, resilience, quality, and leadership
Partnerships’ analysis plans. However, given the experi- – are not easily quantified, and as critical for routine func-
ence of the Partnerships in designing analytic tioning across other building blocks, likely confound the
approaches to assess the effect of their complex inter- assessment of intervention effects on individual building
ventions, the following section describes novel blocks. In addition, given the complexity of Partnership
approaches used to incorporate measures of health interventions, there were differences of opinions about
systems strength into Partnership analytic plans. All using adaptive designs allowing for innovations based on
Partnerships planned to assess 1) if the intervention is lessons learned during implementation, versus strict ad-
associated with improvements in population-level health herence to the initial program design. There are no easy
status; 2) if health systems were strengthened over time answers to these questions, though we expect some clarity
in intervention areas compared with comparison areas; as the field of evaluation sciences develops.
and 3) if health systems strengthening is associated with
improvements in health service coverage and Discussion
population-level health status (5q0 in all countries, Here we present how five Partnerships supported
though countries will also assess neonatal (NN) and in- through the Doris Duke Charitable Foundation’s African
fant (1q0) mortality – and in the case of Zambia and Heath Initiative approached the collection of a set of
Tanzania – adult mortality). All Partnerships assessed core and common metrics for health systems strength-
improvements in collected measures of health systems ening, and approaches to simplify and operationalize
strength by building block, though Ghana, Tanzania and these measures to assess the effect of interventions on
Zambia planned to operationalize their summary meas- health system functioning and population health. Each
ure of health systems strength in their analyses. Partnership was unique in intervention design and
At the time of writing, Partnerships are still collecting setting (located in five sub-Saharan African countries),
final outcome data, or are carrying out final analyses, but shared a list of core (shared by all Partnerships) and
though initial work has generated insights into what has common (shared by multiple Partnerships) metrics that
worked well in evaluating complex health systems inter- provides a solid set of experiences to learn about meas-
ventions in the five countries, as well as challenges in this uring health systems strength and its broader impact.
area of inquiry. Partnerships noted that a prospective, The shared experience of the Partnerships demonstrates
mixed methods approach is essential to understand if the difficulties in quantifying health system inputs and
health systems are improving, and to unpack the middle processes, and health system strength, due to a lack of
of ‘how’ and ‘why’ interventions are or are not leading to scientifically valid measures that are sensitive to varied,
improvements in service delivery coverage and health sta- complex interventions in multiple health system
tus. There were noted challenges, including questions on contexts. While some building blocks (e.g. economic in-
whether and how to adjust for contextual factors – puts), outcomes and health status have established
Sherr et al. BMC Health Services Research 2017, 17(Suppl 3):827 Page 36 of 94
measures and data sources, others (e.g. governance and strengthening by demonstrating different collection
information systems) are particularly challenging. The strategies, and highlighting measures that are feasible,
PHIT Partnership experience provides examples of valid and sensitive to interventions across multiple
health system measures – identifying advancements in settings. Current global indicator standards [14, 36]
some areas, and needs for further development – and are weighted towards outcomes and health status –
describes novel approaches to summarize health system and within the building blocks – towards financing
measures for operationalization in evaluation of complex input levels and health workforce numbers and their
interventions. Ultimately, assessing health systems and distribution, which reflects the greater availability
their impact requires mixed-methods, relying on data and validity of these indicators relative to those
from multiple, complementary sources. across the other health system building blocks. There
Early in the Initiative, Partnerships agreed to use the was consistency across Partnerships in terms of mea-
WHO health system framework to orient the selection of sures of health information, medical technologies and
core and common indicators. At the end of this Initiative, service delivery, collected via facility surveys that are
Partnerships reflected on the strengths and weaknesses of expensive, inconsistently conducted, and in the exam-
this framework. The building blocks framework was found ples presented here were not representative in mul-
to be useful in separating out the ingredients of health sys- tiple countries (either not reflective of the level of
tems, and identifying key measures for these domains to Partnership intervention, or did not include both
enable Partnerships to document inputs and processes as- intervention and comparison areas). The lack of
sociated with their interventions, and quantifying the im- scientifically valid and appropriate measures for the
pact of these inputs and processes with outcomes and building block of ‘governance and leadership,’
impact. However, limitations with the building blocks (including indicators related to leadership and
framework as a guide for metrics for health systems management at sub-national levels, beyond the exist-
strengthening were identified. Partnerships noted that – ence of up-to-date national policies, that can be op-
though the framework isolated key ingredients in the erationalized for analysis), has been noted elsewhere
health system – it did not capture the interaction between [37, 38]. This gap is especially worrisome given that
building blocks. Feedback loops (both positive and nega- leadership and governance is critical to strong, effect-
tive) between building blocks are important in the context ive health systems, and likely reverberates across all
of measuring health systems strength, given the likely in- other building blocks [39]. It is a priority to develop
teractions in intervention effects across blocks and the in- and validate across multiple contexts new measures
ability to impact building blocks in isolation (for example, for leadership and governance, including how differ-
improve health workforce without also improving facility ent types of evidence are used by decision makers.
conditions and/or leadership). Furthermore, the frame- All country teams developed approaches to
work was found to inadequately capture implementation summarize data across health system building blocks or
context – including social and organizational context. As within individual blocks (aligning with Partnership
a predominately supply-side model, the framework does theories of change), and plan to incorporate these
not adequately capture the block of ‘people’, including summary measures in final analysis. These examples
community linkages, linkages with non-formal leaders, the provide guidance for others researching health systems,
role of the private sector, and the importance of demand and working to operationalize summary systems mea-
creation in bridging health needs with service availability. sures in analytic procedures. Further work is needed to
Measuring health systems strength requires a better un- validate these summary measures of health systems
derstanding of how health systems support community strength, which is a still-forming methodologic frontier
needs, and how communities contribute to health systems that must address metric performance given varied
strengthening [2]. Despite its limitations, the building contexts and health system complexity [16]. A common
block framework was useful to guide the complex process methodologic challenge across teams related to the pri-
of identifying core and common measures of health sys- mary outcome measure – 5q0 – especially in countries
tems strength, and could be improved upon by adapting without a health and demographic surveillance system,
the framework to be an open system that recognizes link- where national-level community surveys do not capture
ages between its components, and with the broader con- these relatively rare events with sufficient precision, at
text in which it is situated. Alternative evaluation meaningful time intervals, and at the district level
approaches – such as realist evaluation [28] and evalua- (where Partnerships intervene). As has been described
tions built on complexity science [29] – are also relevant elsewhere, 5q0 may be a sub-optimal measure to evalu-
in explicitly addressing systems complexity and context. ate complex health system interventions, as secular
The experience of the five Partnerships enriches trends in 5q0 may hinder detection of reductions due
previous efforts to develop metrics for health systems to the interventions. Furthermore, multiple pathways to
Sherr et al. BMC Health Services Research 2017, 17(Suppl 3):827 Page 37 of 94
impact 5q0, and concurrent health and non-health Nimako, Nicholas Kanlisi, Elizabeth F. Jackson, Mallory C. Sheff, Pearl Kyei,
sector inputs, may hinder attribution to specific Patrick O. Asuming, Adriana Biney, Roma Chilengi, Helen Ayles, Moses
Mwanza, Cindy Chirwa, Jeffrey Stringer, Mary Mulenga, Dennis Musatwe,
interventions [40]. Masoso Chisala, Michael Lemba, Wilbroad Mutale, Peter Drobac, Felix
Ultimately, health systems are a means to an end – as Cyamatare Rwabukwisi, Lisa R. Hirschhorn, Agnes Binagwaho, Neil Gupta,
delivery platforms to ensure equitable access to high Fulgence Nkikabahizi, Anatole Manzi, Jeanine Condo, Didi Bertrand Farmer,
Bethany Hedt-Gauthier, Kenneth Sherr, Fatima Cuembelo, Catherine Michel,
quality, evidence-based health care, with the end goal Sarah Gimbel, Bradley Wagenaar, Catherine Henley, Marina Kariaganis, João
of improving the health of populations. However, in- Luis Manuel, Manuel Napua, and Alusio Pio.
vestments in health systems continue to be seen as
overly-complex ‘black boxes’ without clear evidence on FundingThe publication cost of this article was funded by the African Health Initiative
what works, and ‘black holes’ requiring substantial of the Doris Duke Charitable Foundation.
resource inputs (potentially at the expense of other
priorities) [5, 41]. The dearth of evidence on how Availability of data and materials
All data generated or analyzed during this study are included in this
complex interventions improve health system function- published article.
ing, and ultimately save people lives, reinforces this
perception [2, 27], underscoring the need to establish a About this supplement
This article has been published as part of BMC Health Services Research
core set of validated health systems indicators across Volume 17 Supplement 3, 2017: Implementation science as an essential
building blocks, and analytic approaches that explore driver for sustainable health systems strengthening interventions: Lessons
interaction across the building blocks and with the learned across the five-country African Health Initiative. The full contents of
the supplement are available online at https://bmchealthservres.biomedcen
outer context [42]. Validated measures and appropriate tral.com/articles/supplements/volume-17-supplement-3.
analytic techniques are essential to continue to build a
body of evidence on how to strengthen health systems, Authors’ contributions
and the potential benefits on improved health service All authors have read and approved the final manuscript.
coverage and population health impact; to establish targets Authors’ information
for health system strengthening; and ensure that substan- Kenneth Sherr, PhD, MPH; Quinhas Fernandes, MD, MPH; Almamy M Kanté,
tial resource investments through global health initiatives PhD; Ayaga Bawah, PhD, MA; Jeanine Condo, MD, MSc, PhD; Wilbroad
Mutale, PhD.
and national budget allocations are maximized.
Ethics approval and consent to participate
Conclusions Not applicable
The African Health Initiative was launched to meaning- Consent for publication
fully strengthen health systems, and to generate evidence Not applicable
on effective approaches to develop health systems that
Competing interests
lead to measurable improvements in health status. Impli- The authors declare that they have no competing interests.
cit in this objective is the ability to measure stronger
health systems, and associate these measures with Publisher’s Note
population-level health outcomes. Measuring health sys- Springer Nature remains neutral with regard to jurisdictional claims in
tem strengthening is complex, and while the WHO frame- published maps and institutional affiliations.
work is useful, it is not sufficient to describe how the parts Author details
function as a system. Innovative approaches to develop 1Department of Global Health, University of Washington, 1959 NE Pacific St,
health systems indicators, validated against health out- Seattle, WA, USA. 2Health Alliance International, Seattle, WA, USA. 3Ministry of
4
comes, are vital, and with the final results of the African Health, Maputo, Mozambique. Bloomberg School of Public Health, JohnsHopkins University, Baltimore, MD, USA. 5Regional Institute for Population
Health Initiative, some of these indicators will be available. Studies, University of Ghana, Accra, Ghana. 6School of Public Health,
University of Rwanda, Kigali, Rwanda. 7Department of Public Health,
Abbreviations University of Zambia School of Medicine, Lusaka, Zambia.
AHI: African Health Initiative; DDCF: Doris Duke Charitable Foundation;
HMIS: Health management information system; LMICs: Low- and middle Published: 21 December 2017
income- countries; NN: Neonatal; PHIT: Population Health Implementation
and Training Partnerships; WHO: World Health Organization References
1. Travis P, Bennett S, Haines A, Pang T, Bhutta Z, Hyder AA, Pielemeier NR,
Acknowledgements Mills A, Evans T. Overcoming health-systems constraints to achieve the
This work was supported by the African Health Initiative of the Doris Duke millennium development goals. Lancet. 2004;364:900–6.
Charitable Foundation. Kenneth Sherr was supported by a grant from the 2. De Savigny D, Adam T. Systems thinking for health systems strengthening.
Fogarty International Center at the US National Institutes of Health [grant Geneva: WHO; 2009.
number K02TW009207]. 3. Atun R, Weil DE, Eang MT, Mwakyusa D. Health-system strengthening and
We would also like to thank the members of the AHI PHIT Partnership tuberculosis control. Lancet. 2010;375:2169–78.
Collaborative for their contributions to this manuscript. Members include: 4. Kruk ME, Myers M, Varpilah ST, Dahn BT. What is a resilient health system?
Ahmed Hingora, Dominic Mboya, Amon Exavery, Kassimu Tani, Fatuma Lessons from Ebola. Lancet. 2015;385:1910–2.
Manzi, Senga Pemba, James Phillips, Almamy Malick Kante, Kate Ramsey, 5. Hafner T, Shiffman J. The emergence of global attention to health systems
Colin Baynes, John Koku Awoonor-Williams, Ayaga Bawah, Belinda Afriyie strengthening. Health Policy Plan. 2013;28:41–50.
Sherr et al. BMC Health Services Research 2017, 17(Suppl 3):827 Page 38 of 94
6. Institute For Health Metrics And Evaluation. In: IHME, editor. Financing 28. Pawson R, Tilley N. Realistic evaluation bloodlines. Am J Eval. 2001;22:317–24.
global health 2013: transition in an age of austerity. Seattle, WA: IHME; 2014. 29. Walton M. Expert views on applying complexity theory in evaluation:
7. Awoonor-Williams JK, Bawah AA, Nyonator FK, Asuru R, Oduro A, Ofosu A, opportunities and barriers. Evaluation. 2016;22:410–23.
Phillips JF. The Ghana essential health interventions program: a plausibility 30. Boyer C, Jackson E, Bawah A, Schmitt M, Awoonor-Williams J, Phillips J.
trial of the impact of health systems strengthening on maternal & child Estimating indices of health system readiness: an example from rural
survival. BMC Health Serv Res. 2013;13(Suppl 2):S3. northern Ghana. Lancet Glob Health. 3:S14.
8. Sherr K, Cuembelo F, Michel C, Gimbel S, Micek M, Kariaganis M, Pio A, 31. Jackson EF, Siddiqui A, Gutierrez H, Kante AM, Austin J, Phillips JF.
Manuel JL, Pfeiffer J, Gloyd S. Strengthening integrated primary health care Estimation of indices of health service readiness with a principal
in Sofala, Mozambique. BMC Health Serv Res. 2013;13(Suppl 2):S4. component analysis of the Tanzania service provision assessment survey.
9. Drobac PC, Basinga P, Condo J, Farmer PE, Finnegan KE, Hamon JK, BMC Health Serv Res. 2015;15:536.
Amoroso C, Hirschhorn LR, Kakoma JB, Lu C, Murangwa Y, Murray M, Ngabo 32. Mutale W, Godfrey-Fausset P, Mwanamwenge MT, Kasese N, Chintu N,
F, Rich M, Thomson D, Binagwaho A. Comprehensive and integrated district Balabanova D, Spicer N, Ayles H. Measuring health system strengthening:
health systems strengthening: the Rwanda population health implementation application of the balanced scorecard approach to rank the baseline
and training (PHIT) partnership. BMC Health Serv Res. 2013;13(Suppl 2):S5. performance of three rural districts in Zambia. PLoS One. 2013;8:e58650.
10. Ramsey K, Hingora A, Kante M, Jackson E, Exavery A, Pemba S, Manzi F, Baynes 33. Mutale W, Stringer J, Chintu N, Chilengi R, Mwanamwenge MT, Kasese N,
C, Helleringer S, Phillips JF. The Tanzania connect project: a cluster-randomized Balabanova D, Spicer N, Lewis J, Ayles H. Application of balanced scorecard
trial of the child survival impact of adding paid community health workers to an in the evaluation of a complex health system intervention: 12 months post
existing facility-focused health system. BMC Health Serv Res. 2013;13(Suppl 2):S6. intervention findings from the BHOMA intervention: a cluster randomised
11. Stringer JS, Chisembele-Taylor A, Chibwesha CJ, Chi HF, Ayles H, Manda H, trial in Zambia. PLoS One. 2014;9:e93977.
Mazimba W, Schuttner L, Sindano N, Williams FB, Chintu N, Chilengi R. 34. Gimbel S, Micek M, Lambdin B, Lara J, Karagianis M, Cuembelo F, Gloyd SS,
Protocol-driven primary care and community linkages to improve population Pfeiffer J, Sherr K. An assessment of routine primary care health information
health in rural Zambia: the better health outcomes through mentoring and system data quality in Sofala Province, Mozambique. Popul Health Metr.
assessment (BHOMA) project. BMC Health Serv Res. 2013;13(Suppl 2):S7. 2011;9:12.
12. Bryce J, Requejo JH, Moulton LH, Ram M, Black RE, Population Health, I. & 35. Wagenaar BH, Gimbel S, Hoek R, Pfeiffer J, Michel C, Manuel JL, Cuembelo F,
Training - Africa Health Initiative Data, C. A common evaluation framework Quembo T, Afonso P, Porthe V, Gloyd S, Sherr K. Effects of a health information
for the African health initiative. BMC Health Serv Res. 2013;13(Suppl 2):S10. system data quality intervention on concordance in Mozambique: time-series
13. World Health Organization. Everybody's business: strengthening health analyses from 2009-2012. Popul Health Metr. 2015;13:9.
systems to improve health outcomes: WHO’s framework for action. 36. United Nations Department Of Economic And Social Affairs. 2015. IAEG-SDG
Geneva: World Health Organization; 2007. Open Consultation on Green Indicators [Online]. Available: http://unstats.un.
14. World Health Organization. Global reference list of 100 Core health org/sdgs/iaeg-sdgs/open-consultation-2 [Accessed 28 Jan 2016].
indicators. Geneva: WHO; 2015. 37. Phillips JF, Sheff M, Boyer CB. The astronomy of Africa's health systems
15. World Health Organization. Monitoring the building blocks of the health literature during the MDG era: where are the systems clusters? Glob Health
system: a handbook of indicators and their measurement strategies. Sci Pract. 2015;3:482–502.
Geneva: World Health organization; 2010. 38. Alliance For Health Policy And Systems Research. Neglected health systems
16. World Health Organization Maximizing Positive Synergies Collaborative, G, research: governance and accountability. Geneva: World Health
Samb B, Evans T, Dybul M, Atun R, Moatti JP, Nishtar S, Wright A, Celletti F, Organization; 2008.
Hsu J, Kim JY, Brugha R, Russell A, Etienne C. An assessment of interactions 39. Balabanova D, Mills A, Conteh L, Akkazieva B, Banteyerga H, Dash U, Gilson
between global health initiatives and country health systems. Lancet. 2009; L, Harmer A, Ibraimova A, Islam Z, Kidanu A, Koehlmoos TP, Limwattananon
373:2137–69. S, Muraleedharan VR, Murzalieva G, Palafox B, Panichkriangkrai W,
17. Kruk ME, Freedman LP. Assessing health system performance in developing Patcharanarumol W, Penn-Kekana L, Powell-Jackson T, Tangcharoensathien
countries: a review of the literature. Health Policy. 2008;85:263–76. V, Mckee M. Good health at low cost 25 years on: lessons for the future of
18. Stuckler D, Basu S, Mckee M. Drivers of inequality in millennium health systems strengthening. Lancet. 2013;381:2118–33.
development goal progress: a statistical analysis. PLoS Med. 2010;7: 40. English M, Irimu G, Wamae A, Were F, Wasunna A, Fegan G, Peshu N. Health
e1000241. systems research in a low-income country: easier said than done. Arch Dis
19. Anand S, Barnighausen T. Human resources and health outcomes: cross- Child. 2008;93:540–4.
country econometric study. Lancet. 2004;364:1603–9. 41. Frenk J. The global health system: strengthening national health systems as
20. Muldoon KA, Galway LP, Nakajima M, Kanters S, Hogg RS, Bendavid E, Mills the next step for global progress. PLoS Med. 2010;7:e1000089.
EJ. Health system determinants of infant, child and maternal mortality: a 42. Gilson L, Hanson K, Sheikh K, Agyepong IA, Ssengooba F, Bennett S.
cross-sectional study of UN member countries. Glob Health. 2011;7:42. Building the field of health policy and systems research: social science
matters. PLoS Med. 2011;8:e1001079.
21. Wang L. Determinants of child mortality in LDCs: empirical findings from
demographic and health surveys. Health Policy. 2003;65:277–99.
22. Fernandes QF, Wagenaar BH, Anselmi L, Pfeiffer J, Gloyd S, Sherr K. Effects of
health-system strengthening on under-5, infant, and neonatal mortality: 11-
year provincial-level time-series analyses in Mozambique. Lancet Glob
Health. 2014;2:e468–77.
23. Masanja H, De Savigny D, Smithson P, Schellenberg J, John T, Mbuya C,
Upunda G, Boerma T, Victora C, Smith T, Mshinda H. Child survival gains in
Tanzania: analysis of data from demographic and health surveys. Lancet. Submit your next manuscript to BioMed Central 
2008;371:1276–83. and we will help you at every step:
24. Kumalija CJ, Perera S, Masanja H, Rubona J, Ipuge Y, Mboera L, Hosseinpoor
AR, Boerma T. Regional differences in intervention coverage and health •  We accept pre-submission inquiries 
system strength in Tanzania. PLoS One. 2015;10:e0142066. •  Our selector tool helps you to find the most relevant journal
25. Craveiro I, Dussault G. The impact of global health initiatives on the health
system in Angola. Glob Public Health. 2016;11(4):475 95. •  We provide round the clock customer support –
26. Scott V, Schaay N, Olckers P, Nqana N, Lehmann U, Gilson L. Exploring the nature •  Convenient online submission
of governance at the level of implementation for health system strengthening: •  Thorough peer review
the DIALHS experience. Health Policy Plan. 2014;29(Suppl 2):ii59–70. •  Inclusion in PubMed and all major indexing services 
27. Adam T, Hsu J, De Savigny D, Lavis JN, Rottingen JA, Bennett S. Evaluating
health systems strengthening interventions in low-income and middle- •  Maximum visibility for your research
income countries: are we asking the right questions? Health Policy Plan.
2012;27(Suppl 4):iv9–19. Submit your manuscript at
www.biomedcentral.com/submit