i PROCESS USE OF PROGRAMME EVALUATION IN THREE EVALUATION CONTEXTS IN NORTHERN GHANA By Bernard Afiik Akanpabadai AKANBANG (10325807) This thesis is submitted to the University of Ghana, Legon in partial fulfilment of the requirement for the award of PhD in Development Studies December, 2012 University of Ghana http://ugspace.ug.edu.gh ii Declaration I Bernard Afiik Akanpabadai AKANBANG, hereby declare that this thesis is the product of my effort under the guidance of my supervisors and that it has not been presented anywhere as a whole or in part for the award of a degree. All references have also been duly acknowledged. ..................................................................................................... Bernard Afiik Akanpabadai AKANBANG We, the undersigned certify that we supervised the thesis of Bernard Afiik Akanpabadai AKANBANG and approve the thesis for submission. …………………………….....…………………................................................... Dr Robert Darko Osei …………………………….....……………………................................................ Dr Paschal B. Atengdem ……………………………..…………………....................................................... Rev. Prof. S. N. Boapeah University of Ghana http://ugspace.ug.edu.gh iii Dedication This thesis is dedicated to the Glory of God. University of Ghana http://ugspace.ug.edu.gh iv Abstract Evaluation has been recognised as having the potential to contribute to the effectiveness of development interventions. Using a mixed method approach involving surveys and Q Methodology, the study examines evaluation processes and evaluation use particularly process use in evaluations in Northern Ghana; a zone in which the framework for monitoring and evaluation of development interventions developed by the National Development Planning Commission has been piloted. Evaluation capacity building, organisational learning/programme strengthening and hybrid, emerged as perspectives on process use types in evaluation. These process use types were influenced by the level of stakeholder involvement, the level of information flow in the evaluation process, the amount of resources committed to the evaluation and the organisational learning capacity of the programme. Stakeholder participation especially at the initial stages, employment of group and learning processes, and the competence and skills of the evaluator in facilitation emerged as the forms of stakeholder participation required to facilitate process use. Quarterly and annual reviews and the use of pre-departure debriefing meetings were found to be important to the use of evaluation. Constructivist learning was found to provide significant basis for explaining process use. The study recommends the profiling of ongoing evaluation processes as well as mandatory debriefing meetings in evaluation systems, and the delinking of mid-term evaluation from the decision on the fate of programmes, as ways by which programme evaluation can contribute to the effectiveness of programmes. University of Ghana http://ugspace.ug.edu.gh v Acknowledgement I register my sincere thanks to my supervisors: Dr. Robert Darko Osei (Principal Supervisor), Dr Paschal B. Atengdem and Rev. Prof. S.N. Boapeah for their guidance and encouragement. I thank the Director and Research Fellows of the Institute of Statistical Social and Economic Research for their useful contributions which have enriched my study. The programme coordinators and administrative assistant also deserve some commendation for the logistical support. The Director and Staff of the Centre for Development Research of the University of Bonn, Germany contributed significantly to shaping my proposal. I thank DAAD for offering the scholarship for my study. I wish to thank the Chief Economic Planning Officers of the three Northern Regions of Ghana and their staff, the Regional Directors and staff of the Community Water and Sanitation Agency in the Upper East and West Regions, the Regional Director of Health Services and his Deputies as well as staff of the coordinating office of Community Based Health Planning Services programme at the regional and district levels in the Upper West Region. My sincere thanks also go to the management and staff of the following District Assemblies: Builsa District Assembly, Kasena Nankana East District Assembly, Bongo District Assembly and Garu Tempani District Assembly all in the Upper East Region; the Wa Municipal Assembly, Jirapa District Assembly, Lawra District Assembly, and Nadowli District Assembly in the Upper West Region; and the Tamale Metropolitan Assembly, the Central Gonja District Assembly and the West Mumprusi District Assembly in the Northern Region. I owe lots of gratitude to my family, colleagues, and friends and loved ones. May the Good Lord richly bless you all, Amen. University of Ghana http://ugspace.ug.edu.gh vi Contents Declaration ................................................................................................................................... ii Dedication ................................................................................................................................... iii Abstract ....................................................................................................................................... iv Acknowledgement ...................................................................................................................... iv Tables ........................................................................................................................................... x Figures ........................................................................................................................................ xii Boxes ..........................................................................................................................................xiii Acronyms ................................................................................................................................... xiv CHAPTER ONE: INTRODUCTION AND BACKGROUND .......................................... 1 1.0 Role and Significance of Evaluation in Development Programmes ................................. 1 1.1 Evaluation Use in Development Programmes ................................................................. 5 1.2 Evaluation and the Programme Cycle in the Developing World ..................................... 8 1.3 Evaluation and Development Planning in Ghana .......................................................... 10 1.4 Problem Statement ....................................................................................................... 20 1.5 Research Questions and Objectives ............................................................................... 23 1.5.1 Research Questions ............................................................................................... 23 1.5.2 Research Objectives ............................................................................................... 24 1.6 Research Approach and Methodology .......................................................................... 25 1.7 Justification for the Study .............................................................................................. 26 1.8 Output of the Study ....................................................................................................... 27 1.9 Organization of the Report ............................................................................................ 27 CHAPTER TWO: THEORETICAL AND CONCEPTUAL FRAMEWORK ................ 28 2.0 Introduction ................................................................................................................... 28 2.1 Theoretical Framework for Evaluation .......................................................................... 28 2.2 Theoretical Framework for Process Use ........................................................................ 32 2.3 Description of Concepts ................................................................................................. 37 2.3.1 Development Programme ...................................................................................... 37 2.3.2 Development effectiveness ................................................................................... 38 2.3.3 Evaluation Inquiry .................................................................................................. 40 2.3.2 Evaluation Use/Consequences ............................................................................... 49 University of Ghana http://ugspace.ug.edu.gh vii 2.3.3 Process Use ............................................................................................................ 52 2.3.4 Factors Affecting Process Use ................................................................................ 55 2.3.5 Stakeholder Involvement in Evaluation ................................................................. 57 2.4 Conceptual Framework for Facilitating Process Use in Evaluation................................ 63 CHAPTER THREE: RESEARCH METHODOLOGY ................................................... 74 3.0 Introduction ................................................................................................................... 74 3.1 The Study Area ............................................................................................................... 74 3.2 Study Design .................................................................................................................. 78 3.3 Data Types and Sources ................................................................................................. 82 3.4 Choice of Sample ........................................................................................................... 82 3.5 Data Collection Methods ............................................................................................... 89 3.5.1 Survey Method ....................................................................................................... 90 3.5.2 In-depth Interviews ................................................................................................ 91 3.5.3 Q Methodology ...................................................................................................... 91 3.6 Data Processing ............................................................................................................ 107 3.7 Data Analysis ................................................................................................................ 107 CHAPTER FOUR: RESULTS AND DISCUSSIONS ................................................... 111 4.0 Introduction ................................................................................................................. 111 4.1 Evaluation Processes in Northern Ghana .................................................................... 111 4.1.1 Profile of Programmes and Respondents ............................................................ 111 4.1.2 Evaluation Status and purpose in Development Programmes ............................ 112 4.1.3 Evaluation Types in Programmes ......................................................................... 116 4.1.4 Funding of Evaluation .......................................................................................... 121 4.1.5 Main Evaluation Participants ............................................................................... 123 4.1.6 Evaluation Reporting ............................................................................................ 126 4.2 Programme Evaluation Use in Northern Ghana .......................................................... 128 4.2.1 Staff Expectations of Evaluation .......................................................................... 128 4.2.2 Examples of Uses of Evaluation ........................................................................... 131 4.2.3 Users of Evaluation .............................................................................................. 137 4.2.4 Factors Affecting the Use of Evaluation ............................................................... 140 4.3 Results and Discussions on Process Use Types in Evaluation ...................................... 147 4.3.1 Factor 1: Organisational Development /Programme Strengthening Perspective149 University of Ghana http://ugspace.ug.edu.gh viii 4.3.2 Factor 2: Integrative Perspective ......................................................................... 152 4.3.3 Factor 3: Evaluation Capacity Building Perspective ............................................. 154 4.3.4 Consensus Statements ......................................................................................... 157 4.4 Results and Discussions on Factors Affecting the Occurrence of Process Use ............ 161 4.4.1 Integrative Group ................................................................................................. 163 4.4.2. Resources Group .................................................................................................. 167 4.4.3 Information Flow and Evaluation Culture Group ................................................ 170 4.4.4 Consensus Statements ......................................................................................... 173 4.5 Results and Discussions on Forms of Stakeholder Involvement for Evaluation Process Use 177 4.5.1 Timing of Involvement and Information Flow Perspective .................................. 179 4.5.2 Who to Participate and How Group..................................................................... 184 5.5.3 Role and Competences of the Evaluator Group .................................................. 187 4.5.4 Perspectives Common to the Groups .................................................................. 189 CHAPTER 5: FINDINGS, CONCLUSIONS AND RECOMMENDATIONS ............. 191 5.1 Summary of Findings.................................................................................................... 191 5.1.1 Summary of Findings on Evaluation Processes .................................................... 191 5.1.2 Summary of Findings on the Uses of Evaluation in Programmes ........................ 193 5.1.3 Summary of Findings on Process Use Types in Evaluation .................................. 196 5.1.4 Summary of Findings on Factors Affecting Process Use ...................................... 198 5.1.5 Findings on Form of Stakeholder Participation Required for Process Use .......... 201 5.2 Conclusions .................................................................................................................. 203 5.2.1 Conclusions on Evaluation Processes .................................................................. 203 5.2.2 Conclusions on Uses of Evaluation ...................................................................... 204 5.2.3 Conclusions on Process Use Types in Evaluation ................................................. 205 5.2.4 Conclusions on Factors Affecting Process Use ..................................................... 205 5.2.5 Conclusions on Form of Stakeholder Participation Required for Process Use .... 206 5.3 Recommendations ....................................................................................................... 207 5.3.1 Recommendations on Evaluation Processes ....................................................... 207 5.3.2 Recommendations on Uses of Evaluation ........................................................... 208 5.3.3 Recommendations on Process Use Types in Evaluation ...................................... 209 5.3.4 Recommendations on Factors Affecting Process Use.......................................... 209 University of Ghana http://ugspace.ug.edu.gh ix 5.3.5 Recommendations on Form of Stakeholder Involvement Required for Process Use 209 5.4 Overall Conclusion ....................................................................................................... 210 List of References ..................................................................................................................... 217 Appendix 1: Data Collection Package ...................................................................................... 247 Appendix 2: Analytical Results for Research Question 3 ......................................................... 270 Appendix 3: Analytical Results for Research Question 4 ......................................................... 281 Appendix 4: Analytical Results for Research question 5 .......................................................... 302 University of Ghana http://ugspace.ug.edu.gh x Tables Table 3.1: Research Design and Methodological Framework .......................................... 80 Table 3.2: Example of Score Sheet of a Q Sort .............................................................. 105 Table 3.3: Example of a Completed Q Sort .................................................................... 106 Table 4.1: Profile of Programme Evaluations Analyzed ................................................ 111 Table 4.2: Organisational Background of Respondents ................................................. 113 Table 4.3: Disciplinary Backgrounds of Respondents .................................................... 113 Table 4.4: Status of Evaluation in Programmes.............................................................. 114 Table 4.5: Perception of Evaluation Participants of the Purpose of Evaluation ............. 114 Table 4.6: Types of Evaluation in Programmes .............................................................. 117 Table 4.7: Sources of Funds for Evaluation.................................................................... 122 Table 4.8: Proportion of Programme Funds that Go into Evaluation ............................. 123 Table 4.9: Areas of Staff Involvement in Evaluation ..................................................... 125 Table 4.10: Methods Used to Involve Participants in Evaluation .................................. 126 Table 4.11: Programme Staff Expectations of Evaluation .............................................. 128 Table 4.12: Accomplishments of Evaluation .................................................................. 132 Table 4.13: Concrete Benefits Derived from the Evaluation .......................................... 133 Table 4.14: Ways of Involving Stakeholders to Support Use ......................................... 140 Table 4.15: Three Factor Matrix for Research Question 3 ............................................. 148 Table 4.16: Correlations between Factor Scores for Research Question 3..................... 149 Table 4.17: Factor Characteristics for Research Question 3........................................... 149 Table 4.18: Summary of Statements Illustrating Organizational Learning Group ......... 150 Table 4.19: Statements Illustrating Integrative Perspective ........................................... 153 Table 4.20: Statements Illustrating Evaluation Capacity Building Perspective ............. 156 University of Ghana http://ugspace.ug.edu.gh xi Table 4.21: Statements in which there is Consensus ...................................................... 158 Table 4.22: Three Factor Matrix for Research Question 4 ............................................. 162 Table 4.23: Correlation between Factors for Research Question 4 ................................ 163 Table 4.24: Factor Characteristics for Research Question 4........................................... 163 Table 4.25: Statements Illustrating Factor 1 ................................................................... 164 Table 4.26: Statements Illustrating Factor 2 ................................................................... 168 Table 4.27: Distinguishing Statements for Factor 3 ....................................................... 171 Table 4.28: Consensus Statements for Research Question 4 .......................................... 174 Table 4.29: Factor Matrix for Research Question 5 ....................................................... 178 Table 4.30: Correlations between Factor Scores ............................................................ 179 Table 4.31: Factor Characteristics for Research Question 5........................................... 179 Table 4.32: Statements that Illustrate Factor 1 Perspective ............................................ 180 Table 4.32: Statements Illustrating Factor 2 Perspective ................................................ 185 Table 4.34: Statements Illustrating the Perspective of Factor 3 ..................................... 188 Table 4.35: Consensus Statements for Research Question 5 .......................................... 190 University of Ghana http://ugspace.ug.edu.gh xii Figures Figure 1.1: Stages in the Planning Cycle ............................................................................ 9 Figure 1.2: Institutional Arrangements for M&E in Ghana .............................................. 18 Figure 2.1: Conceptual Framework for Facilitating Process Use of Evaluation .............. 72 Figure 3.1: Study Area in National Context. .................................................................... 75 Figure 4.1: Types of evaluation - Internal Verses External ............................................ 119 Figure 4.2: Main Evaluation Participants ....................................................................... 124 Figure 4.3: Users of Evaluation ...................................................................................... 138 Figure 4.4: Types of Organisational Arrangements in Support of Evaluation Use ........ 144 Figure 4.5: Ways in which Evaluators can Promote Evaluation Use ............................. 144 University of Ghana http://ugspace.ug.edu.gh xiii Boxes Box 4.1: Types of Learning Expected by Respondents from Evaluation ....................... 129 Box 4.2: Respondents Comments that Elucidate their Perspective ................................ 151 Box 4.3: Quotes from Respondents on their Views on Statements 1, 2 and 6. .............. 159 Box 4.4: Expressions on the Importance of People Skills in Evaluation........................ 165 Box 4.5: Comments on Organisational and Leadership Support for Evaluation ............ 166 Box 4.6: Views on the Need for Early Stakeholder Involvement of Stakeholders ......... 181 Box 4.7: Need for Frankness, Openness and Information Flow in Evaluation .............. 183 Box 4.8: Importance of Timing in Evaluation ................................................................ 184 Box 4.9: Views on Following up and the Use of Focus Groups in Evaluation .............. 186 Box 4.10: Skills Required by Evaluators in order to Facilitate Process use ................... 189 University of Ghana http://ugspace.ug.edu.gh xiv Acronyms APRs Annual Progress Reports CHPS Community Based Health Planning Services CIDA Canadian International Development Agency CSOs Civil Society Organisations DAs District Assemblies DPCU District Planning Coordinating Unit DPMGs District Poverty Monitoring Groups DWAP Districtwide Assistance Project GPRS Ghana Poverty Reduction Strategy GSGDA Ghana Shared Growth and Development Agenda GSS Ghana Statistical Service IMF International Monetary Fund INGOs International Non Governmental Organisations JICA Japanese International Cooperation Agency MDGs Millennium Development Goals M&E Monitoring and Evaluation MMDAs Metropolitan, Municipal, District Assemblies MDAs Ministries Departments Agencies NDPC National Development Planning Commission NGOs Non Governmental Organisations NR Northern Region PPMEDs Policy Planning Monitoring and Evaluation Divisions RPCU Regional Planning Coordinating Unit RPMGs Regional Poverty Monitoring Groups SPSS Statistical Packages for Social Sciences STWSSP Small Towns Water Supply and Sanitation Project UER Upper East Region UWR Upper West Region University of Ghana http://ugspace.ug.edu.gh 1 CHAPTER ONE: INTRODUCTION AND BACKGROUND 1.0 Role and Significance of Evaluation in Development Programmes The Programme-Based Approach to development has gained increasing recognition in the search for ways of delivering overseas development assistance to developing countries (Holvoet & Renard, 2007; Paris Declaration, 2005). However, poorly conceived and executed projects and programmes (Bourguignon & Sundberg, 2007), failures in learning and accountability (Clements, Chianca and Sasaki, 2008; Wenar, 2006) continue to undermine progress in reducing poverty by international development programmes. As a result, actors in international development have been on the search for ways of ensuring effectiveness of programmes to enable them bring about the envisaged desirable changes (Kusek & Rist, 2001). Evaluation, defined as a systematic process through which organisations obtain information on their activities, impacts, and effectiveness in order to describe their accomplishments and improve upon their activities (Mark, Henry & Julnes, 2000; Mattesich, 2003) has been accepted as one of the ways of bringing effectiveness to development interventions (Clements et al., 2008; Holvoet & Renard, 2007; Kusak & Rist, 2001; Morra, Linda & Rist, 2009; Newcomer, Hatry & Wholey, 2010). The demand for programme evaluation, the subject matter of this study, which is defined as the application of systematic methods to respond to questions on programme operations and results through either ongoing monitoring of a programme and or one-shot studies of programme process or programme impacts (Newcomer et al., 2010; Rossi, Lipsey & Freeman, 2004) has seen phenomenal growth and interest (African University of Ghana http://ugspace.ug.edu.gh 2 Development Bank, Operations Evaluation Department, 2006; Andersen, Birchall, Jessen & Money, 2006; and Kusek & Rist, 2004 as cited in Leeuw & Furubo, 2008). Increasing need and demand for public accountability both downwards and upwards (African Development Bank, 2006; Ferraro & Pattanayak, 2006; Newcomer et al., 2010 ), the prominence given to evidence-based policy and the requirement for policy makers to develop prudent policies based upon a firm comprehension of the existing context (Davies et al., 2000; MacKenzie, Rivara, Jurkovich, Nathens, Frey, Egleston, Scharfstein, 2006; Martin & Sanderson, 1999), demand for results-based budgeting and management and iterative learning in line with the new international development paradigm’s principles (Engel, Carlsson & Zee, 2003; Holvoet & Renard, 2007), increased desire by programme designers and implementers to be prudent in their use of scarce funds in the light of thinly stretched budgets for programmes (Ferraro & Pattanayak, 2006) have contributed significantly to this growth in demand for programme evaluation. Generally, it has been noted in the literature (Bamberger & Eleanor, 1986; Conyers & Hills, 1984; Martin & Sanderson, 1999) that evaluation systems provide feedback on goal achievements; initial problems identification as well as proposition of solutions; measuring equity and efficiency; and providing guidelines for the planning of future programmes. A review of the poverty reduction strategy approach by the World Bank and International Monetary Fund (IMF) in September, 2005 underscored the centrality of a strong evaluation system for enhancing development effectiveness, reinforcing mutual accountabilities and realizing the Millennium Development Goals (MDGs) (African Development Bank, 2006). Similarly, Andersen et al. (2006) found that successful project management, regardless of cultural differences, depended on, in University of Ghana http://ugspace.ug.edu.gh 3 addition to professional planning and cost control, rich communication and learning from experience in order to achieve superior project success. Kusek & Rist (2004) Cited in Leeuw & Furubo (2008) in their contribution to the significance of evaluation articulated that building an evaluation system adds the fourth leg to the governance chair which traditionally consisted of budget systems, human resource systems, and auditing systems. Building a monitoring and evaluation system brings to the fore the outcomes and consequences of governmental actions. In response to the growing need for and importance of evaluation, evaluation units or divisions have been established in most organisations involved especially in international development. In the same way, evaluation is demanded as a requirement of programmes for accessing funds (Crawford & Bryce, 2003; Hatry et al., 2010; Morra et al., 2009) and account for about fifteen percent of programme funds. International donor agencies have therefore mostly been at the forefront of evaluation efforts in the developing world. As a result, most evaluations conducted in the developing world are geared towards the needs of these organisations. Thus, the timely and useful feedback needed by programme managers from evaluation in developing countries to enable the continuous improvement of quality, scale, access, equity, and impact is lacking in most evaluations (McDonald, 1999; Thoenig, 2000 as cited in Forss, Rebien & Carlsson, 2002). The Swedish International Development Agency, for instance, in an assessment of its evaluation practices, was disappointed to find that most stakeholders never saw the findings and that the few who did, found nothing very new or useful in them. The agency University of Ghana http://ugspace.ug.edu.gh 4 arrived at a conclusion that evaluation does not mean anything to the majority of programme staff and managers in the developing world (Bamberger, 2009). Alaimo (2008), Podems (2007) and Kusek and Rist ( 2001) had found earlier that evaluation was mostly used as a reporting requirement to funding institutions and remain overly centred on compliance with government requirements and regulations and on tracking of implementation of policies, programmes and projects, rather than end-results of policy, programme and project efforts. Besides, it has been observed that most evaluation processes are used as tools to cover up abuses and mistakes and to advocate or advertise for more funding (House, 2008; Lempert, 2010; Michaelowa & Borrmann, 2006). Evaluation has therefore generally had limited effect on programme performance and outcomes in the developing world (Bamberger, 2009, McDonald, 1999). As noted by Elbers, Gunning, Jan and de Hoop (2009), underutilisation of evaluation research as a method especially in Africa has accounted for the inability of evaluation to contribute effectively to learning and programme development. The drive behind this study is, therefore, to contribute to the development of evaluation research and practice in ways that enables it to contribute meaningfully to ensuring better performance of programmes designed and implemented to improve the living conditions of the poor. Hatry et al. (2010) for instance, have argued for learning and more importantly programme improvement as a cardinal attribute for an evaluation to be classified as being useful or merit the kind of investment put into it. This study is grounded in the argument of Hatry et al. (2010). The argument in this study is that, it is only when evaluation has a learning and improvement orientation that its potential as an avenue for programme strengthening, evidence-based implementation, replanning and University of Ghana http://ugspace.ug.edu.gh http://econpapers.repec.org/RAS/pgu94.htm http://econpapers.repec.org/RAS/pde689.htm 5 scaling-up of development programmes in resource-limited settings as noted by Nash, Elul, Rabkin, Tun, Saito, Becker, Nuwagaba-Biribonwoha (2009), as well as its opportunity for developing evaluation capacity of participants and organisations (King, 2007; Patton, 2008) can be realised. There is therefore the need to contribute to evaluation processes and purposes that go beyond merely fulfilling donor demands or external accountability to processes that enable managers to improve upon their programmes. This is even more important in the context of Northern Ghana, which in spite of the numerous poverty reduction interventions has still consistently been ranked as the three poorest regions in Ghana. 1.1 Evaluation Use in Development Programmes There has always been interest in knowing the contribution of evaluation to programme goals. Evaluation use has thus occupied a major part of the evaluation research and literature and it continues to be a major concern of development practitioners (Christie, 2007; Cousins et al., 2004; Henry & Mark, 2003; Johnson, Greensei, Toal, King, Lawrenz & Volkov, 2009; Smits & Champagne, 2008). Evaluators in particular are unanimous that, evaluation should be judged by their use, that is, in terms of its contribution to social betterment (Henry & Mark, 2003; Preskill & Caracelli, 1997). However, evaluation use has traditionally been limited to the use of findings and recommendations contained in evaluation reports (Forss et al., 2002; Weiss, 1998). Various studies in the 70s and 80s on evaluation use have, therefore, gone into determining the various ways in which evaluation findings were used. Since 1997, University of Ghana http://ugspace.ug.edu.gh 6 however, Patton (1997) coined the word process use as another type of evaluation use (Harner & Preskill, 2007; Henry & Mark. 2003; Patton, 1997, 2008) referring to it as the effects of evaluation that are not the result of evaluation findings or recommendations but the evaluation process in its right (Alkin & Taut 2003; Cousins et al., 2004, Preskill, Zuckerman & Matthews, 2003). Since then, a lot of interest has been shown in process use because of its potential to optimise the use of evaluation (Amo & Cousins, 2007; Harner & Preskill, 2007; King, 2007; Podems, 2007) mostly in the western world. Process use has been viewed as a long term measure for building organisational capacity for undertaking evaluation that is used (Cousins et al., 2004; King, 2007; Patton, 2008). Other potential benefits of process use have included its capacity to change management thinking about future options; mobilize staff around a course of action (Greene, 1988; Smits & Champagne, 2008; Turnbull, 1999); make evaluation to become part of the treatment of a development problem rather than just being an independent assessment of effects (Forss et al., 2002). To Podems (2007) and Forss et al. (2002), process use is the only means of guaranteeing the use of evaluation especially in circumstances where it is uncertain that the evaluation report will be used. It is thus, a cost effective way of strengthening the overall utility of an evaluation. Process use therefore, not only justifies the evaluation, but the time and resources that are not used directly to solve the development problem. It involves any useful learning for any person involved in the evaluation that takes place during an evaluation, planned or unplanned, intentional or unintentional, that is not directly related to the evaluation findings (Podems, 2007). It facilitates organizational learning (Amo & Cousins, 2007; Cousins et al., 2004) and builds evaluation capacity (Carden & Earl, University of Ghana http://ugspace.ug.edu.gh 7 2007; King, 2007; Preskill et al., 2003). Process use contributes to a learning phase that reinforces understanding and the ownership of results and, eventually, a greater sense of obligation to follow through on the results (Greene, 1988; Preskill et al., 2003; Smits & Champagne, 2008). In spite of the significance of process use to organisational learning, evaluation capacity building among others, various studies on the operationalisation of process use in empirical research note that process use has not been adequately put to empirical test even though it has been present in the conceptual literature on evaluation use for over a decade. Process use is noted to still be in its infancy requiring more research to enhance it (Amo & Cousins, 2007; Harnar & Preskill, 2007). Forss et al. (2002, p 43) made the following observation on the need to explore more into process use. ‘We do not doubt that process use exists, but we do not know how common it is... there is therefore the need to know more about process use, and to verify whether it occurs as often as some like us to believe it does.’ Similarly, Amo & Cousins (2007, p 20) noted that, ‘there is still much opportunity to study, question, probe, test and substantiate process use.’ This study explores process use in Northern Ghana as a way of finding out how it can be used as a catalyst for enhancing poverty reduction efforts in the area. University of Ghana http://ugspace.ug.edu.gh 8 1.2 Evaluation and the Programme Cycle in the Developing World One of the basic elements in the planning of development programmes is that, it is a cyclical process. The planning of programmes is described as a continuous cyclical process of decision making, involving a sequence of stages which are designed to link basic policy goals with specific programmes, projects and to ensure that lessons learned from the implementation of these programmes are fed into subsequent planning cycles (Conyers & Hills, 1984). According to Biggs and Smith (2003), the planning cycle is a rational way of conceptualizing and managing development interventions and has over the years since its introduction in the 1960s become the standard practice for development agencies to organize their activities using the planning/project cycle management framework. As shown in Figure 1.1, the planning cycle consists of a number of progressive stages that generally begin with needs identification and objectives formulation, through planning and implementation of activities, to assessment of the outcomes (Biggs & Smith, 2003). One significant attribute of the planning cycle is the centrality of learning in it (Biggs & Smith, 2003; Conyers & Hills, 1984). Learning is conceived as making adjustments during the planning cycle in response to ongoing events and taking account of past experience in future (Biggs & Smith, 2003) and is constituted /represented by the evaluation stage. Evaluation serves as a feedback mechanism/loop into the other stages of the planning cycle, thus ensuring its cyclical nature. The iterative nature of the planning process in which the goals of planning are achieved through a stepwise movement instead University of Ghana http://ugspace.ug.edu.gh 9 of a momentary leap from problem to solution highlights the inherent value of evaluation in the programme cycle. The emergence of evaluation as a specific stage of programme cycle is alluded to by Ocampo (2002) to have occurred in the developing world formally in the 1970s in response to the then development failures. Evaluation was viewed as a tool that will facilitate improvements in the planning and implementation of future programmes. By establishing a causal relationship among projected outcomes and results actually obtained, reasons for failures were explained and lessons generated from the reasons for failure were used to guide future planning and management of similar programmes (Ocampa, 2002). Figure 1.1: Stages in the Planning Cycle Goals formulation Appraisal Definition of alternative courses of action Implementation Problems definition/definiti on/identification Evaluation Adapted from Conyers and Hills (1984) The need to analyse and determine the causes of failure resulted in the incorporation of specific activities of information collection (monitoring), and periodic analyses (evaluation) during the programme life. During this period, the focus of University of Ghana http://ugspace.ug.edu.gh 10 evaluation was to diminish the chance of errors rather than learning from them. However, projects increasingly continued to fail to achieve predetermined objectives resulting in the emergence of adaptive planning. Adaptive planning focuses on adaptation, flexibility, and learning from experience. Evaluation within the framework of adaptive planning is an ongoing process and not a time bounded stage in the programme cycle, which provides feedback into programme implementation management and in the design of future programmes. 1.3 Evaluation and Development Planning in Ghana Ghana has a long history of development planning. The preparation of development plans have characterised the major government regimes of the country. The first ever plan to be prepared in the country’s development history was the ten years Guggisberg’s plan prepared in 1919 during the period of colonial rule. Several other plans including the 1930-40 plan, the 1946-56 plan, the 1951-56 plan, the 1958-59 consolidation plan, and the 1959-64 were all prepared during the period before the country attained a republican status. Upon the attainment of a republican status, various governments implemented development plans. The first republican government under the charismatic leadership of Nkrumah prepared a seven-year plan spanning 1964 to 1970. This plan had industrialization as the strategy for achieving national development. Domestic production of import-substitutes by state-owned enterprises formed the nucleus of the development plan. Under the second republic, a two-year rural development plan (1971-72) was prepared. Under the Provisional National Defense Council government, a five-year development plan (1981-1986) was developed. The focus of this plan was economic University of Ghana http://ugspace.ug.edu.gh 11 recovery under the support of the IMF and the World Bank. It is worth mentioning here that most of these plans had little involvement of the citizenry in their preparations (National Development Planning Commission (NDPC), 2010). The return of Ghana to constitutional rule in 1993 under the 1992 fourth republican constitution witnessed the preparation of a number of development plans in response to Article 36 (5) of the Constitution which requires that the president presents to parliament a coordinated programme of economic and social development of the country within two years on assumption of office. In 1995, the Government under the leadership of President Rawlings presented to Parliament the coordinated programme of economic and social development policies with the theme ‘Ghana: Vision 2020.’ The overall aim of Vision 2020 was to take the country to a middle income status. In line with Ghana vision 2020, the second government of the fourth republican constitution under President Rawlings presented a Medium-Term Development Plan (1997-2000) to guide the socioeconomic development of the country. This development plan had six thematic areas of focus: human development, economic growth, rural development, urban development, infrastructure development, and an enabling environment. With the change of government from one political party to another political party in 2001, the Ghana Vision 2020 plan was truncated and replaced by the Ghana Poverty Reduction Strategy (GPRS I, 2003-2005) and the Growth and Poverty Reduction Strategy (GPRS II, 2006-2009) which were both implemented under the third and fourth parliaments of the fourth republican constitution under president Kufour. GPRS I was in line with Heavily Indebted Poor Countries debt relief initiative in 2002 supported by the IMF and the World Bank. The programme sought to restore macroeconomic stability and University of Ghana http://ugspace.ug.edu.gh 12 reduce the incidence of poverty and had four thematic areas: production and gainful employment, human resource development and basic services, special programmes for the poor and vulnerable, and governance. Five areas under these four themes were profiled for action: infrastructure, rural development based on modernized agriculture, enhanced social services, good governance, and private sector development. The GPRS-II placed emphasis on growth as the basis for sustained poverty reduction. It had the objective of accelerating the pace at which Ghana could attain a middle income status. Macroeconomic stability, private sector competitiveness, human resource development, and good governance and civic responsibility were the thematic areas of GPRS II. Both GPRS I and GPRS II contributed significantly to guiding the allocation of resources and also provided a platform for dialogue between the Government and the development partners, and mainstreamed the MDGs and other international commitments relevant to poverty reduction into the national development effort. In 2009, Ghana witnessed yet another peaceful change of power from a ruling government to an opposition political party. As is required of government, a developed plan dubbed Ghana Shared Growth and Development Agenda (GSGDA) (2010-2013) was launched. The thematic focus of the GSGDA include; human development, transparent and accountable governance, infrastructural development, agricultural modernization, oil and gas development, and private sector development. All the plans prepared upon the return of the country to constitutional democracy in 1993 were guided by two Acts of Parliament; the National Development Planning (System) Act, 1994 (Act 480) and the accompanying National Development Planning University of Ghana http://ugspace.ug.edu.gh 13 Commission Act, 1994 (Act 479) which sought to democratize planning practice. A hybrid of planning systems; centralised planning, functional or sectoral planning and decentralised planning are recognised by the two Acts. The central planning agency (NDPC) under the National Development Planning (Systems) Act, 1994 (Act 480), coordinates the planning activities of the various functional ministries and the MMDAs who undertake detailed planning and implementation of programmes at the sector and district levels respectfully. The Regional Planning Coordinating Unit (RPCU) harmonizes the plans prepared by the MMDAs. They also have oversight role of the MMDAs as well as providing capacity development support to the MMDAs. Under Act 462, MMDAs have been established as planning authorities with responsibility for planning the overall development of the areas under their jurisdiction. The District Planning Coordinating Unit (DPCU) coordinates the preparation of development plans as well as their implementation and evaluation. Planning at the district level commences with the identification of community problems, opportunities, goals and objectives. These are collated at the various levels of the local government structure (unit commit, area/town/urban council) until the district level is reached. At the district level, the sub-committees of the executive committee of the DA examine the issues emanating from the lower levels and prioritise the goals and objectives of the people. The prioritised areas are taken up by the departments of the MMDAs as well as other organisaations as NGOs into their sectoral plans. The District Planning Coordinating Unit (DPCU) integrates and co-ordinates district sectoral plans into medium term, short term plans and annual plans and budget for consideration by the executive committee and debate by the Assembly. The district plan once approved is University of Ghana http://ugspace.ug.edu.gh 14 presented to the RPCU for co-ordination and harmonization with the plans of the other MMDAs in the region (Bandie, 2007). Steps involved in implementing the prepared plans include; plan review, plan adoption, annual action planning and monitoring and evaluation. A major limitation in development planning in Ghana is the lack of integration between spatial and human settlement planning which has resulted in uneven and unplanned spatial development in many parts of the country, especially selected urban centers (NDPC, 2010). Another characteristic of development planning is the lack of continuity in the implementation of development plans when there is a regime change. This has accounted for the several development plans that have been produced in the history of the country. The participation of the citizenry in the decision making process especially the vulnerable groups, even though, attempts have been and continue to be made to involve a broad range of stakeholders including civil society groups, political parties in the policy formulation process is inadequate (NDPC, 2010). The absence of a legislative instrument for enforcement of compliance with Act 480; lack of clarity in the roles and responsibilities and functional relationships between the key players and stakeholders resulting sometimes in overlaps and or conflicts; treatment of planning and budgeting as separate activities undertaken by different institutions resulting in a weak linkage between national plans and national budgets continue to affect the effectiveness of planning in the country (NDPC, 2010). University of Ghana http://ugspace.ug.edu.gh 15 Even though, evaluation has always been central to development planning in Ghana, limited attention was paid to the issue of evaluation until the advent of poverty reduction strategy papers (NDPC, 2010). Along with the poverty reduction strategy papers is the requirement that Annual progress reports (APRs) should be prepared. These APRs are prepared every three years. In line with the preparation of the APRs, the NDPC developed a framework (GhanaInfo) for evaluation of poverty reduction in the country. This framework is a comprehensive national socio-economic database made up of suitable indicators adopted for planning and policy formulation. This framework was launched by NDPC and Ghana Statistical Service (GSS) in July 2005 (IMF, 2009). The objective of the comprehensive monitoring and evaluation (M&E) framework is to serve as basis for capturing data and information for informing national development planning; supporting sector policy and programme design; informing the budget allocation process; enhancing transparency and accountability; encouraging continuous improvement in public policy management; as well as policy dialogue within Government and with civil society organisations (CSOs) and development partners (NDPC, 2010). The National approach to evaluation is anchored on a participatory monitoring. Integral to this approach is ensuring high awareness and participation by citizenry in the evaluation process. Citizenry service providers’ interfaces as part of the participatory process is to ensure that citizens provide feedback to service providers in order to improve upon service delivery. Use of evaluation for the preparation of new phases of programmes as well as future programming generally is also integral to the evaluation strategy. University of Ghana http://ugspace.ug.edu.gh 16 As a way of anchoring evaluation as an agenda for national development, guidelines for district evaluation plans have been developed and issued by the NDPC in 2006 to assist the District Assemblies to develop their own district evaluation plans. The following have been outlined as objectives of the district evaluation guidelines: to organise and coordinate district level evaluation within the national evaluation system; to facilitate results based programming and management; to monitor the sustainability of benefits and outcomes of development programmes; to promote and ensure both downward and upward accountability for use of resources; and to promote lesson learning and sharing among MMDAs (IMF, 2009). The interest and adoption of a decentralized evaluation system is informed by the recognition that a decentralized evaluation system provides district authorities, the government, development partners and the general public with a better means for learning from past experience, improving service delivery, planning and allocating resources, and demonstrating results as part of accountability to key stakeholders (Jackson, 2004). Key aspects of the decentralized M&E system include: definition of 20 core indicators and provision for district specific indicators; utilization of PlanInfo to support NDPC’s evaluation process. PlanInfo has the ability to create and maintain district profiles; establishment of Regional Poverty Monitoring Groups (RPMGs) to coordinate the monitoring and evaluation process. RPMGs have the responsibility of collating data from the districts, supporting capacity building on monitoring and evaluation, review of evaluation data in order to verify inconsistencies, ensure adherence to gender based analysis, and holding of bi-annual workshop involving all MMDAs for cross district University of Ghana http://ugspace.ug.edu.gh 17 GPRS review and policy recommendations among others. The process also includes the establishment of District Poverty Monitoring Groups (DPMGs). The DPMGs collect and reviews data on poverty indicators, as well as conferring with the RPMGs to agree on goals, indicators and targets. They are also responsible for the monitoring and evaluation of the progress of projects and programmes in the districts, and to use the outcome of evaluation as a key input into the policy development process (IMF, 2009; Dery, 2004). Proposals have been made to extend the evaluation efforts especially the citizens’ engagement with service providers/dissemination workshops to the towns and villages. Similarly, the need for monitoring and evaluation information to be captured in formats that allow for easy digestion and use by the citizenry have been recognised. Efforts at improving evaluation capacity have included; training of the staff of key departments as the policy planning monitoring and evaluation departments, extension of the GhanaInfo to the regions, and workshops on general data collection and statistical literacy. Figure 1.2 shows the institutional arrangements for monitoring and evaluation of development in the country. NDPC is the institution with the legal mandate to coordinate the decentralized M&E system while the GSS is responsible for the production of statistics. The National Development Planning Systems Act, 1994, Act 480 defines clearly the roles and responsibilities of NDPC, policy planning monitoring and evaluation divisions (PPMEDs), RPCUs and DPCUs. University of Ghana http://ugspace.ug.edu.gh 18 Figure 1.2: Institutional Arrangements for M&E in Ghana Institutional arrangements for monitoring and evaluation MDA MDA Regional Director MDA District Director Information flow & feedback NDPC President RPCU DPCU M&E Division of NDPC PPMEDs Cross Sectoral working Groups CSOs, Private Sector Actors Etc RPCU Members Other sector agencies Representatives of CSOs Private Sector Actors Etc DPCU Members Other sector agencies Representatives of TAs & CSOs Private Sector Actors Etc Key actors Roles Preparation of guidelines and training manuals Build M&E Capacity Creation of conditions for effective M&E Institution of rewards and sanctions Support districts to develop & implement M&E plans Coordinate district M&E activities Prepare regional APRs etc Preparation of district M&E plans Data collection and analysis Preparation of district APRs Parliament Source: NDPC (2010) Progress so far made since the profiling of monitoring and evaluation in 2003 include: increase in the percentage of indicators reported on from 62% in 2005 to 70% in 2006 and 87% in 2007; all MMDAs prepare their own M&E plans while about 65% of district assemblies now prepare Annual Progress Reports (APRs) on the implementation of their District Medium-Term Development Plans based on their M&E Plans; conduct and production of poverty and social impact analysis reports as well as citizens’ assessment reports covering decentralisation, modernised agriculture, vulnerability and exclusion. Similarly, GSS continuous to generate relevant data for national level monitoring including the monthly Consumer Price Index, Producer Price Index as well as Ghana Living Standards Survey and Core Welfare Indicators Questionnaire survey reports. The GSS also launched its Corporate Plan and the Ghana Statistical Development University of Ghana http://ugspace.ug.edu.gh 19 Plan aimed at improving the national statistical system over a five year period (2009 - 2013), and aligning the development of the country’s statistical system with the reporting requirements of national development programmes. Key challenges confronting evaluation in the country include: the differences in reporting cycles by ministries, departments and agencies (MDAs), and MMDAs; the inadequacy of updated data in general and the unavailability of established baselines and targets, uncoordinated nature of data collection with different formats for the different donor-assisted development programmes at the district level, and the generally limited use of data for sub-national level planning. There is also a weak demand for M&E at the district level (IMF, 2009). The District Capacity Building Project (DISCAP) is one of the projects that have been at the forefront of evaluation especially in the three Northern Regions of Ghana. DISCAP has been working with RPCUs in the three Northern Regions to develop a decentralized evaluation system that is based on the NDPC guidelines. DISCAP’s efforts which were anchored on the NDPC guidelines for decentralized monitoring and evaluation involves engaging MMDAs, MDAs, and communities to explore the impact of poverty reduction programmes, thus making the three Northern Regions a fertile area for exploring evaluation use at the sub-national level in Ghana. University of Ghana http://ugspace.ug.edu.gh 20 1.4 Problem Statement Process use emerged strongly as one of the ways of using programme evaluation to facilitate development effectiveness and poverty reduction. Process use is explored in Northern Ghana as a way of contributing to the development of a zone which is synonymous to poverty. Patton (1997, 1998, and 2008) revealed that, the actual process of preparing for and being involved in an evaluation can be very helpful in many ways to the participants, programmes and their organisations. These various helpful ways have been appropriately labelled by Patton (1997), as process use. Following Patton’s (1997) work, Forss et al. (2002) explored further process use using two specific case studies; the Nordic Development Fund and Swedish Museums. They interviewed staff about the benefits and problems concerned with their involvement in the evaluation to find out whether this was a worthwhile activity, or whether the evaluation would be equally effective if out-sourced. A careful analysis was carried out to ascertain what the staff members actually learned from the evaluation, and what benefits resulted from their involvement. There was a deliberate attempt to verify whether the patterns of process use suggested by Patton (1997) were present or not. They concluded that in addition to the generally accepted notion that evaluation, through feedback processes, adds knowledge to organizational/programme development after the evaluation, there is in fact much that is useful that takes place during the evaluation process, and that process use make up an equally important aspect of evaluation utility as the use of findings and recommendations. Hence, they found it imperative for more work to be undertaken on process use in order to know more about it and to verify its occurrence. University of Ghana http://ugspace.ug.edu.gh 21 Recently, Amo and Cousins (2007) examined the operationalisation of process use in the evaluation use literature as basis for setting the agenda for research into process use. They reviewed 18 empirical research studies published in journals. They identified three broad types of process use from the review of the 18 studies; learning, changes in attitude, and others (including social justice, opportunity, and networking). They also found out that the occurrence of process use was mainly through observation by evaluators and self reported statements of stakeholders captured by evaluators. These were found to be adequate in documenting the occurrence of process use but not enough to quantify the occurrence of process use by type, depth, range and amount or of its non occurrence. They therefore called for further research on process use to entrench it within the evaluation use literature. Harnar and Preskill (2007) pursuing the same objective as Amo and Cousins (2007) undertook a study to find out what evaluators knew about and define as process use. They found out from their question that sought multiple responses that, a large proportion of respondents (39%) view process use as related to stakeholder involvement in the evaluation process. Fifty-seven percent perceive process use as what happens during the evaluation process while 34% view process use as an outcome of evaluation process manifested in changes in perspective about their programmes, changes in attitudes towards evaluation, or stakeholders making programmatic improvements. While they acknowledged that changes in individuals, groups and organisations occur as a result of their participation in the evaluation, they conceded that it was not often clear what variables led to the changes in stakeholders’ behaviour and thinking as a result of their engagement in the evaluation. Besides, it was also not often clear, what level of depth, University of Ghana http://ugspace.ug.edu.gh 22 scope and breadth of stakeholder involvement was required to bring about the changes in the behaviour of people and their organisations. This study is designed and implemented taking into consideration the various gaps the earlier studies on process use have identified in the process use literature which were: limited contribution to the subject matter from a southern perspective, limited exploration of the perspective of the demand side (programme implementers), and limited knowledge on the form of stakeholder participation required to facilitate the occurrence of process use. The study captures programme implementers’ experiences of process use across three evaluations in Northern Ghana in order to compliment the literature which has mainly reflected the views of evaluators and mainly a northern perspective. Apart from contributing to filling the knowledge gap in the global literature, the study will particularly contribute to knowledge on evaluation in Ghana. Since the advent of the 1992 constitution, Ghana has been implementing a decentralised planning system. Over twenty years down the lane, it is still observed that demand for evaluation and use of evaluation at the local level is still low, even though, different forms of uncoordinated monitoring and evaluation systems go on at this level. Besides little systematic efforts have gone into looking at what evaluation processes are used in Ghana and what evaluation brings to programmes, organisations and the programme beneficiaries at large. The case of Northern Ghana is even more urgent because, there is the need to know what is being done right and wrong in terms of evaluation so that evaluation can contribute significantly to alleviating poverty in the area, which has so far proved insurmountable. Thus the problem this study investigated is that, in spite of the fact that the programme approach remains a major path for development in Ghana and Northern University of Ghana http://ugspace.ug.edu.gh 23 Ghana in particular, and that most programmes undertake evaluation, little research has been done on evaluation processes and evaluation use in Northern Ghana. Process use in particular which is the substantive issue investigated in the study has not been subjected to much empirical examination especially from a Ghanaian perspective and from the perspective of programme implementers. The study seeks to bring to the fore the actual behaviour of decision-makers, their abilities and willingness to use evaluation, which have largely been neglected in the utilization of evaluation research (Hyyrylainen & Viinamaki, 2008). 1.5 Research Questions and Objectives This section presents the research questions and objectives addressed by the study. A Research Question is a statement that identifies the phenomenon to be studied (Kishore, Vasundhra & Anand, 2011). A research question must be anchored in the current knowledge base; must be easy to formulate clear operational definitions of the variables involved, and clear hypotheses about the relationships between the variables; and it must elicit sufficient information to enable it contribute to theory and practice. 1.5.1 Research Questions The study addressed the following research questions from the perspective of programme implementers of development programmes: • What programme evaluation processes are used in development programmes? • What are the various ways in which programme evaluation is used? • What different types of process use emerge when programme implementers reflect on their participation in evaluation of development programmes? University of Ghana http://ugspace.ug.edu.gh 24 • What factors affect the occurrence of process use in the evaluation of programmes? • What stakeholder involvement is required to realize process use in the evaluation of programmes? 1.5.2 Research Objectives This study is about the use of evaluation. Evaluation has been widely used as a mechanism or a tool for assessing the effectiveness of development programmes and learning from the implementation of programmes. However and even with the emergence of adaptive planning, not much reflection and attention has been given to the use of evaluation in terms of evaluation’s contribution to programme improvement and organisational as well as individual learning, especially from a Ghanaian context. The purpose of this study is to examine the various ways in which evaluation is used and the factors affecting the use of evaluation from the perspective of programme/project implementers in a developing country context, with special reference to Northern Ghana. The study seeks to achieve the following specific objectives:  To document the evaluation processes used in development programmes and analyse their implications for the use of evaluation.  To examine the various uses to which programme evaluation is put.  To identify the different types of process use that stakeholders experience in evaluation.  To identify the factors that influence process use occurrence  To determine the nature of stakeholder involvement required to realize process use occurrence in evaluation. University of Ghana http://ugspace.ug.edu.gh 25  To identify ways of using process use as a medium for enhancing the effectiveness of evaluation in development programmes. 1.6 Research Approach and Methodology Research methodology is defined as the systematic way of solving a research problem. Inherent in research methodology are the various steps that are generally adopted by a researcher in studying his research problem along with the logic behind them. In this study, a mixed method approach was used. This approach included an institutional survey, Q Methodology and in-depth interviews. Q Methodology was the main methodology used to explore the substantive issue of the research. Q Methodology is a mixed methodological approach that combines the strengths of quantitative and qualitative research methodologies in order to allow for the subjective realities of participants involved in a study to be revealed (Amin, 2000; Ward, 2009). At the point of data collection, qualitative methodologies are used while quantitative approaches are used to do the analysis. The institutional survey was carried out to determine the evaluation processes and the various ways in which evaluation was used in development programmes. Besides providing information for addressing the first two research questions, the information from the survey formed the basis for the selection of the three programmes in which Q Methodology was used to address the three research questions on process use of evaluation. In-depth interviews were conducted with programme managers in order to provide insight to the analytical results generated from the survey and Q Methodology. Details on how these methods were used are provided in Chapter Three of the thesis. University of Ghana http://ugspace.ug.edu.gh 26 1.7 Justification for the Study The justification for the study lies in the fact that evaluation research in Africa is generally low, even though, a significant proportion of programme evaluation is conducted in Africa. Evaluation research is of essence if evaluation is to be able to support development programmes to impact positively on the lives of programme beneficiaries. In this regard, this study is seen as setting the pace for such an effort in a context that is confronted with endemic poverty. The focus on programme implementers (demand side of evaluation) as subjects of the study is expected to bring a new perspective to the discussion on process which has largely reflected the perspectives of evaluators (supply side of evaluation). By bringing the perspective of the demand side to bear on process use, it will allow for the fitting of evaluation findings into the users’ existing construction of reality while the expertise that they bring will culminate in the conduct of evaluation in ways that it is used. The focus on programme implementers is also expected to bring credibility and triangulation into the process use discussion. The study will also augment the scant literature on factors affecting process use especially the forms of stakeholder participation required to facilitate process use. This will help in the design of evaluation that will be used and contribute to individual as well as organisational learning and programme enhancement. The study also brings the empirical dimension to the process use literature which has generally been scant or remained at the conceptual level. The findings of the study will thus provide empirical basis for substantiating process use or otherwise. University of Ghana http://ugspace.ug.edu.gh 27 1.8 Output of the Study The output of this study is a set of proposals that will enhance the conduct of evaluation in development programmes in ways that enables it to contribute to learning, programme strengthening and ultimately poverty reduction. Specifically, the study addressed the following issues; getting stakeholders to participate and own evaluation; organisational arrangements required for mainstreaming learning; and evaluation methods, skills, and competencies required for designing and conducting an effective evaluation. 1.9 Organization of the Report The report has been organized into five chapters. Chapter One consists of the introduction and background to the research while Chapter Two covers the theoretical and conceptual framework for the study. Chapter Three outlines the research methodology while Chapter Four presents and discusses the output of the analysis. Chapter Five contains the summary of findings, conclusion and recommendations of the study. University of Ghana http://ugspace.ug.edu.gh 28 CHAPTER TWO: THEORETICAL AND CONCEPTUAL FRAMEWORK 2.0 Introduction Theory and practice have been acknowledged to be intricately related. In the view of Gaffney and Anderson (1991) as cited in Savery and Duffy (2001), there’s nothing as practical as good theory just as there is nothing theoretically interesting as good practice. Consequently, in this chapter, an attempt is made to present process use within its theoretical context. The theory that guides the learning process in evaluation and the guiding principles for the occurrence of learning are discussed. This is followed by a description of the key concepts inherent in the study as basis for the development of a conceptual framework to guide the conduct of the study. 2.1 Theoretical Framework for Evaluation The theoretical framework is the structure that supports a research study. It describes the theory which explains why the research problem under study exists. A theoretical framework may consist of concepts, together with their definitions, and existing theory or theories that guide or regulate the topic under investigation. The theoretical framework specifies the general relationship between variables in a certain phenomena while the conceptual framework specifies the relationship between specific variables in a given study. Evaluation cuts across many disciplines. Thus its theoretical framework could be traced to several theories. In planning, evaluation is an integral part of the planning process and therefore has roots in procedural planning theories as incrementalism and advocacy. It is also linked to the new public management reforms where greater need is University of Ghana http://ugspace.ug.edu.gh 29 placed on evidenced based decision making and on transparency and accountability for the use of public resources. Realist and theories of change have also guided the conduct of evaluation in the development process. Incrementalist Planning Theory Increamentalism emerged in the 1960 as a procedural planning theory by Lindblom (1959) as a criticism of rational comprehensive planning theory. Lindblom’s (1959) argument is that political leaders cannot agree on goals in advance as the rational model requires. To Lindblom (1959), they have a preference for policies and goals at the same time. The rational comprehensive theory’s preoccupation with the comparison of all possible alternatives and their comprehensive assessment in the view of Lidblom (1959) is not practicable. Lindblom (1959) proposed incrementalism which involves the simultaneous selection of goals and policies, concern for alternatives only marginally different from the existing situation, and limited comparisons among alternatives before making a choice. Incremetalism also has preference for results of social experimentation over theory as the basis of analysis. Increamentalism suggests that planning should proceed from limited knowledge of a phenomenon while making room for needed amendments to be made in the process as and when necessary to ensure that the goals of planning are achieved. The space provided in incrementalism for adjustment to decision making lends itself strongly to evaluation. University of Ghana http://ugspace.ug.edu.gh 30 Advocacy Planning Theory Evaluation theory is also anchored in advocacy planning theory which has its roots in Davidoff's (1965) article, advocacy and pluralism in planning. Davidoff (1965) called for the physical location of advocate planners in poor neighbourhoods in order to represent the interests of neighbourhood residents in city-level planning processes. Along the principles of the legal profession, Davidoff (1965) thought neighbourhoods would arrange their own advocates or the city government would appoint advocates to represent the neighbourhood. Public interest is a main strength or area of concern to advocacy planning (Hudson, 1979). The planner is viewed as someone who collects and processes information for the purposes informing policy makers of the effect of policies on especially the poor and making a strong case for policies that support the poor and underprivileged in society. Thus the planner is viewed as someone who monitors and evaluates policies as basis for informing policy. New Performance Management Model Evaluation is also associated with the New Performance Management Model which has accountability as a cardinal objective. According to Day and Klein (1987), accountability involves both the political justification of decisions and actions, and managerial answerability for implementation of agreed tasks according to agreed criteria of performance. Political accountability is concerned about those with authority being directly or indirectly answerable for their actions to the citizens. On the other hand, managerial accountability is focused on making those with delegated authority responsible for carrying out agreed tasks according to agreed criteria of performance. The interest in accountability within public sector reform is a desire to make public sector University of Ghana http://ugspace.ug.edu.gh 31 staff more accountable for their decisions and actions. Evaluation as well as other instruments such as Citizens Charters has emerged as ways of assuring accountability within the framework of the new performance management model (Economic Commission on Africa, 2003). Increased transparency and explicit performance targets in the view of Economic Commission on Africa (2003) are further steps towards better democratic control and accountability of the bureaucracy. It is a means of getting results from individuals, teams and the organizations at large, and allows for the development of indicators against which performance can be measured. Theories of Change Evaluation involves assessing programme theory. Programme theory assessment is in two parts; assessment of level of articulation of programme objectives to warrant the specified effects as one aspect and programme process; the extent of implementation of the programme to have a reasonable chance of producing the intended effects as the other aspect (Roosi et al., 2004). Theories of change (Connell et al., 1995), is one of the theories for assessing programme theory. The underpinning principle in theories of change is that programmes are iterative sequences of theories, that is, if action ‘A’ is undertaken, it is expected to achieve a certain quantity of output ‘B’, and when ‘B’ is in place we will be in a position to attempt ‘C’, which will then enable the next output ‘D’, and so on to be achieved. Evaluation consists of setting a microscope to each stage and making observations to see if the theories conform to actuality. University of Ghana http://ugspace.ug.edu.gh 32 Realist Theory Realist theory has been articulated by Pawson and Tilley (1997). The central principle in realist theory is that interventions work when the resources such as material, cognitive, social or emotional reach an agreement with programme subjects. This pathway from resource to reasoning is referred to as the programme ‘mechanism’. Realist evaluation research is thus primarily about detecting and inspecting vital programme mechanisms. 2.2 Theoretical Framework for Process Use Evaluation tries to discover whether programmes work and what can be done to enhance their performance. Learning is thus intrinsic to evaluation and provides the theoretical framework inherent in the learning that occurs in programme evaluation (Amo & Cousins, 2007; Harner & Preskill, 2007; Smits & Champagne, 2008). According to van der Knaap (2004), learning in evaluation refers to the processes of knowledge generation that result in an enhanced understanding. To him, the learning inherent in an evaluation is based on the principle that, there can always be improvement. From the perspective of van der Knaap (2004), evaluation as learning activity is viewed as a mechanism that supports programmes to perform to their optimum. One of the ways in which evaluation supports programmes to perform that has engaged the attention of researchers as well as practitioners and policy makers, is process use. Process use argues that the evaluation process provides a platform for the exchange of information and sharing of experiences which benefits the implementation of the ongoing programme as well as improvement in the entire organisational capacity (Amo & Cousins, 2007; Forss et al., 2002; Harnar & Preskill, 2007; Patton, 2008). Baptist (2010), University of Ghana http://ugspace.ug.edu.gh 33 Lempert (2010) and Amo and Cousins (2007) all root this type of learning (process use) in constructivist learning theory. Consequently, constructivist-learning theory has been the main theory that has served as the theoretical basis for the discussions on process use of evaluation in this study. It should however be noted that constructivism is not the only theory that guides social learning. Other learning theories as behaviourism which views knowledge as a repertoire of behavioural responses to environmental stimuli; cognitivism, humanism and design-based exist. Constructivists perceive learning as a social process in which learners generate knowledge and meaning through the interaction that takes place between their experiences and ideas, and knowledge that they come into contact with (Bandura, 1986, cited in Amo & Cousins, 2007; Resnick, 1989, cited in Richardson, 2003). To constructivists, there is nothing as knowledge independent of the knower. They perceive knowledge as something that is constructed in the process of learning. Similarly, learning is not conceived as the accumulation of facts or understanding the "true" nature of things or remembering perfect ideas; but a personal and social construction of meaning out of a bewildering array of sensations which have no order or structure besides the explanations that are formulated for them (Hein, 1991, cited in Richardson, 2003). Constructivists follow a pedagogy that provides learners with the opportunity to interact with sensory data and construct their own world. Phillips (2000) and Richardson (2003) identified two main forms of constructivism: social constructivism and psychological constructivism. Social constructivism postulates that the understanding and knowledge a person or group have of their world is related to their status in terms of power, economy, political as well as University of Ghana http://ugspace.ug.edu.gh 34 other social factors. Psychological constructivism on the other hand, advance that individuals generate understanding or knowledge around a phenomenon. The meaning generated around the phenomena is idiosyncratic depending upon the individual’s background knowledge. The development of meaning could also take place in a group when they have the opportunity to share their meanings. If they reach consensus on a certain understanding relating to a phenomena, it becomes formal knowledge to them (Richardson, 2003). This study is aligned with the psychological constructivism of the constructivist learning theory. The linkage between process use and psychological constructivism is that, process use is based upon constructivist principles of learning in which people as individuals and as a group construct knowledge around a phenomenon (evaluation). Process use views evaluation as a learning process in which as groups of people come together to undertake an evaluation, they individually and socially learn from the evaluation process. Constructivist learning theory has a long history and is generally attributed to Jean Piaget who articulated mechanisms by which knowledge is internalized by learners (Phillips, 2000; Richardson, 2003). He suggested that through the processes of accommodation and assimilation, individuals construct new knowledge from their experiences (Richardson, 2003 and Phillips, 2000). When individuals assimilate, they incorporate the new experience into an already existing framework without changing it. Assimilation occurs when the individuals' experiences are aligned with his/her internal representations of the world. It may also occur because of a failure to change a faulty understanding. In situations when individuals' experiences contradict their internal University of Ghana http://ugspace.ug.edu.gh 35 representations, assimilation occurs when they change their perceptions of the experiences to fit their internal representations. Accommodation on the other hand, is the process of reframing one's mental representation of the external world to fit new experiences. Accommodation is depicted as a mechanism by which failure leads to learning. Constructivist learning theory is based on certain key principles which inform how learning can be facilitated (Hein, 1991, cited in Richardson, 2003). In the first place, learning is conceived as an active process in which the learner uses sensory input and constructs meaning out of it. That is, learning is not the passive acceptance of knowledge which mysteriously exists somewhere but something that occurs as a consequence of the learners engaging with the world. Thus for effective learning to occur, opportunities must be presented to the learners to participate in the process. The second principle in constructivist learning is that, people learn to learn as they learn. In other words, learning consists both of constructing meaning and constructing systems of meaning. This principle is illustrated clearly in chronological studies in which as one learns the chronology of dates of a series of historical events, he/she simultaneously learns the meaning of a chronology. Similarly, the process of learning about the achievement of a programme results also involves learning how to find out the achievement of projects. In constructivist learning, the crucial action of constructing meaning happens in the mind. Consequently, opportunities are offered for learners to question their beliefs, assumptions, knowledge, attitudes among others on a phenomenon. Activities which engage the mind as well as the hands (reflective activity) should be provided. Important University of Ghana http://ugspace.ug.edu.gh 36 to constructivist learning is the acknowledgement that learning is a social activity. Consequently, constructivists use conversation, interaction with others, and the application of knowledge as an integral aspect of learning. According to Billett (2004) and Kalliola, Nakari & Pesonen (2006), when learning is perceived as a social activity, it aids the development of an interactive learning environment that provides a common perspective for all the stakeholders involved in the learning process. Learning is contextual in that learning occurs in relation to what is already known by the learner as well as the learner’s believes, prejudices and fears. Another principle is that, one needs knowledge to learn. This means that for meaningful learning to take effect, learners must have some previous knowledge. It takes time to learn: learning is not instantaneous and therefore needs ample time to allow for reflection, analysis and feedback. Motivation is a key component in learning. It includes an understanding of ways in which the knowledge can be used. These various principles/elements of constructivist-learning have formed the basis for the conceptual framework and for that matter the research design and methodology for the study especially during the various stages of the Q Sampling process as well as in the discussion of the results of the study. University of Ghana http://ugspace.ug.edu.gh 37 2.3 Description of Concepts 2.3.1 Development Programme A development programme has been defined more generically as “a set of resources and activities directed toward one or more common goals.’ (Newcomer, Hatry & Wholey, 1994, p 3) Development programmes involve people, resources, management, and environmental forces (Fitzpatrick, Sanders & Worthen, 2004; Gibelman, 2003). In much of the poor world, development programmes are main channels for delivering interventions towards addressing poverty; and are usually implemented by designated programme personnel who are involved in direct interactions with persons receiving the service within a particular locality (Pawlak & Vinter, 2004). Development programmes usually address a development problem; has targeted goals and objectives that it wants to achieve; has designated resources including personnel; and represents a significant proportion of the total activity of the organization implementing it (Martin & Kettner, 1996). In the context of this study, a development programme is defined to include any social and or economic designed intervention with the objective of increasing and or improving accessibility to essential services and or reducing poverty. These programmes are essential to the sustenance of wellbeing of people living in poor countries. However, there is even more need to find out exactly what development programmes work, how they work and under what contexts/environments they work. In the search for ways of improving the effectiveness of development programmes, evaluation has been identified and profiled as a way of increasing the effectiveness of development programmes. However, how evaluation does this University of Ghana http://ugspace.ug.edu.gh 38 especially from the perspective of development programmes in Ghana has not been fully explored. This study explores the evaluation processes in development programmes in order to draw lessons for the conduct of evaluation in ways that enhances the effectiveness of development programmes. 2.3.2 Development effectiveness Concern for development effectiveness is manifested by the increasing concern for organizations involved in the financing, design, and implementation of development programmes to improve their performance, in terms of not only the outcomes of the programmes, but also of the quality of the processes of programme design, financing and implementation management as well as their institutional capacity (Engel, Carlsson & Zee, 2003). Development effectiveness has simply been defined by Quibria (2004) as the effectiveness of aid/development programmes in fostering development. Development effectiveness is about the ‘how’ of development that is; it is about the factors and processes that contribute to the achievement of the goals of development programmes or aid. The DAC working party on aid evaluation is cited by Quibria (2004) as defining development effectiveness as “the extent to which the development intervention’s objectives were achieved or are expected to be achieved, taking into account their relative importance.” Development effectiveness as a concept has evolved with the concept of development. As noted by World Bank (2002), the goal of development in the developing world was to increase the rate of growth of the national economies of these countries. Development effectiveness was consequently measured in terms of the level of increase in per capita income or growth rate of the national economy. However, it was soon University of Ghana http://ugspace.ug.edu.gh 39 realized that economic growth measured in terms of per capita income, rate of growth of national economy, level of urbanization, and level of industrialization came along with them different problems – widening gap in inequality, rise of social problems, and emergence of new diseases among others (Conyers &Hills, 1984). Consequently, in the mid 70s the concept of development was redefined with emphasis being placed on human beings as the centre of development as against the earlier development paradigm which was believed to focus on things rather than man. Development effectiveness was measured in terms of how the basic needs of man – shelter, health, food and clothing have been met. In the advent of the MDGs, development effectiveness has been defined by United Nations Development Programme (2003) cited in Quibria (2004) as follows: “Development effectiveness refers to a fundamental question about how to reach the (MDG) Goals. It is about the factors and conditions that help improve people’s lives.” The 2005 Paris Declaration and the 2010 Accra agenda also profiled the issue of development effectiveness. The Paris Declaration is for instance seen as a forthright and challenging approach to overcoming some of the structural problems confronting the attainment of development effectiveness in development programmes. Five core elements or principles of the Paris Agenda convey this message; alignment- adapting development programmes to national strategies and planning processes as well as budgets and institutional frameworks; harmonization - harmonization of procedures in planning, implementation, disbursement mechanisms and evaluation in development cooperation; donor coordination - increase in donor coordination in order to avoid parallel or even contradictory programme structures; decentralization – call on donor agencies to shift an University of Ghana http://ugspace.ug.edu.gh 40 increasing share of decision-making processes from headquarters to their local agencies in recipient countries; and MDG orientation – call on donor agencies to prioritize the MDGs as against the policy of structural adjustment programmes. 2.3.3 Evaluation Inquiry Evaluation inquiry is perceived here as encompassing the purpose and type of the evaluation, the level of stakeholder involvement, and the methods employed in the evaluation among others (Cousins & Whitmore, 1998, cited in Cousins et al., 2004). In this section, the concept of evaluation is first examined and defined before exploring into the range of issues that constitute an evaluative inquiry. Evaluation has been variously defined. The Development Assistance Committee Working Party on Aid Evaluation describes evaluation as follows: “An evaluation is an assessment, as systematic and objective as possible, of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, developmental efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors.’’ (Organisation for Economic Cooperation and Development, 1991, p 5) Alaimo (2008) outlined various definitions of evaluation put forward by some authorities in the evaluation field. Chen (2005, p 3) was quoted as defining evaluation as ‘the application of approaches, techniques, and knowledge to systematically assess and improve the planning, implementation, and effectiveness of programmes.’ Patton (1997, p 23), the author of utilisation focused evaluation in which process use was first coined, was referred to as defining evaluation as ‘the systematic collection of information about University of Ghana http://ugspace.ug.edu.gh 41 the activities, characteristics, and outcomes of programmes for use by specific people to reduce uncertainties, improve effectiveness, and make decisions with regard to what those programmes are doing and affecting.’ Another definition quoted by Alaimo (2008) which brings out some essential elements of evaluation is Mattesich (2003), who defined evaluation as; “...a systematic process for an organization to obtain information on its activities, its impacts, and the effectiveness of its work, so that it can improve its activities and describe its accomplishments.” (Mattesich, 2002, p 3) Significant attributes of evaluation emerging from the different definitions as noted by Alaimo (2008) include the recognition that evaluation is a systematic process for collecting information about the programme; it involves assessment - judging the worth or value of the programme; and the ultimate purpose of programme evaluation is to gather and use information to improve upon the programme. Following from the various definitions elucidated, this study defines evaluation to be any conscious orderly process of finding out the level of performance of development programmes, the factors responsible for their performance and the processes involved in generating and feeding the lessons into the programme development process with the view to enhancing the overall performance of the programme and improvement in the design and management of future programmes. Key in this definition is the emphasis placed on the feedback systems which are viewed as being essential to the use of evaluation. University of Ghana http://ugspace.ug.edu.gh 42 Purpose of Programme Evaluation A key component of evaluation enquiry is evaluation purpose. According to Chelimsky (1997) and Patton (2008), the purpose of an evaluation determines the use that it will be put into. Evaluation is not conducted for any one purpose (Patton, 2008; Powell, 2006; Chelimsky, 1997). According to Chelimsky (1997), evaluation purposes are classified into three namely: evaluation for accountability, evaluation for programme improvement and evaluation for knowledge generation. Evaluation for accountability