Salehi et al. Human Resources for Health (2023) 21:18 Human Resources for Health https://doi.org/10.1186/s12960-023-00804-w RESEARCH Open Access Evaluation of a continuing professional development strategy on COVID-19 for 10 000 health workers in Ghana: a two-pronged approach Roxana Salehi1, Stephanie de Young1* , Augustine Asamoah2, Sawdah Esaka Aryee2, Raymond Eli2, Barbara Couper1, Brian Smith1, Charity Djokoto2, Yaa Nyarko Agyeman5, Abdul‑Fatawu Suglo Zakaria6, Nancy Butt1, Amma Boadu4, Felix Nyante3, Gifty Merdiemah7, Joseph Oliver‑Commey8, Lawrence Ofori‑Boadu4, Samuel Kaba Akoriyea4, Megan Parry1, Cindy Fiore1, Faustina Okae2, Archibald Adams2 and Hannah Acquah2 Abstract Background COVID‑19 has created unprecedented challenges for health systems worldwide. Since the confirma‑ tion of the first COVID‑19 case in Ghana in March 2020 Ghanian health workers have reported fear, stress, and low perceived preparedness to respond to COVID‑19, with those who had not received adequate training at highest risk. Accordingly, the Paediatric Nursing Education Partnership COVID‑19 Response project designed, implemented, and evaluated four open‑access continuing professional development courses related to the pandemic, delivered through a two‑pronged approach: e‑learning and in‑person. Methods This manuscript presents an evaluation of the project’s implementation and outcomes using data for a subset of Ghanaian health workers (n = 9966) who have taken the courses. Two questions were answered: first, the extent to which the design and implementation of this two‑pronged strategy was successful and, second, outcomes associated with strengthening the capacity of health workers to respond to COVID‑19. The methodology involved quantitative and qualitative survey data analysis and ongoing stakeholder consultation to interpret the results. Results Judged against the success criteria (reach, relevance, and efficiency) the implementation of the strategy was successful. The e‑learning component reached 9250 health workers in 6 months. The in‑person component took con‑ siderably more resources than e‑learning but provided hands‑on learning to 716 health workers who were more likely to experience barriers to accessing e‑learning due to challenges around internet connectivity, or institutional capacity to offer training. After taking the courses, health workers’ capacities (addressing misinformation, supporting indi‑ viduals experiencing effects of the virus, recommending the vaccine, course‑specific knowledge, and comfort with e‑learning) improved. The effect size, however, varied depending on the course and the variable measured. Overall, participants were satisfied with the courses and found them relevant to their well‑being and profession. An area for improvement was refining the content‑to‑delivery time ratio of the in‑person course. Unstable internet connectivity and the high upfront cost of data to access and complete the course online were identified as barriers to e‑learning. *Correspondence: Stephanie de Young stephanie.deyoung@sickkids.ca Full list of author information is available at the end of the article © The Author(s) 2023. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creati veco mmons. org/ licen ses/b y/4. 0/. The Creative Commons Public Domain Dedication waiver (http:// creat iveco mmons.o rg/ publi cdoma in/z ero/1. 0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Salehi et al. Human Resources for Health (2023) 21:18 Page 2 of 13 Conclusions A two‑pronged delivery approach leveraged distinct strengths of respective e‑learning and in‑person strategies to contribute to a successful continuing professional development initiative in the context of COVID‑19. Keywords Continuing professional development, Evaluation, Low‑resource settings, E‑learning, Nursing education Introduction partners [15, 16]. Courses were designed by the project’s COVID-19 has created an unprecedented crisis world- content experts in Ghana and Canada, and one of the wide, highlighting that a well-trained health workforce project’s primary stakeholders, the Nursing and Mid- who can continuously update their skills in response to wifery Council of Ghana (N&MC) accredited and pro- new pathogens and emergencies is imperative to a popu- moted the courses. lation’s health [1–4]. However, health worker education Courses were delivered through a "two-pronged strat- in sub-Saharan Africa still faces numerous challenges egy": e-learning and in-person. The courses were also that limit progress toward provision of quality care and adapted for and made available to health workers outside attainment of Sustainable Development Goal-3 (Good of Ghana. The project, which continues until April 2023, Health and Well-being). These include shortage of quali- at the time of writing (July 2022) has surpassed its targets. fied faculty and teaching resources, outdated curricula, A total of 18 855 health workers in Ghana and outside of and lack of institutional capacity [5–7]. In Ghana, these Ghana (primarily from East and West Africa) have com- challenges exist in the larger context of health worker pleted at least one of the four courses. This manuscript shortage, despite significant gains in the last decade, and presents the results of evaluating the project’s implemen- the unequitable distribution of this workforce across tation and outcomes using data for a subset of Ghanaian the country’s different regions [8]. Nurses and midwives health workers who have taken the courses (n = 9966). remain the primary, and, in underserved areas, some- times the only health workforce. Therefore, investing in Key evaluation questions their continuing professional development is crucial, so This evaluation answers two questions: that they can update and enhance their professional skills and keep themselves and their communities safe [9–11]. (1) How successful was the project in implementing Since the confirmation of the first COVID-19 case in the two-pronged strategy (e-learning and in-per- Ghana in March 2020 [12], Ghanaian health workers son) for the four professional development courses? have reported low perceived preparedness to respond to (2) To what extent did the courses contribute to COVID-19. Although several factors contribute to their strengthening health workers’ capacity regarding level of perceived preparedness (e.g., access to personal COVID-19 prevention and health promotion? protective equipment and supportive management), training was found to be the strongest predictor [13]. Implementation success [17–19] was defined by stake- Health workers who had not received COVID-19-re- holders as (i) reaching a broad range of frontline health lated training were at highest risk for stress, anxiety, and workers across Ghana, including in underserved areas; burnout [13, 14]. One of the initiatives that responded to (ii) delivering courses that are relevant to Ghanaian the gap in COVID-19 training to address health worker health workers’ learning needs; and (iii) efficient resource preparedness was SickKids-Ghana Paediatric Nursing use (time, human resources, technology, and financial) to Education Partnership COVID-19 Response Project, deliver the courses. Health workers’ capacity is defined in hereafter referred to as “the project.” It set out to train "Measures" section. 10 000 health workers in Ghana by designing and imple- menting four relevant continuing professional develop- Existing evaluations: the larger context ment courses, provided at no cost to course participants. Currently, there is a gap in the evaluation of continu- The implementing partners for this project include ing professional development efforts in low-resource Ghana College of Nurses and Midwives (GCNM), Ghana settings, especially e-learning interventions [6, 20, 21]. Health Service (GHS), Ministry of Health (MoH) Ghana, Well-designed e-learning interventions—those deliv- and the Centre for Global Child Health at The Hospital ered through information and communication technol- for Sick Children (SickKids), Canada. The project, funded ogy using a variety of instructional designs [6, 20]—can by the Government of Canada through Global Affairs potentially alleviate the burden of faculty shortages and Canada, is an extension of the SickKids-Ghana Paediatric provide access to affordable education irrespective of Nursing Education Partnership (PNEP) and builds on 10 geography. They offer a convenient way to gain knowl- years of collaboration between SickKids and its Ghanaian edge and skills and are easier to update than curricula S alehi et al. Human Resources for Health (2023) 21:18 Page 3 of 13 delivered by other means [22]. Financially, they cost less The courses were designed to target a wide range of than traditional face-to-face training because of reduced health workers including nurses, midwives, and physi- costs associated with human resources, travel, and insti- cians. The first three courses were designed for delivery tutional infrastructure, and enable scaling up the reach via the World Continuing Education Alliance (WCEA) of training at a comparatively low cost [23, 24]. Many e-learning platform, while Course Four was designed health workers in Ghana already use their mobile phones for in-person delivery. The e-learning courses included for learning purposes, have a positive attitude toward narrative descriptions, interactive activities, and case e-learning, and exhibit high levels of preparedness for studies where learners applied their knowledge. The par- using e-learning interventions [25, 26]. Challenges with ticipants could take as many of the e-learning courses as internet connectivity and the cost of data are recognized they wanted. The in-person course allowed for additional barriers to maximizing the benefits of e-learning inter- multi-modal teaching and learning strategies, includ- ventions, particularly in rural areas [6, 25, 26]. ing skills stations, role play, and case presentations. The Evaluations comparing the quality of e-learning to tra- participants of the in-person course were also provided ditional in-person learning have produced mixed results. with technical support to access the e-learning platform Some argue that e-learning can produce comparable or and were encouraged to take the other three courses. All better gains in knowledge and practice [27]; others sug- four courses included a pre-and post-knowledge test and gest that in-person training is superior, because it offers a commitment to change action plan.  In January 2022, opportunities for hands-on exercises [22]. A meaning- courses were updated to reflect new information about ful comparison, however, depends on assessing the spe- COVID-19 and its management. cific context surrounding the training (course duration, topic complexity, technology used, learners’ baseline Implementation knowledge, trainers’ level of expertise) and its evalua- Approach 1: e‑learning tion (design, sample size, types of variables). For example, The e-learning courses were launched in September 2021. training on specialized topics, training combined with The World Continuing Education Alliance (WCEA) supportive supervision, and training whose evaluations e-learning platform allows users to download course con- are conducted right afterward generally have a larger tent to their smartphones or other portable devices and effect size compared to general topic courses, training continue the course offline. However, users must be con- without supportive supervision, and training with evalu- nected to the internet to verify course completion and ations undertaken months afterward [28, 29]. Finally, submit answers to multiple-choice exams and evaluation effect sizes are generally larger when the training inter- questions and to obtain their certificates. To recruit par- vention has a targeted sample specifically selected based ticipants, the Ghana College of Nurses and Midwives and on the assumption that participants will benefit, rather the Nursing and Midwifery Council (Ghana) promoted than when the sample is more diverse [29, 30]. Unfor- the courses through their websites and social media plat- tunately, much of the existing evaluation literature lacks forms. WCEA also profiled the courses on their platform this important contextual information [23, 28, 31, 32]. to encourage participation. Curriculum Approach 2: in‑person From May to August 2021, the Ghana College of Nurses In-person training started in November 2021 while and Midwives and the SickKids Centre for Global Child adhering to COVID-19 risk mitigation safety protocols. Health developed the following courses, working in col- A targeted approach was used to recruit health workers laboration with experts from the Ghana Health Service from identified rural areas with poor internet connectiv- and Ministry of Health: ity and who do not have frequent training opportunities. The Ghana Health Service Director-General, regional Course One: COVID-19 in Ghana: Prevention and and district managers, facility managers, and the pro- Health Promotion (e-learning), ject team collaborated to select participants. Similarly, Course Two: COVID-19 Vaccines in Ghana: Com- the head offices of the Christian Health Association of munication and Behaviour Change (e-learning), Ghana (CHAG) and the Ahmadiyya Muslim Mission Course Three: COVID-19 in Ghana: Child and Ado- nominated participants. During the in-person course, lescent Health (e-learning), and participants not only developed competencies around Course Four: COVID-19 in Ghana: Promoting Physi- COVID-19, but they also received support to navigate cal and Mental Health of Children, Families, and the WCEA platform and were encouraged to take addi- Health Workers (in-person). tional e-learning courses to support their ongoing profes- sional development. Salehi et al. Human Resources for Health (2023) 21:18 Page 4 of 13 Methodology • Knowledge about COVID-19 using the score of a Ethics approval was obtained from the Ghana Health multiple-choice exam covering key aspects of the Service Ethics Review Committee in October 2021. A set curriculum in each course (zero to 100), and of evaluation questionnaires were designed by curricu- • Comfort level with e-learning (5-point scale). lum development team members and the project’s evalu- ation team. A group of graduates from the Ghana College Responses were captured using pre–post-surveys. of Nurses and Midwives piloted the evaluation tools to ensure clarity of language. For in-person sessions, Analytical approach questionnaires identical to those used in the e-learning Statistical analysis was performed using IBM SPSS Sta- courses were filled out using paper surveys. tistics, version 25. Descriptive statistics were conducted to summarize the characteristics of the participants. Data included and missing data The distribution of variables was analyzed and, subse- The results presented here focus on 9966 health workers quently, Wilcoxon Signed Ranks test for paired samples who either took the e-learning courses between October was used to measure the capacity of the participants 26, 2021, and February 22, 2022 (n = 9250), or attended regarding COVID-19 prevention and health promotion one of 19 two-day in-person courses offered between before and after each course. The only exception was December 29, 2021, and April 2, 2022 (n = 716). The the multiple-choice exam score of Course Four (in-per- missing data consists of (i) individuals who chose not to son): due to logistical challenges, it was not possible to answer certain evaluation questions, and (ii) those who link the pre- and post-knowledge scores for this group; did not click on the last button on the e-learning plat- therefore, the Mann–Whitney U test was used. The sig- form and did not go to the page containing the evalua- nificance level was set at 0.01. Effect sizes associated with tion questions and their certificate of completion. A push Wilcoxon Signed Ranks tests and Cohen’s criteria of 0.1 notification was sent out to remind and encourage the small effect, 0.3 medium effect, and 0.5 large effect were learners to complete the process, but it did not improve used to report findings [35, 36]. Since these thresholds the rates of missing data. are arbitrary and fail to consider important differences in characteristics of educational intervention, the practi- Measures cal significance of the results was determined by ongoing Implementation: Reach was captured using a demo- team and stakeholder discussions and situating results graphic survey completed after each course. A satis- within the relevant literature [29, 30, 34]. A total of 4 034 faction survey (5-point scale) with an open-ended text qualitative comments for the open-ended survey ques- question measured the relevance of the strategy. The tion were analyzed using a thematic content analysis strategy’s efficiencies were measured qualitatively by method [35], identifying the top five categories of com- analyzing ongoing partner discussions throughout the ments. Percentages in tables are rounded and might not project. Although comprehensive economic analysis was add up to exactly 100. outside the scope of the evaluation, the financial cost of developing and delivering the courses was monitored to Results inform implementation. Cost categories included Human Reach Resources (HR) and meeting costs for curriculum devel- As shown in Tables 1 and 2, 75% of participants were opment, HR and platform costs for e-learning, and HR under the age of 34. Female health workers accounted and travel costs for in-person training. for 71% of learners. Most of the participants were Outcomes: Health workers’ capacity measurement was general nurses, enrolled nurses, or community health informed by Finn and Colleague’s conceptual framework nurses. The top three facility types were district hos- [33] and used five variables: pitals, health centres, and Community-based Health Planning and Services (CHPS) compounds. Partici- • Confidence in addressing misinformation about pants in e-learning courses followed these overall COVID-19 with patients, colleagues, and community demographic trends. Participants in the in-person members (5-point scale), course, however, were 52% female, consisted of 19% • Ability to communicate effectively to support indi- midwives (third highest frequency), and the top facility viduals experiencing negative impact of COVID-19 type in which they worked was CHPS compounds. (5-point scale), Table  2 shows that 25% of all participants came • Recommending the COVID-19 vaccine to patients from Central and Greater Accra Regions. Less than and colleagues, if available (5-point scale), 4% were from North East and Ahafo regions. While S alehi et al. Human Resources for Health (2023) 21:18 Page 5 of 13 Table 1 Characteristics of 9966 Ghanaian health workers who took the four courses Total (n = 9966) E‑learning (n = 9250) In‑Person (n = 716) n % n % n % Age < 24 179 1.8 175 1.9 4 0.6 25–34 7286 73.2 6764 73.2 522 72.9 35–44 1871 18.8 1712 18.6 159 22.3 45–54 102 1.1 87 1.0 15 2.1 55 + 43 0.5 40 0.5 3 0.4 Missing 485 4.9 472 5.2 13 1.8 Gender Female 7100 71.2 6726 72.7 374 52.2 Male 2805 28.1 2476 26.7 329 45.9 Missing 61 0.6 48 0.5 13 1.8 Primary qualification General nurse 3459 34.3 3256 35.2 203 28.4 Enrolled nurse 1696 16.9 1645 17.8 51 7.2 Community health nurse 1220 12.1 1055 11.4 165 23.1 Midwife 1145 11.4 1010 10.9 135 18.9 Public health nurse 248 2.5 178 1.9 70 9.8 Physician or Surgeon 112 1.2 112 1.2 0 0 Othera 725 7.2 648 7.0 77 10.8 Missing 1361 13.5 1346 14.4 15 2.1 Level of Facility District hospital 3137 31.1 2964 32.1 173 24.2 Health centre 1683 16.7 1488 16.1 195 27.3 CHPS compound 993 9.9 779 8.5 214 29.9 Teaching hospital 754 7.5 754 8.2 0 0 Regional hospital 441 4.4 419 4.6 22 3.1 Poly clinic 439 4.4 421 4.6 18 2.6 Academic institution 108 1.1 108 1.2 0 0 O therb 1051 10.5 971 10.5 80 11.2 Missing 1360 13.5 1346 14.6 14 2.0 CHPS Community-based Health Planning and Services a For e-learning courses, the ’other’ category included paediatric nurses, mental health nurses, administrators and managers, students, and health assistants. For in-person courses, the ’other’ category included health promotion officers, disease control officers, health assistants, and mental health nurses b For e-learning courses, the ’other’ category includes the Ministry, private facilities, and NGOs. For in-person courses, the ’other’ category included Regional Directorate, District Health Directorate, Municipal Hospital, Mission Hospital, and Municipal Health Directorate the participants of e-learning courses followed these (n = 6381), 4.49 ± 0.59 (n = 3385), 4.47 ± 0.57 geographic trends, the participants in the in-person (n = 2553), and 4.73 ± 0.48 (n = 714), respectively. For study mainly came from Volta, Western North, Bono, e-learning courses, participants described the learn- and Oti regions, reflecting the recruitment strategy ing as "relevant to my work" and "well-structured" described in "Approach 2: in-person" section. (Table  3). They described the e-learning environment as "logically laid out" and "a faster, safe, and convenient Relevance way of learning." Some felt "encouraged to take other Mean overall satisfaction for Courses One (health pro- courses on the WCEA app" and "regret[ted] not join- motion), Two (vaccine), Three (child and adolescent), ing the platform earlier." Interactive photos and videos and Four (physical and mental health) was 4.49 ± 0.59 Salehi et al. Human Resources for Health (2023) 21:18 Page 6 of 13 Table 2 Geographic distribution of 9966 Ghanaian health workers who took the courses Total (n = 9966) E‑learning (n = 9250) In‑Person (n = 716) n % n % n % Region Central 1256 12.7 1256 13.6 0 0 Greater Accra 1215 12.2 1215 13.1 0 0 Eastern 971 9.8 971 10.5 0 0 Ashanti 834 8.4 834 9 0 0 Volta 853 8.6 740 8 113 15.8 Upper West 558 5.6 558 6 0 0 Western 487 4.9 487 5.3 0 0 Northern 388 3.9 299 3.2 89 12.5 Western North 338 3.4 237 2.6 101 14.2 Savannah 272 2.8 223 2.4 49 6.9 Upper East 216 2.2 216 2.3 0 0 Bono 309 3.2 207 2.2 102 14.3 Oti 308 3.1 207 2.2 101 14.2 Bono East 239 2.4 188 2 51 7.2 North East 186 1.9 138 1.5 48 6.8 Ahafo 177 1.8 128 1.4 49 6.9 Missing 1359 13.7 1346 14.6 13 1.9 made learning easier, and participants wanted more Efficiency of them. However, some participants, especially those Decisions regarding resource efficiency (time, human working in rural areas, reported that limited network resources, technology, and financial) took place through- coverage and unstable internet created difficulties for out the project, during both curriculum development fully downloading the course content, especially pho- and implementation. Curriculum development meetings tos and videos. Some participants commented about between experts in Ghana and Canada took place virtu- the cost of data associated with downloading and ally. The content team made efforts to avoid duplication uploading content. A suggestion for improvement was of content and to focus on specific priority areas, such to create a space on the courses’ platform, where learn- as child health and vaccine hesitancy, for which limited ers could interact. open access content was available. Participants who completed Course Four (in-person) Core curriculum development cost approximately 35% described it as "helpful" and "the best." They appreci- of the total investment for the project between May 2021 ated the opportunity to participate in this course, and and April 2022. E-learning implementation cost 10% of comments were made about the need for this type of the total investment for the project in the same period training to be conducted in person "due to poor net- and reached 9250 health workers across Ghana. The work service in rural settings." A frequent comment World Continuing Education Alliance (WCEA) platform was the wish for training to be repeated for other was selected, because over 80 000 nurses, midwives, and health workers "to also build their capacity." Case- students in Ghana were already registered on it, repre- based learning was viewed positively, leading to "better senting a broad geographical reach. WCEA’s integration understanding of the material." The facilitators and the with Nursing and Midwifery Council, Ghana’s system for organization of the course were described as "superb." continuing professional development, facilitated course Areas for improvement included dedicating more time accreditation. to training, given the content volume (Table 3). Implementation of the in-person strategy (Course Four) cost 55% of the total investment, reaching 716 health workers living in underserved regions. Regional trainers were chosen to deliver the course, cascading Salehi et al. Human Resources for Health (2023) 21:18 Page 7 of 13 Table 3 Top five categories of comments by health workers who took the courses Category Selected quotes General short positive comments about the course "This module met the mental health needs of healthcare workers […]. The best education I’ve had on COVID-19 so far."—Nurse, Course One "Very educative and interactive. I didn’t feel bored, unlike some other modules which only involve read- ing and no activities."—Registered community nurse, Course Two "[It] was impactful, involved presentations from participants and role play which was educative."—Gen‑ eral nurse, Course Four Knowledge, skills, and confidence gained "[…] it was in clear and simple language which will enable me to explain everything about COVID-19 to my community members."—Enrolled nurse, Course Two "I had my doubts about the vaccine even as a health worker, but all that is cleared now"—Registered nurse, Course Two "The module motivated [me] and increased my skills on management of children and adolescents with COVID-19."—Midwife, Course Three "Not so many people are confident taking e-learning courses and exams, including me, but after this module, my confidence is boosted and want to do more courses here."—Nurse, Course One Challenges of e‑learning "It is data consuming to [download]!"—Enrolled nurse, Course One "To access this course online is very complicated, especially to those who don’t have much knowledge in ICT and [live in] hard-to-reach areas."—Midwife, Course One "At times, you will be eager to work, but the network will be disturbing."—Midwife, Course Four Opportunities presented by e‑learning "I am encouraged to take other courses on the WCEA app."—Midwife, Course One "It is faster, safe and convenient way of learning and updating oneself on the job content."—Enrolled nurse, Course Three "The app is a good one, and I regret not joining earlier."—Community health nurse, Course Three "The course was very informative and had awakened my desire to take more online CPD courses."— General nurse, Course Four Areas for improvement/suggestions “Because the presentations are packed [it] should have been a three day training."—Registered com‑ munity nurse, Course Four "Please, more video demonstration or picture demonstration for further studies."—Registered com‑ munity nurse, Course One "The course is very involving, but the CPD point earned is too small. So if anything can be done about it?"—Midwife, Course One "Is it possible, a platform where individuals can interact and share ideas or even study together?"—Gen‑ eral nurse, Course Two "The training is very useful and should be extended to cover more healthcare workers."—Public health nurse, Course Four training nationwide and strengthening regional training Course two: vaccine capacity. A total of 4023 individuals completed Course Two (Table  5). The statistically significant shifts in levels of Participant’s capacity agreement (strongly agree or agree) with the statements Course one: prevention and health promotion ‘I feel confident addressing misinformation’ and ‘I am A total of 7243 individuals completed Course One able to effectively support individuals experiencing nega- (Table  4). The statistically significant shift in levels of tive impact of COVID-19’ increased between baseline agreement (strongly agree or agree) with the statements and post-training, from 71.7% to 81.1% and from 70.9% ‘I am able to effectively support individuals experienc- to 80.9%, respectively. Both effect sizes were medium. ing negative impact of COVID-19’ increased from 74.7% Similarly, the statistically significant shift in levels of to 84.9% between baseline and post-training, with a agreement in relation to the statement ‘I feel comfort- medium effect size. Similarly, the statistically significant able taking online courses’ increased from 71.6% to 80.6% shift in levels of agreement in relation to the statement with a medium effect size. The shifts in levels of agree- ‘I feel comfortable taking online courses’ increased from ment with the statement ‘I would recommend the vac- 72.2 to 84% with a medium effect size. The shifts in lev- cine’ was statistically significant with small effect size. els of agreement with the statements ‘I feel confident The knowledge score increased by 32% from baseline to addressing misinformation’ and ‘I would recommend after course completion with a medium effect size. The the vaccine’ were statistically significant with small effect ’clear language’ used in this course was considered help- sizes. The knowledge score increased by 14% from base- ful in transmitting knowledge about COVID-19 to the line to after course completion with a medium effect size. larger community. Some learners said the course had Salehi et al. Human Resources for Health (2023) 21:18 Page 8 of 13 Table 4 Participants’ capacity before and after taking Course One: Prevention and Health Promotion (N = 7243) Variablesa Before Immediately Pb Zc Effect size (r) after n (%) n (%) I feel confident addressing misinformation about COVID‑19 with patients, < 0.01 − 32.4 − 0.28 colleagues and community members Strongly disagree or disagree 270 (3.7) 61 (0.8) Neither disagree nor agree 372 (5.1) 76 (1.0) Agree or strongly agree 5615 (77.5) 6163 (85.1) Missing 986 (13.6) 943 (13.0) I am able to communicate effectively to support an individual who is experi‑ < 0.01 − 38.5 − 0.34 encing negative impacts of COVID‑19 Strongly disagree or disagree 365 (5.0) 52 (0.7) Neither disagree nor agree 527 (7.3) 110 (1.5) Agree or strongly agree 5408 (74.7) 6148 (84.9) Missing 943 (13.0) 933 (12.9) I would recommend the COVID‑ 19 vaccine to patients and colleagues if it < 0.01 − 31.0 − 0.27 was available Strongly disagree or disagree 212 (2.9) 62 (0.9) Neither disagree nor agree 365 (5.0) 140 (1.9) Agree or strongly agree 5727 (79.1) 6114 (84.4) Missing 939 (13.0) 927 (12.8) I feel comfortable taking an online CPD < 0.01 − 36.3 − 0.32 Strongly disagree or disagree 467 (6.4) 64 (0.9) Neither disagree nor agree 592 (8.2) 163 (2.3) Agree or strongly agree 5233 (72.2) 6083 (84.0) Missing 951 (13.1) 933 (12.9) Knowledge (mean score ± SD) d 74.8 ± 17.0 85.5 ± 14.1 < 0.01 − 46.9 − 0.39 a All Likert variables were 5-point scale (1 = strongly disagree, 2 = disagree, 3 = neither disagree nor agree 4 = agree, 5 = strongly agree). Mean score for the Knowledge variable is out of 100 b Wilcoxon Signed Ranks Test, 2-tailed for paired samples, was used for all comparisons c Based on negative ranks d (n for pre-knowledge test = 6799, n for post-knowledge test = 7243) ’eradicated misconceptions’ and ’cleared doubts about the Course four: mental and physical health vaccine’ (Table 3). A total of 716 individuals completed Course Four (Table 7). The statistically significant shifts in levels of Course three: child and adolescent health agreement (strongly agree or agree) with the statements A total of 2892 individuals completed Course Three ‘I feel confident addressing misinformation’ and ‘I feel (Table  6). The statistically significant shift in levels of comfortable taking online courses’’ increased between agreement (strongly agree or agree) with the statements baseline and post-training, from 79.1% to 99.3% and ‘I am able to effectively support individuals experienc- from 66.6% to 98.2%, respectively. Both effect sizes ing negative impact of COVID-19’ increased between were large. The shifts in levels of agreement with the baseline and post-training, from 76.6% to 85%. Similarly, statements ‘I am able to effectively support individu- the statistically significant shift in levels of agreement in als experiencing negative impact of COVID-19’ and ‘I relation to the statement ‘I feel comfortable taking online would recommend the vaccine’ were statistically sig- courses’ increased from 75.1% to 84.5%. Both effect sizes nificant with medium effect sizes. The knowledge score were medium. The shifts in levels of agreement with the increased by 19% from baseline to after course comple- statement ‘I feel confident addressing misinformation’ tion with a medium effect size. Participants reported and ‘I would recommend the vaccine’ were statistically being encouraged to take online courses (Table 3). significant with small effect sizes. The knowledge score increased by 25% from baseline to after course comple- tion with a medium effect size. S alehi et al. Human Resources for Health (2023) 21:18 Page 9 of 13 Table 5 Participants’ capacity before and after taking Course Two: Vaccine (N = 4023) Variablesa Before Immediately Pb Zc Effect size (r) after n (%) n % I feel confident addressing misinformation about COVID‑19 with patients, < 0.01 − 24.9 − 0.31 colleagues and community members Strongly disagree or disagree 201 (5.0) 31 (0.8) Neither disagree nor agree 217 (5.4) 46 (1.1) Agree or strongly agree 2883 (71.7) 3262 (81.1) Missing 722 (17.9) 684 (17.0) I am able to communicate effectively to support an individual who is experi‑ < 0.01 − 27.5 − 0.33 encing negative impacts of COVID‑19 Strongly disagree or disagree 205 (5.1) 20 (0.5) Neither disagree nor agree 286 (7.1) 73 (1.8) Agree or strongly agree 2854 (70.9) 3256 (80.9) Missing 678 (16.9) 674 (16.8) I would recommend the COVID‑19 vaccine to patients and colleagues if it < 0.01 − 24.3 − 0.29 was available Strongly disagree or disagree 102 (2.5) 23 (0.6) Neither disagree nor agree 213 (5.3) 68 (1.7) Agree or strongly agree 3027 (75.2) 3249 (80.8) Missing 681 (16.9) 683 (17.0) I feel comfortable taking an online CPD < 0.01 − 25.5 − 0.31 Strongly disagree or disagree 203 (5.0) 27 (0.7) Neither disagree nor agree 251 (6.2) 70 (1.7) Agree or strongly agree 2879 (71.6) 3242 (80.6) Missing 690 (17.2) 684 (17.0) Knowledge (mean score ± (SD)) d 62.3 ± 17.8 82.0 ± 16.1 < 0.01 − 43.9 − 0.37 a All Likert variables were 5-point scale (1 = strongly disagree, 2 = disagree, 3 = neither disagree nor agree 4 = agree, 5 = strongly agree). Mean score for the Knowledge variable is out of 100. b Wilcoxon Signed Ranks Test, 2-tailed for paired samples, was used for all comparisons c Based on negative ranks d (n for pre-knowledge test = 3333, n for post-knowledge test = 3339) Discussion in a 6-month period. Awareness building of the courses Judged against the evaluation criteria (broad reach, effi- and choosing a learning platform that was already known ciency, and relevance) the implementation of the strategy to health workers in Ghana improved efficiency. How- was found to be successful. Rapid curriculum develop- ever, as has been observed by others [3, 21, 26], the evalu- ment using virtual meetings was made possible by a well- ation identified challenges with e-learning, including established partnership, and good working relationships unstable internet and the cost of data for downloading between Ghanaian and Canadian partners. Investment and uploading content. was maximized by delivering training to a broad range of As others have observed [25], the in-person train- health workers across Ghana using a two-pronged strat- ing took considerably more resources than e-learning, egy of e-learning and in-person training, each offering especially given the need for pandemic-related safety benefits and challenges in a Ghanaian, and pandemic, and logistics. Compared to e-learning, fewer individuals context. (n = 716) were able to be trained via this training modal- The speed at which health workforce training can be ity. However, the in-person course offered hands-on scaled up is particularly important during a pandemic, learning to health workers who needed COVID-19-re- when timely information is essential. Despite some initial lated training and were more likely to experience barriers concerns about the feasibility of this approach, particu- to accessing e-learning. larly around the reach and uptake of e-learning courses, Learners found the courses relevant to their needs this strategy was successfully implemented, reaching a and their feedback identified several implication for large number of health workers (n = 9250) across Ghana future design of such courses. For the e-learning courses, Salehi et al. Human Resources for Health (2023) 21:18 Page 10 of 13 Table 6 Participants’ capacity before and after taking Course Three: Child and Adolescent Health (N = 2892) Variablesa Before Immediately Pb Zc Effect size (r) after n % n % I feel confident addressing misinformation about COVID‑19 with patients, < 0.01 − 20.4 − 0.28 colleagues and community members Strongly disagree or disagree 122 (4.2) 21 (0.7) Neither disagree nor agree 155 (5.4) 40 (1.4) Agree or strongly agree 2218 (76.7) 2457 (85) Missing 397 (13.7) 374 (12.9) I am able to communicate effectively to support an individual who is experi‑ < 0.01 − 22.3 − 0.31 encing negative impacts of COVID‑19 Strongly disagree or disagree 128 (4.4) 22 (0.8) Neither disagree nor agree 176 (6.1) 39 (1.3) Agree or strongly agree 2214 (76.6) 2457 (85.0) Missing 374 (12.9) 374 (12.9) I would recommend the COVID‑19 vaccine to patients and colleagues if it < 0.01 − 20.8 − 0.29 was available Strongly disagree or disagree 73 (2.5) 15 (0.5) Neither disagree nor agree 138 (4.8) 61 (2.1) Agree or strongly agree 2303 (79.6) 2433 (84.1) Missing 378 (13.1) 383 (13.2) I feel comfortable taking an online CPD < 0.01 − 21.7 − 0.31 Strongly disagree or disagree 142 (4.9) 21 (0.7) Neither disagree nor agree 188 (6.5) 48 (1.7) Agree or strongly agree 2172 (75.1) 2445 (84.5) Missing 390 (13.5) 378 (13.1) Knowledge (mean score ± (SD)) d 69.8 ± 17.6 87.3 ± 15.2 < 0.01 − 36.1 − 0.37 a All Likert variables were 5-point scale (1 = strongly disagree, 2 = disagree, 3 = neither disagree nor agree 4 = agree, 5 = strongly agree). Mean score for the Knowledge variable is out of 100 b Wilcoxon Signed Ranks Test, 2-tailed for paired samples, was used for all comparisons c Based on negative ranks d (n for pre-knowledge test = 2694, n for post-knowledge test = 2892 interactive content, visual aid (photos, videos), and small already high at baseline, thus a large change (large file sizes to help with the download of course content effect size) would not be expected or possible. were important. For the in-person course, ensuring the Although the in-person course is not directly compa- optimal content-to-delivery time ratio and using plain rable to the e-learning courses due to several key dif- language learning material proved important. ferences (length, teaching methods, and participants’ All four courses contributed to strengthening health characteristics as described in "Approach 2: in-person" workers’ capacity regarding COVID-19 prevention and section), the observed large effect sizes for the in-per- health promotion but the effect size (i.e., the amount son course were encouraging, albeit expected. As men- of change) varied from small to large depending on the tioned earlier, the project made concerted efforts to course and variable measured (Tables 4, 5, 6 and 7). For target participants with less access to training opportu- e-learning courses, the effect sizes varied from small to nities for the in-person course ("Approach 2: in-person" medium, and for the in-person course, the effect sizes and "Reach" section); targeted samples typically result varied from medium to large. While these effect sizes in larger effect sizes [29]. Furthermore, the in-person are useful in painting an overall picture, with educa- course allowed the participants to ask the trainers tion evaluation, a ’small’ effect size on a difficult-to- questions in real-time, work through a hands-on exer- change variable (e.g., attitude toward recommended the cise about addressing misinformation, and receive in- vaccine) could be as valuable as a larger effect size on person support for accessing the e-platform, which are something easier to change (e.g., knowledge) [29–31]. likely to have contributed to the observed larger effect In some situations, the initial levels of confidence were sizes for these variables (Table 7). Salehi et al. Human Resources for Health (2023) 21:18 Page 11 of 13 Table 7 Participants’ capacity before and after taking Course Four: Mental and Physical Health (n = 716) Variables a Before Immediately Pb Zc Effect size (r) after n % n % I feel confident addressing misinformation about COVID‑19 with patients, < 0.01 − 20.4 − 0.53 colleagues and community members Strongly disagree or disagree 69 (9.6) 0 (0.0) Neither disagree nor agree 75 (10.5) 0 (0.0) Agree or strongly agree 566 (79.1) 711 (99.3) Missing 6 (0.8) 5 (0.7) I am able to communicate effectively to support an individual who is experi‑ < 0.01 − 22.3 − 0.33 encing negative impacts of COVID‑19 Strongly disagree or disagree 47 (6.6) 2 (0.3) Neither disagree nor agree 61 (8.5) 1 (0.1) Agree or strongly agree 603 (84.2) 707 (98.7) Missing 5 (0.7) 6 (0.8) I would recommend the COVID‑ 19 vaccine to patients and colleagues if it < 0.01 − 20.8 − 0.45 was available Strongly disagree or disagree 29 (4.1) 3 (0.4) Neither disagree nor agree 52 (7.3) 2 (0.3) Agree or strongly agree 629 (87.8) 706 (98.6) Missing 6 (0.8) 5 (0.7) I feel comfortable taking an online CPD < 0.01 − 21.7 − 0.38 Strongly disagree or disagree 105 (14.7) 1 (0.1) Neither disagree nor agree 129 (18.0) 4 (0.6) Agree or strongly agree 477 (66.6) 703 (98.2) Missing 5 (0.7) 8 (1.1) Knowledge (mean score ± SD) d 66.4 ± 15.8 78.9 ± 15.7 < 0.01 − 36.1 − 0.51 a All Likert variables were 5-point scale (1 = strongly disagree, 2 = disagree, 3 = neither disagree nor agree 4 = agree, 5 = strongly agree). Mean score for the Knowledge variable is out of 100 b Wilcoxon Signed Ranks Test, 2-tailed for paired samples, was used for all comparisons except for comparing the knowledge score for which a Mann–Whitney U-test was used c Based on negative ranks d (n for pre-knowledge test = 716, n for post-knowledge test = 716) Limitations and strength were involved in the interpretation of results, mak- This evaluation lacks a longitudinal perspective as no ing them valid and useful [38]. Finally, this evalua- follow-up evaluations were conducted to determine tion contributes to bridging a gap in the assessment whether participants were applying what they learned of large-scale continuing professional development in their practices [36]. It was beyond the scope of the efforts in low-resource settings, especially e-learning evaluation to conduct a comprehensive economic interventions. analysis, and as such, only costs to the program were calculated. Other costs such as the cost to the health system when health workers leave work to participate Conclusions in training, or the cost of data for e-learners were not The success of the strategy was due to well-established analyzed. Similarly, benefits of training master train- partnerships, the quality and relevance of the curricu- ers for the in-person course were not part of this cost lum, and the two-pronged delivery approach which analysis. Evaluation was done internally by the research maximized reach while reducing barriers to accessing teams at SickKids and the Ghana College of Nurses and education. Health workforce training efforts need to be Midwives, with the potentially biases associated with accompanied by other investments in health systems, internal evaluation. Strengths associated with this eval- notably facility infrastructure, faculty development, and uation include the use of both quantitative and qualita- good quality data. tive data [37]. In addition, a broad range of stakeholders Salehi et al. Human Resources for Health (2023) 21:18 Page 12 of 13 Abbreviations References CPD Continuing professional development 1. World Health Organization. Keep health workers safe to keep patients GHS G hana Health Service safe: WHO. World Health Organization news release. Geneva: World GCNM G hana College of Nurses and Midwives Health Organization; 2020. MoH M inistry of Health 2. World Health Organization. Global strategic directions for nursing and N&MC Nursing and Midwifery Council of Ghana midwifery 2021–2025. Geneva: World Health Organization; 2021. WCEA W orld Continuing Education Alliance 3. Hanefeld J, Mayhew S, Legido‑Quigley H, Martineau F, Karanikolos M, PNEP S ickKids‑Ghana Paediatric Nursing Education Partnership Blanchet K, et al. Towards an understanding of resilience: responding to health systems shocks. Health Policy Plan. 2018;33(3):355–67. Acknowledgements 4. Tsiouris F, Hartsough K, Poimbouef M, Raether C, Farahani M, Ferreira T, The authors wish to acknowledge the Ministry of Health Ghana, Ghana Health et al. Rapid scale‑up of COVID‑19 training for frontline health workers in Services, Ghana College of Nurses and Midwives, Nursing and Midwifery 11 African countries. Hum Resour Health. 2022;20(1):43. Council of Ghana and The Hospital for Sick Children for their commitment to 5. Bvumbwe T, Mtshali N. Nursing education challenges and solutions in children’s health. Special thanks to members of the project teams in Canada Sub Saharan Africa: an integrative review. BMC Nurs. 2018;17(1):3. and Ghana for supporting the CPD implementation and to the health profes‑ 6. Baloyi OB, Jarvis MA. Continuing professional development status in the sionals who participated in the evaluation and who were on the frontline of World Health Organisation, Afro‑region member states. Int J Afr Nurs Sci. the COVID‑19 response in Ghana. The authors would like to thank Dr. Stanley 2020;13: 100258. Zlotkin for reviewing the manuscript draft. 7. Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T. Health professionals for a new century: transforming education to strengthen health systems Author contributions in an interdependent world. Lancet. 2010. https:// doi. org/1 0. 1016/ S0140‑ RS conceived the manuscript, performed statistical analyses, drafted the 6736(10)6 1854‑5. manuscript, and revised drafts assisted by AA, SdY and BS. HA was the 8. Asamani JA, Ismaila H, Okoroafor SC, Frimpong KA, Oduro‑Mensah principal investigator. All other authors contributed to the interpretation of E, Chebere M, et al. Cost analysis of health workforce investments for the results as well as drafting and revising the final paper. All authors read and COVID‑19 response in Ghana. BMJ Glob Health. 2022;7(Suppl 1): e008941. approved the final manuscript. 9. Ghana Ministry of Health, Nursing & Midwifery Council. Continuing Pro‑ fessional Development (CPD): Programmes for Nurse Assistants, Nurses, Funding and Midwives. Accra: N&MC. 2020. https:// www. nmc. gov. gh/w eb/ downl The SickKids‑Ghana Pediatric Nursing Education Partnership COVID‑19 oads/c ateg ory/3‑t raini ng‑ and‑c pd?d ownlo ad= 20: cpd‑ progra ms‑f or‑ Response Project was funded by the Government of Canada through Global nurse‑ assist ants‑n urses‑ and‑ midwi ves. Accessed 15 June 2022. Affairs Canada with in‑kind contributions provided by the Ghana Ministry of 10. World Health Organization. State of the world’s nursing 2020: investing Health. Global Affairs Canada had no implementing role in any PNEP activities in education, jobs and leadership. Geneva: WHO; 2020. Contract No.: and were provided with evaluation results through regular reporting. Ghana Licence: CC BY‑NC‑SA 3.0 IGO. Ministry of Health officials had no direct formal role in evaluation activities but 11. World Health Organization. Global strategy on human resources for were engaged in various committees that oversaw PNEP operations and were health: workforce 2030. Geneva: WHO; 2016. https://w ww.w ho. int/ publi also provided with evaluation results through regular committee reporting. catio ns/i/ item/ 97892 415111 31. Accessed 15 June 2022. 12. World Health Organization. WHO Coronavirus (COVID‑19) Dashboard. Availability of data and materials Geneva: WHO; 2020. https:// covid1 9. who.i nt/ region/a fro/ countr y/ gh. The data sets supporting the conclusions of this article are available on Accessed 15 June 2022. request from the corresponding author. 13. Afulani PA, Gyamerah AO, Aborigo RA, Nutor JJ, Malechi H, Laar A, et al. Perceived preparedness to respond to the COVID‑19 pandemic: a study with healthcare workers in Ghana. J Glob Health Sci. 2020. https:// doi. Declarations org/1 0. 1101/2 020. 07. 10. 20151 142. abstra ct. 14. Ofori AA, Osarfo J, Agbeno EK, Manu DO, Amoah E. Psychological impact Ethics approval and consent to participate of COVID‑19 on health workers in Ghana: a multicentre, cross‑sectional Ethics approval was obtained from the Ghana Health Service Ethics Review study. SAGE Open Med. 2021;9:20503121211000920. Committee in October 2021. We received informed consent from the 15. Stevens B, Donkor ES, Goka BQ, Odame I, Langins M, Riahi S, et al. Impact participants. and sustainability of an accredited paediatric nursing training pro‑ gramme in Ghana. J Nurs Educ Pract. 2014;4:142. Consent for publication 16. Salehi R, Asamoah A, de Young S, Acquah H, Agarwal N, Aryee SE, et al. The consent form stated that the results would be used for publication Scaling up pediatric nurse specialist education in Ghana – a longitudinal, purposes. All data is presented in either aggregate form or deidentified if it is mixed methods evaluation. BMC Nurs. 2021;20(1):32. a quote. 17. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, Competing interests measurement challenges, and research agenda. Adm Policy Ment Health. The authors declare that they have no competing interest. 2011;38(2):65–76. 18. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Author details Fostering implementation of health services research findings into prac‑ 1 Centre for Global Child Health, The Hospital for Sick Children, Toronto, tice: a consolidated framework for advancing implementation science. Canada. 2 Ghana College of Nurses and Midwives, Accra, Ghana. 3 Nursing Implement Sci. 2009. https://d oi.o rg/1 0.1 186/ 1748‑5 908‑4‑5 0. and Midwifery Council of Ghana, Accra, Ghana. 4 Ghana Health Service, Accra, 19. The Organisation for Economic Co‑operation and Development. Better Ghana. 5 University for Development Studies, Tamale, Ghana. 6 Nurses’ and Mid‑ criteria for better evaluation: revised evaluation criteria definitions and wives’ Training College, Tamale, Ghana. 7 University of Ghana Medical Centre, principles for use OECD/DAC network on development evaluation. Paris: Accra, Ghana. 8 Ghana Infectious Disease Centre, Accra, Ghana. OECD. 2019. https://w ww.o ecd. org/ dac/ evalu ation/r evis ed‑ evalu ation‑ crite ria‑d ec‑ 2019. pdf. Accessed 15 June 2022. Received: 17 August 2022 Accepted: 15 February 2023 20. Barteit S, Guzek D, Jahn A, Bärnighausen T, Jorge MM, Neuhann F. Evalu‑ ation of e‑learning for medical education in low‑ and middle‑income countries: a systematic review. Comput Educ. 2020;145:103726. 21. Lawn S, Zhi X, Morello A. An integrative review of e‑learning in the deliv‑ ery of self‑management support training for health professionals. BMC Med Educ. 2017;17(1):183. S alehi et al. Human Resources for Health (2023) 21:18 Page 13 of 13 22. Ruggeri K, Farrington C, Brayne C. A global model for effective use and evaluation of e‑learning in health. Telemed J E Health. 2013;19(4):312–21. 23. Ruiz JG, Mintzer MJ, Leipzig RM. The impact of e‑learning in medical education. Acad Med. 2006;81(3):207–12. 24. Meinert E, Eerens J, Banks C, Maloney S, Rivers G, Ilic D, et al. Exploring the cost of e‑learning in health professions education: scoping review. JMIR Med Educ. 2021;7(1): e13681. 25. Alhassan RK, Spieker N, van Ostenberg P, Ogink A, Nketiah‑Amponsah E, de Wit TFR. Association between health worker motivation and health‑ care quality efforts in Ghana. Hum Resour Health. 2013;11:37. 26. Feldacker C, Jacob S, Chung MH, Nartker A, Kim HN. Experiences and perceptions of online continuing professional development among clini‑ cians in sub‑Saharan Africa. Hum Resour Health. 2017;15(1):89. 27. Sullivan LG. A systematic review to compare the effectiveness of face‑to‑ face versus online (including blended learning) delivery of CME/CPD for healthcare practitioners (HCPs). St Lucia (AU): University of Queensland; 2017. https:// espace.l ibra ry. uq.e du. au/ view/U Q: 674199/s 4251 763_ mphil_ thesis. pdf. Accessed 15 June 2022. 28. Rowe AK, Rowe SY, Peters DH, Holloway KA, Ross‑Degnan D. The effec‑ tiveness of training strategies to improve healthcare provider practices in low‑income and middle‑income countries. BMJ Glob Health. 2021;6(1): e003229. 29. Kraft MA. Interpreting effect sizes of education interventions. Educ Res. 2020;49(4):241–53. 30. Bakker A, Cai J, English L, Kaiser G, Mesa V, Van Dooren W. Beyond small, medium, or large: points of consideration when interpreting effect sizes. Educ Stud Math. 2019;102(1):1–8. 31. Uwimpuhwe G, Singh A, Higgins S, Coux M, Xiao Z, Shkedy Z, et al. Latent class evaluation in educational trials: what percentage of children ben‑ efits from an intervention? J Exp Educ. 2022;90(2):404–18. 32. Fairman JA, Rowe JW, Hassmiller S, Shalala DE. Broadening the scope of nursing practice. N Engl J Med. 2011. https://d oi.o rg/ 10.1 056/ NEJMp 10121 21. 33. Finn M, Gilmore B, Sheaf G, Vallières F. What do we mean by individual capacity strengthening for primary health care in low‑ and middle‑ income countries? A systematic scoping review to improve conceptual clarity. Hum Resour Health. 2021;19(1):5. 34. Cohen J. Statistical power analysis for the behavioral sciences. Hillsdale: L. Erlbaum Associates; 1988. 35. Boyatzis RE. Transforming qualitative information: thematic analysis and code development. Thousand Oaks: SAGE Publications; 1998. 36. Moore DE Jr, Green JS, Gallis HA. Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009;29(1):1–15. https:// doi. org/ 10. 1002/ chp.2 0001. 37. Levin H, Belfield CR. Guiding the development and use of cost‑effective‑ ness analysis in education. J Res Educ Effect. 2015;8:400–18. 38. Patton MQ. Utilization‑focused evaluation. Thousand Oaks: SAGE Publica‑ tions; 2008. Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in pub‑ lished maps and institutional affiliations. Ready to submit your research ? Choose BMC and benefit from: • fast, convenient online submission • thorough peer review by experienced rese archers in your field • rapid publication on acceptance • support for research data, including large and complex data types • gold Open Access which fosters wider collaboration and increased citations • maximum visibility for your research: over 100M website views per year At BMC, research is always in progress. Learn more biomedcentral.com/submissions