Abstract

Background and aim: The introduction of rotavirus vaccine (RVV) in the universal immunization program of India is a big feat as it became the first nation in the World Health Organization (WHO) Southeast Asia region to do so. The involvement of huge numbers of frontline workers in introducing new vaccines in India and the underlined deficits in skills and knowledge require efficient capacity building programs. In view of this, limited research is available on the effectiveness of capacity-building interventions for healthcare workers. There is a dearth of studies from India measuring the “on-spot” impact of immunization trainings on healthcare workers. This study aims to assess the effectiveness of training in RVV introduction in enhancing the knowledge of the participants. Methods: The study was conducted among the participants attending two training workshops for the introduction of RVV: a state workshop in Pune and a regional workshop in Guwahati. The participants who attended the workshops and participated in both the pre- and post-test were included in the study. Real-time data was collected via Google forms pre- and post-training sessions. Results: In both workshops, a comparison of pre- and post-test scores of all questions taken together showed a significant increase in the knowledge level of the participants (p < 0.05). In Guwahati, the knowledge of the participants regarding doses of RVV, inadequate dosing, vaccine vial monitor (VVM), open vial policy, operationalization of RVV and monetary incentive increased significantly. In Pune, the knowledge of the participants regarding doses of RVV, bundling approach, schedule and dose, storage temperature for RVV, VVM, open vial policy, vaccine delivery and operationalization of RVV increased significantly after the training. Conclusion: A pre-planned and well-designed knowledge assessment tool can be used to understand the impact of training workshops in enhancing the knowledge and practical skills of the participants prior to the introduction of a new vaccine.

Keywords: Rotavirus vaccine, capacity building, staff development, training support, training assessment, training of trainers

Rotavirus is a leading cause of moderate-to-severe acute diarrhea in India. The Global Burden of Disease Study showed that in India 21,357.6 (13,150.8-33,967.0) deaths occur in children below 5 years due to rotavirus infection. Hospitalizations due to rotaviral gastroenteritis among under-five children are estimated to reach up to 8,72,000 admissions every year. Rotavirus vaccine introduction in the national immunization program is the most effective intervention in preventing severe rotavirus disease.¹ India became the first nation in the World Health Organization (WHO) Southeast Asia region to introduce the rotavirus vaccine (RVV) in its Universal Immunization Program (UIP). By 2019, India had introduced RVV across 29 states and 8 union territories, covering a birth cohort of 26.7 million.²

Almost 2,00,000 auxiliary nurse midwives (ANMs) work on the ground to successfully implement the UIP as they vaccinate the children and pregnant women, and are supported by around 1 million accredited social health activists (ASHA), who provide counseling to the beneficiaries and help mobilize them.³ Despite the increasing demands placed on the public health workforce due to the introduction of several newer vaccines, skill deficits are evident and may reflect as inadequate preparation leading to on-job trial and errors. Hence, an important pre-requisite of any new vaccine introduction is capacity building of the health functionaries, from the program managers to the frontline health workers (FLWs). Studies have shown that the knowledge, attitudes and practices of the healthcare providers, including the frontline workers and supervisors, as well as parents, are the determining factors for improving immunization coverage.^4-8

However, limited research is available on the effective­ness of capacity-building interventions related to capacity building of health workers. To our knowledge, an evaluation of the effectiveness of capacity-building interventions in new vaccine introduction has not been done so far. Evidence related to the effectiveness of these strategies can go a long way in considering choices about strategies and training focus, leading to better training outcomes.⁹ The pre-test and the post-test pattern help in evaluating the enhancement in the knowledge of the participants. The pre-test assessment may further help in improving the focus of the audience in those technical sessions or topics that they could not answer correctly during the pre-test. It may also enable the facilitators to emphasize specific topics during the technical sessions based on the questions-wise scores’ analysis of the pre-test.¹⁰

Currently, there is limited evidence available from India and other regions of the world measuring the impact of training via pre- and post-assessment in the context of immunization and other public health programs. Also, there is a dearth of literature evaluating the efficacy of short-term training programs in new vaccine introductions. To the best of our knowledge, this study is the first attempt to assess the effectiveness of training in RVV introduction in enhancing the knowledge of the participants. We believe that the results will enable public health professionals and program managers to develop tailor-made training modules in new vaccine introductions to ensure better application of knowledge during the rolling out of new vaccines.

METHODOLOGY

Ethics Statement

The study was designed in accordance with the Declaration of Helsinki. Ethics approval was obtained from the MGM-ECRHS, MGM Medical College, Aurangabad, Maharashtra (Ref. No. MGM-ECRHS/2019/16).

In the study, an assessment of the pre and post scores of the participants attending a government organized training workshop (supported by development partners) was conducted. The government authorities nominated the participants, and the standard agenda was shared with them before the workshop. Before administering the pre- and post-test, the participants were briefed by the facilitators during the workshop. The participants gave their informed consent before study initiation. The analysis of the proportion of question-wise correct responses was shared with the participants real-time.

Study Design

The pre- and post-test were conducted as per the standardized agenda approved by the Ministry of Health and Family Welfare (MoHFW). The pre-test was conducted using a self-administered questionnaire before the start of the workshop to assess the participants’ knowledge of selected program domains essential for introducing the RVV. The facilitators briefed the participants on the objective and the process of conducting the pre-test. The questionnaire, which had 10 questions, was then administered online using a Google form. At the end of all technical sessions on Day 2, the post-test using the same questionnaire was again self-administered to the participants using the Google form.

Questions were close ended with one correct answer for each question. The number of options for each question ranged from 2 to 5. The participants were allotted 20 minutes to complete the tests. To prevent data contamination, facilitators ensured that there was no discussion amongst the participants and that doubts, if any, were clarified individually. The participants were not allowed to refer to any training materials, like operational guidelines, for answering the questions. The participants had to answer every question before submission.

As the tool was linked to a live Google form, real-time analysis of the responses was done. In the final wrap-up session of the workshop, those questions where the proportion of correct responses were below 90% were again discussed, and the facilitators clarified all doubts of the participants.

To capture the difference in the formulation and the mode of administration of the two types of RVV, question numbers 2, 3 and 4 were different between the questionnaires used in the Guwahati and Pune training of trainers (ToTs).

Training of Trainers

In the year 2019, before the scale-up of RVV, a cascading approach was adapted to train health functionaries at all levels from regional through state and district to sub-district level. This model was conceptualized to complete the training of many healthcare workers in a short period of 2 months. To train the master trainers at the regional and state level, a national pool of facilitators, comprising senior officials from the MoHFW, immunization partner representatives, and senior faculties from academic or research institutions, was formed. A total of 7 ToTs were conducted at different geographical zones, and a pool of 778 master trainers was created. Out of these 7 ToTs, 4 were conducted to introduce lyophilized RVV at Pune, Hyderabad, Ahmedabad and Kolkata, while three ToTs were held at Guwahati, Chandigarh, and Raipur for the introduction of liquid frozen RVV. The training modules were appropriately customized for the introduction of the two different of RVVs.

The master trainers trained at the regional/state level facilitated the state, district and sub-district training. In these ToTs, the focus was on the capacity building of the participants. Each ToT was accompanied by assessing the participants’ knowledge both before and after the technical sessions through a pre- and post-test questionnaire.

Study Participants and Sampling Technique

This study was conducted amongst participants attending a state-level ToT in Pune and another group of participants attending a regional ToT in Guwahati to introduce RVV in their respective regions. The sampling technique used in the study was the complete enumeration method. Table 1 depicts the demography of the participants in the two ToTs included in the study.

ToTs = Training of trainers

The Pune state ToT participants included state and district level government officials, including medical officers from government health facilities, pediatricians, supply chain managers and partner representatives from Maharashtra.

In the regional ToT at Guwahati, the participants were only state-level government officials, supply chain managers, and partner representatives from the seven states, namely Arunachal Pradesh, Meghalaya, Nagaland, Manipur, Mizoram, Sikkim and Bihar.

All participants in the two workshops who participated in both the pre- and post-test, respectively were included for the study. The participants who participated only in the pre-test or only in the post-test were excluded from the study.

Statistical Analysis Plan

As the Google form also captured each participant’s name and mobile number to create the unique field, for each workshop, a database of participant-wise responses in pre- and post-test was available. To calculate the pre- and post-test scores, correct and incorrect responses to each question were assigned scores of 1 and 0, respectively. The pre- and post-test results were then plotted in a simple bar graph to compare the difference in the proportion of question-wise correct responses. Then, to understand whether these differences in correct responses in each question and the overall difference in correct responses were statistically significant, three tests of significance were done. These tests were administered separately on the data compiled for each of the two ToTs. Statistical package for the social sciences (SPSS) version 20.0 (Chicago, SPSS Inc.) was used for the final analysis.

The number of correct responses of the participants before and after the workshop, captured through the pre- and post-test, was compared for statistical significance using McNemar’s test. As the scores could not be assumed to be normally distributed, Wilcoxon signed-rank test was administered to test the significance of the change in the participants’ level of knowledge, as measured by the median total score. We also found several ties in scores between the pre- and post-test, which may potentially dilute the comparison of the median score, as demanded by the Wilcoxon signed-rank test. Hence, paired t-test was applied to further validate the results.

RESULTS

The ToTs were 2-day workshops with a structured and standardized agenda and a customized adult learning-based training methodology. A total of 84 and 60 participants attended the ToTs at Pune and Guwahati, respectively. However, the total number of participants who completed both the tests was 59 and 53 participants in the Pune and Guwahati workshops.

Regional ToT at Guwahati

In the regional ToT held in Guwahati, it was found that the knowledge about doses of RVV in one vial (77%), inadequate dosing (28%), vaccine vial monitor (VVM) (68%), phasing in of RVV in the immunization schedule (74%) and full immunization incentive for ASHA (68%) were the least amongst the participants. Table 2 shows the difference in the participants’ level of knowledge before and after completing the ToT. It also shows whether the difference in the knowledge level is significant or not. The analysis of post-test data revealed that the proportion of correct responses increased to more than 90% in all the questions except that on inadequate dosing (66%).

RVV = Rotavirus vaccine; ASHA = Accredited social health activist; VHND = Village health and nutrition day; RI = Routine immunization; OPV = Oral poliovirus vaccine; fIPV = Fractional inactivated poliovirus vaccine.

The knowledge regarding doses of RVV in one vial (77.4%), inadequate dosing (28.3%), VVM
(67.9%), open vial policy (86.8%), phasing in of RVV in the immunization schedule (73.6%) and full immunization incentive for ASHA (67.9%), increased significantly to 100%, 66%, 100%, 100%, 100% and 96.2%, respectively.

Table 3 shows the change in the level of knowledge for each question before and after completing the training workshop. Paired t-test revealed a statistically significant increase in knowledge following the completion of the workshop in knowledge around doses of RVV in one vial (p < 0.005), inadequate dosing (p < 0.005), VVM (p < 0.005), open vial policy (p < 0.05), phasing in of RVV (p < 0.005) and full immunization incentive for ASHA (p < 0.005).

RVV = Rotavirus vaccine; ASHA = Accredited social health activist.

Table 4 shows the change in the overall level of knowledge before and after completing the training workshop. Wilcoxon signed-rank test revealed a statistically significant increase in knowledge following the completion of the workshop, Z = -5.779, p < 0.05, with a large effect size (r = 0.94). The median knowledge score increased from 8 to 10 after the training workshop.

State ToT at Pune

In the state ToT held in Pune, it was found that the knowledge about schedule and dose of RVV (66.1%), doses of RVV in one vial (69.5%), rotavirus diarrhea prevention (74.6%) and VVM (74.6%) were the least amongst the participants. Table 2 shows whether, for each of the questions, the difference in the level of knowledge of the participants before and after the completion of the ToT was significant or not. After the completion of all technical sessions in the workshop, the data captured in the post-test revealed that the proportion of correct responses increased to more than 90% in all the questions except the ones on rotavirus diarrhea prevention (88.1%) and full immunization incentive for AHSA (78%).

The knowledge regarding doses of RVV in one vial (69.5%), bundle approach for each RVV vial (79.7%), schedule and dose of RV (66.1%), storage temperature for RVV (96.6%), VVM (74.6%), open vial policy (88.1%), delivery of RVV (93.2%), and phasing-in of RVV in the immunization schedule (78.0%), increased significantly to 100%, 94.9%, 100%, 100%, 100%, 100%, 100% and 98.3%, respectively.

Table 3 shows the change in the level of knowledge for each question before and after completing the training workshop. Paired t-test revealed a statistically significant increase in knowledge following the completion of the workshop in knowledge around doses of RVV in one vial (p < 0.005), schedule and dose of RVV (p < 0.005), VVM (p < 0.005) and open vial policy (p < 0.05).

Table 4 shows the change in the overall level of knowledge before and after completing the training workshop. Wilcoxon signed-rank test revealed a statistically significant increase in knowledge following the completion of the workshop, Z = -4.996, p < 0.05, with a large effect size (r = 0.73). The median knowledge score increased from 9 to 10 after the training workshop.

DISCUSSION

This study was conducted to assess the effectiveness of ToTs in increasing the awareness and practical knowledge of the participants. A comparison of the pre- and post-test scores on all questions demonstrated an increase in correct responses from 78% to 95% in the Guwahati workshop and an average increase from 81% to 96% in the Pune workshop. The post-test average correct responses were similar in both the workshops, although the participant profile was different in state-level program managers in Guwahati and sub-state or district level program managers in the Pune workshop. The increase in the knowledge level in both workshops is statistically significant (p < 0.05). Uskun et al, in their study, concluded that training on immunization increased the knowledge of primary healthcare workers and the vaccination coverage in the study region. The results of this study showed that the technical sessions during the training led to a significant increase (p < 0.01) in the health workers’ knowledge of immunization. It was also seen that the vaccination coverage increased significantly (p < 0.001) over 3 months post the training-based intervention.¹¹ Earlier studies have also established the positive effects of a training intervention, showing an increased score of the post-test compared to the pre-test.^12,13

While conducting a capacity-building workshop helps in familiarizing the participants with the various aspects and domains of the new vaccine introduction, at the same time, monitoring the effectiveness of the technical sessions in enhancing the actual knowledge of the participants is an important activity during the workshop. There is a dearth of studies to assess the impact of training the cadre of supervisors or future master trainers. The WHO conducted a study to evaluate the training system and the processes followed in certain good and bad performing states in terms of immunization rates. In the study by WHO, Das et al suggested that the quality of the training in terms of large pools of facilitators, prior knowledge, training methodology and trainees’ feedback were found to be salient enabling factors in such trainings.¹⁴

The overall study emphasizes the importance of a knowledge assessment format for the training in public health. On the one hand, the pre-test individually is important for both the participants and the facilitators of the training: (i) to intimate the participants towards the various topics to be covered during the training, (ii) to highlight in their minds those topics beforehand in which they scored less in the pre-test, (iii) for facilitators to adopt a more interactive and easier approach towards the same topic where the scores were less during the pre-test. The post-test, on the other hand, individually imparts confidence among the participants when they score well. Together, the pre-test and post-test, when viewed comparatively, gave the actual picture about the quality of the training.

An advantage of this study is that it methodologically deployed an ensemble approach by applying the three most appropriate tests of significance in a hierarchical way to assess the trainees’ actual difference between pre- and post-training knowledge status. Though each method suffers from its limitations, this ensemble approach gives more confidence to the findings.

The results of this study show that after the intervention of a 2-day training, significant improvement is seen in the knowledge of the participants while answering the post-test as compared to the pre-test. However, the retention of this knowledge gain is not assessed after a long gap, such as 6 months or 1 year. Similar studies have shown a progressive decline in the mean scores of the participants after 3 or 6 months of the intervention. Though a decline in knowledge is seen after the said time, the then scores were still higher than those before the training intervention.^15,16

To create data-based evidence of the results of a training intervention, a periodic and systematic follow-up knowledge assessment after 3 months,¹⁷  6 months,¹⁸ 9 months,¹⁹ 12 months and 24 months²⁰ should be included in planning such interventions. Also, it has been seen that for short-term results, a single intervention of continuing education can be useful, but for long-term sustainability and effectiveness, additional interventions to address health system gaps and community issues should be planned.²¹

A limitation of this study is that it is a complete enumeration of the participants attending a particular workshop, and hence it may lack external generalizability. As the tests were done for “on the spot”, immediate training needs assessment and post-session feedback and clarification. Our study also could not capture temporal change in participant’s knowledge.

CONCLUSION

This study signifies and highlights the effectiveness of a pre-planned and well-designed knowledge assessment tool before and after a training workshop in bridging the knowledge gap and improving the practical approach needed for new vaccine introduction. The findings signify that the pre- and post-test tool can be effectively used to capture the participants’ pre-training knowledge levels, suggest the key areas to focus during the training, and finally test the knowledge enhancement post-training. The post-test, being the last event of training, allows reinforcing the key take-home points. This study recommends that the same model of knowledge assessment through live pre- and post-test approach should be used in all the trainings of public health. It will help make an accurate real-time assessment of the participant’s knowledge and make real-time corrections or modifications in the focus of the training to ensure the best outcomes in terms of knowledge and practical skills gained.

References

1. Nair NP, Reddy NS, Giri S, Mohan VR, Parashar U, Tate J, et al; Investigators of the Rotavirus vaccine Impact Surveillance Network, Kang G. Rotavirus vaccine impact assessment surveillance in India: protocol and methods. BMJ Open. 2019;9(4):e024840.
2. Koshal SS, Ray A, Mehra R, Kaur A, Quadri SF, Agarwal P, et al. Partnering for rotavirus vaccine introduction in India: a retrospective analysis. Vaccine. 2021;39(44):6470-6.
3. Malik A, Haldar P, Ray A, Shet A, Kapuria B, Bhadana S, et al. Introducing rotavirus vaccine in the Universal Immunization Programme in India: from evidence to policy to implementation. Vaccine. 2019;37(39):5817-24.
4. Stockwell MS, Irigoyen M, Martinez RA, Findley S. How parents’ negative experiences at immunization visits affect child immunization status in a community in New York City. Public Health Rep. 2011;126 Suppl 2(Suppl 2):24-32.
5. Al-lela OQB, Bahari MB, Salih MRM, Al-abbassi MG, Elkalmi RM, Jamshed SQ. Factors underlying inadequate parents’ awareness regarding pediatrics immunization: findings of cross-sectional study in Mosul-Iraq. BMC Pediatr. 2014;14(1):1-7.
6. Anastasi D, Di Giuseppe G, Marinelli P, Angelillo IF. Paediatricians knowledge, attitudes, and practices regarding immunizations for infants in Italy. BMC Public Health. 2009;9:463.
7. Pelly LP, Pierrynowski Macdougall DM, Halperin BA, Strang RA, Bowles SK, et al. THE VAXED PROJECT: an assessment of immunization education in Canadian health professional programs. BMC Med Educ. 2010;10:86.
8. Nikula AE, Rapola SPT, Hupli MI, Leino-Kilpi HT. Factors strengthening and weakening vaccination competence. Int J Nurs Pract. 2009;15(5):444-54.
9. DeCorby-Watson K, Mensah G, Bergeron K, Abdi S, Rempel B, Manson H. Effectiveness of capacity building interventions relevant to public health practice: a systematic review. BMC Public Health. 2018;18(1):684.
10. Shivaraju PT, Manu G, Vinaya M, Savkar MK. Evaluating the effectiveness of pre- and post-test model of learning in a medical school. Natl J Physiol Pharm Pharmacol. 2017;7(9):947-51.
11. Uskun E, Uskun SB, Uysalgenc M, Yagiz M. Effectiveness of a training intervention on immunization to increase knowledge of primary healthcare workers and vaccination coverage rates. Public Health. 2008;122(9):949-58.
12. Marotta C, Raia DD, Ventura G, Casuccio N, Dieli F, D’Angelo C, et al. Improvement in vaccination knowledge among health students following an integrated extra curricular intervention, an explorative studyin the University of Palermo. J Prev Med Hyg. 2017;58(2):E93.
13. Brown VB, Oluwatosin OA, Ogundeji MO. Impact of training intervention on immunization providers’ knowledge and practice of routine immunization in Ibadan, south-western Nigeria: a primary health care experience. PAMJ. 2017;26:216.
14. Das JK, Sosler S, Bhattacharya M, et al. Evaluation of Immunization Training of Medical Officers, Cold Chain Handlers and Technicians. 2013.
15. Anderson GS, Gaetz M, Masse J. First aid skill retention of first responders within the workplace. Scand J Trauma Resusc Emerg Med. 2011;19:11.
16. Amaral F. Retention of knowledge and clinical skills by medical students: a pro-spective, longitudinal, one-year study using basic pediatric cardiology as a model. Open Med Educ J. 2013;6(1):48-54.
17. Monod C, Voekt CA, Gisin M, Gisin S, Hoesli IM. Optimization of competency in obstetrical emergencies: a role for simulation training. Arch Gynecol Obstet. 2014;289(4):733-8.
18. Sutton RM, Niles D, Meaney PA, Aplenc R, French B, Abella BS, et al. Low-dose, high-frequency CPR training improves skill retention of in-hospital pediatric providers. Pediatrics. 2011;128(1):e145-51.
19. Homaifar N, Mwesigye D, Tchwenko S, Worjoloh A, Joharifard S, Kyamanywa P, et al. Emergency obstetrics knowledge and practical skills retention among medical students in Rwanda following a short training course. Int J Gynaecol Obstet. 2013;120(2):195-9.
20. Yang CW, Yen ZS, McGowan JE, Chen HC, Chiang WC, Mancini ME, et al. A systematic review of retention of adult advanced life support knowledge and skills in healthcare providers. Resuscitation. 2012;83(9):1055-60.
21. Dieleman M, Gerretsen B, van der Wilt GJ. Human resource management interventions to improve health workers’ performance in low and middle income countries: A realist review. Heal Res Policy Syst. 2009;7(1):1-13.