The state of Illinois obstetric hemorrhage project: pre-project and post-training examination scores.

Abstract Objective: The Illinois Department of Public Health mandated that all clinicians who provide care to obstetric patients participate in the Illinois Obstetric Hemorrhage Project. The aim of the current report is to describe change in knowledge among providers engaged in the project, as assessed by pre- and post-tests. Methods: The project, implemented 2008 to 2010, included four components: a written 25-item multiple-choice examination (pre-test), a didactic lecture, skill stations (for teaching blood loss estimation), and a simulation drill and debriefing. Participants completed a post-test 6 months later. Pre- and post-test examination scores were compared. Results: Data from 95 hospitals are included in this analysis (9456 paired test results). The proportion of participants who scored ≥88% correct answers increased from 10.9% on the pre-test to 49.1% on the post-test (p < 0.0001). Registered nurses made greater improvements in test scores than anesthesia and obstetric providers (p < 0.0001). Conclusions: The Illinois Obstetric Hemorrhage Project was successful in improving knowledge of obstetric hemorrhage in a large number of providers with different expertise and experience levels. Further long-term study is essential to determine whether the skills acquired during the Project contribute to improved obstetric hemorrhage outcomes for the women of Illinois.


Introduction
Hemorrhage is a leading cause of maternal morbidity and mortality worldwide [1]. The incidence of hemorrhage appears to be increasing [2], and even in high-income countries such as the United States, maternal hemorrhage contributes significantly to maternal mortality. Disturbingly, death from maternal hemorrhage in the United States and other high-resource countries is often judged to be preventable [3][4][5][6].
In 2000, the State of Illinois and the Illinois Department of Public Health (IDPH) created the Illinois Maternal Mortality Review Committee (MMRC) to conduct reviews of pregnancy-related deaths within 1 year of pregnancy [7]. The Committee reviewed all cases of death from obstetric hemorrhage between 2000 and 2006. Among the 45 deaths that underwent complete review, 11 were secondary to hemorrhage; 100% of these deaths were judged potentially avoidable by the Committee [7]. With the hope of reducing the incidence of maternal morbidity and mortality secondary to hemorrhage, the Committee developed the Illinois Obstetric Hemorrhage Project (OBHEP) [7].
The IDPH mandated that all facilities in the State of Illinois that provided childbirth services, and every health care provider who practiced in these facilities and routinely provided care to obstetric patients, complete the project. Training for individual providers consisted of a pre-test, training in the recognition and treatment of obstetric hemorrhage, and 6 months later, a post-test. The aim of the current report is to describe the implementation of the OBHEP project and to report on change and retention in knowledge among providers (physicians, registered nurses, advanced practice nurses), as assessed by the pre-and post-tests.

Methods
The initial phase of the Obstetric Hemorrhage Education Project was implemented from July 2008 to December 2009. One million dollars for the project were allocated by the IDPH; each perinatal center received $100 000 to distribute to their network hospitals. The goal of the program was to improve (1) risk assessment and preparation for possible hemorrhage, (2) estimation of blood loss skills, and (3) recognition and treatment of hemorrhage and hypovolemia. The program curriculum was developed by a work group of the MMRC (two maternal fetal medicine specialists, a generalist obstetrician-gynecologist, an advanced practice nurse, three perinatal network administrators [nurses with expertise in perinatal nursing] and three perinatal educators). The program included four components and was designed to be completed in a 4-h time frame: a written benchmark examination (pre-test) (30 min), a didactic lecture (90 min), skill stations (30 min), and a simulation drill (30 min) followed by debriefing (60 min). Based on hospital resources, facilities were also required to develop an obstetric hemorrhage policy and rapid response team, and incorporate hemorrhage reviews as a part of their quality processes. The program's educational material, including didactic material, instructions for setting up the skill stations, and simulations, is available through the Illinois Department of Public Health website (www.dph.illinois.gov).
The program was administered through Illinois's Perinatal Network system. Each hospital was asked to identify physician and nurse ''champions'' to lead the project in each institution. Two ''train-the-trainer'' sessions were held to review the instructional material. The content of the didactic lecture was standardized; however, the presenter and format varied among institutions, and included grand rounds, videotape and web-based presentations. Not all institutions completed the required material in a single 4-h session. Topics covered in the didactic presentation included case presentations, cardiovascular physiologic changes of pregnancy and the purpureum, recognition of obstetric hemorrhage and accurate estimation of blood loss (including visual estimation and gravimetric methods), fluid management, indications for uterotonic medications, an overview of procedures used to treat hemorrhage (e.g. B-Lynch technique, uterine artery ligation or embolization), and the importance of the team approach to management.
The skill stations were developed to teach accurate visual estimation of blood loss using materials commonly used to absorb blood during childbirth, including perineal pads, disposable underpads, laparotomy sponges and 4 Â 4 inch gauze squares. The 10 skill stations were prepared using predetermined volumes of imitation blood and clots. Participants were asked to view the stations and estimate blood loss, and were then told the actual blood loss for each station. Finally, standardized simulation cases were prepared by the MMRC workgroup, along with instructions and assessment checklists for the simulation drills. Drills were led by the team leaders in the individual institutions.
The purpose of the written pre-test was to assess participants' baseline knowledge before the education intervention. A draft of the written test was reviewed for face validity by a group of subject matter experts consisting of 10 maternalfetal medicine specialist co-directors of perinatal centers with current clinical activity on labor and delivery units and 10 perinatal administrators. Unclear or confusing questions were rewritten. Members of the MMRC assigned the test questions to one of three categories: risk and assessment, diagnosis, and management. The final examination consisted of 25 multiple-choice questions with one correct answer and three distractors. The same test (post-test) was repeated 6 months after the completion of the program. Each hospital was given the option to use a web-based or classroom format to administer the test. All hospitals were required to complete the post-test by December 2010.
Each hospital was responsible for administering the test in a format that did not compromise the integrity of the questions. It was the hospital's responsibility to ensure that the participants took the test prior to participating in the didactic, skill stations and simulation components. The hospital was also charged with keeping a record of the individual pre-test and post-test scores. The examinations were graded locally, reported on an Excel spreadsheet, and sent to the perinatal centers. The participants were informed of their scores, but were not allowed to keep a copy of the test. Providers who practiced at more than one institution were required to complete the pre-test, attend the didactic lecture and review the skills stations once. They were required to complete the simulation drill at each hospital in which they provided services.
The following data were collected for each provider who completed the hemorrhage project: provider type/specialty, level of training, raw pre-training and post-training examination scores (number of correct answers), primary practice hospital, and hospital level of maternity care (1 [low risk], 2, or 3 [high-risk requiring complex care]). The participants selfidentified themselves into one of the following 12 categories: attending anesthesiologist, anesthesiology resident, certified registered nurse anesthetist (CRNA), attending family medicine physicians, family medicine resident, maternal-fetal medicine specialist, attending obstetrician, obstetrics/gynecology resident, certified registered nurse midwife or advanced practice nurse, labor and delivery or antepartum registered nurse, postpartum registered nurse, or other (see below).
Hospitals were asked to list nurses on their final reporting grids in the area where they spent most of their time. Additionally, some hospitals tested clinical nurse specialists, nurse managers and supervisors, perinatal center staff and other educators. While these employees do not usually provide direct patient care, they were included in this training program because as leaders and educators they should understand the procedures and drills necessary for the education and treatment of maternal hemorrhage. These providers make up the ''other'' category, along with intensive care unit, general operating room and emergency department staff members (who often contribute to the management of maternal hemorrhage in smaller hospitals).
Inclusion criteria for the current analysis included perinatal networks that documented provider type and test scores. Networks were excluded from participation if there was incomplete documentation, or providers were allowed to take the pre-and post-tests more than once. The Northwestern University Institutional Review Board deemed that the analysis and publication of the data did not require IRB review because the project did not meet the definition of research on human subjects.

Statistical analysis
We hypothesized that the overall percent of post-test scores !88% would be greater than the proportion of pre-test scores !88%. The percent of providers who answered 88% or more questions correctly on the pre-and post-tests (22 of 25 correct answers, ''passing grade'') as well as the percent correct answers for the pre-test and post-test for each of the 12 categories of providers are reported. Within categories, the difference in percent of providers who scored 88% or greater correct answers was compared for the pre-and post-tests with a t-test. The cut-off of 88% was chosen after scoring the pretest because this value corresponded to the median pre-test score for maternal-fetal medicine specialists, considered the gold standard for testing due to their in-depth knowledge of maternal hemorrhage.
Additionally, providers were grouped by specialty (anesthesiology, obstetrics, and nursing), and by training and experience ((1) experienced providers: attending and fellow physicians, and certified nurse midwives and registered nurse anesthetists; (2) providers in training: resident physicians; and (3) registered nurses; categories chosen a priori). The improvement in test scores was compared among groups using a one-way, two-sided ANOVA test, followed by post-hoc comparisons between groups (Bonferroni's multiple comparison test). Test scores were also compared among hospital levels (1, 2 and 3). A p value50.05 was considered significant.
The internal consistency of the test was assessed with Cronbach's a using a subset of all of the pre-and post-tests from a single level 3 hospital for which complete data were available for individual questions (n ¼ 126 examinations). Item difficulty (proportion of test takers who correctly answered the question) and discrimination (using pointbiserial correlation) were calculated for each test question. All analyses were conducted using SPSSv 22 (Chicago, IL) and GraphPad Prism v6 (La Jolla, CA).

Results
Nine of the 10 perinatal networks participated in the analysis. Data from one network were excluded from the analysis because scores were recorded after reviewing the correct answers as a group; original scores were not recorded. Within the nine perinatal centers, 121 hospitals participated in the initial project; however, 11 units were lost to follow-up and an additional 15 hospitals were omitted due to various reasons such as self-grading the test. Data from 1379 providers were not included in the analysis because only pre-or post-test scores were available (14.6% of total). The most common reason for missing scores was resignation or retirement. Other reasons included termination, graduation, leave of absence, new hire, or testing at another hospital without an identified match. The reason for missing data was not documented for approximately one-third of the missing text scores. Paired pre-and post-examination results were available for 9456 providers from 95 hospitals.
The mean (SD) number of correct answers for the pre-and post-test for all providers were 16.8 (3.7) and 20.7 (3.7), respectively. The percent correct answers and the percent of providers with pre-and post-examinations passing scores are presented in Table 1. The frequency of all providers with passing scores increased from 10.9% on the pre-test to 49.1% on the post-test (p50.0001). The 95% confidence interval of the paired pre-and post-test difference in percent passing scores (!88% correct) was 22.5 to 23.6%. The percent of providers with correct scores !88% was higher on the postcompared to pre-test for all 12 provider categories (p 0.02 for all categories).
The improvement in percent correct answers between the pre-and post-test scores by provider type and training level are presented in Table 2. Registered nurses made the greatest improvement in scores compared with attending physicians/advanced practice nurses and resident physicians. Anesthesia providers (physicians, residents, and certified registered nurse anesthetists) made greater improvements than obstetric providers (physicians residents, midwives and advanced practice nurses). Resident physicians scored significantly lower than fully trained providers (attending physicians, fellows, and advanced practice nurses) and registered nurses on the post-test (p50.0173). Almost 50% of providers working at level 2/2+ and level 3 hospitals scored !88% the post-test compared to 43% of providers at level 1 hospitals (

Discussion
The important finding of this analysis of the Illinois Obstetric Hemorrhage Project pre-and post-project training assessment is that the proportion of providers who scored greater than or equal to 88% correct answers was dramatically greater 6 months after training. Thus, the knowledge gained during the training was retained for at least half a year. Scores improved for all types and categories of providers; therefore, we concluded that the project was successful in improving knowledge of obstetric hemorrhage for providers of several specialties and levels of experience and training.
Resident physicians scored significantly lower than attending physicians on both the pre-and post-tests, emphasizing that trainees must have adequate supervision when they provide obstetric care. Nurses made the greatest gains in test scores, starting with lower scores than fully-trained physicians on the pre-test, but similar scores on the post-test. Providers from smaller hospitals did less well than those from larger urban hospitals. This finding may reflect the system in which high-risk patients are appropriately referred to level 2 and 3 hospitals, and providers from these hospitals care for women with obstetric hemorrhage more often. Finally, the pre-test scores were surprisingly low, especially for some provider groups. These low scores suggest that a significant knowledge gap existed prior to the intervention, and that filling this gap was a necessary intervention toward improving outcomes.
Many studies describe late recognition of obstetric hemorrhage [4,5]. Pregnant women are usually young and healthy, and blood volume is increased by approximately 50% during pregnancy. Therefore, women may lose a significant amount of blood before they become symptomatic. Late recognition and underestimation of blood loss are important factors contributing to poor outcomes. Improved recognition of blood loss and hemorrhage are the first steps to improve morbidity and mortality rates. Although test scores are an intermediate outcome, the analysis of the examination results was critical to the initial assessment of this project. Without an improvement and retention in knowledge of recognition and treatment of postpartum hemorrhage, it is unlikely that improvements in the important outcomes will be observed. *Improvement in test scores represents the mean difference in the pretest and post-test scores in terms of percent correct answers. yAnesthesia providers include MD anesthesiologists, residents, and certified registered nurse anesthetists (CRNA). Obstetric providers include obstetricians, residents, advanced practice nurses (APN) and midwives. Registered nurses include antepartum, labor and delivery, and postpartum nurses as well as other nurses (e.g. operating room, emergency department). Family medicine providers include attending physicians and residents. zFully licensed providers includes all attending physicians, fellows, and mid-level providers (CRNA and midwives/APN), including ''other'' category providers. Resident physicians include all residents from all specialties. Registered nurses include antepartum, labor and delivery, and postpartum nurses as well as other nurses (e.g. operating room, emergency department). ôPost-hoc tests: p value for all post-hoc comparisons between groups 0.0002. All post-hoc p values were corrected for multiple comparisons using the Bonferroni method. xPost-hoc tests: fully licensed providers versus resident physicians (p ¼ 0.0173); fully licensed providers versus registered nurses (p50.0001); resident physicians versus registered nurses (p50.0001).
All post-hoc p values were corrected for multiple comparisons using the Bonferroni method. The assessment also served as a means of evaluating whether individual perinatal centers had actually taught the material in an effective fashion. The requirement for documentation of examinations scores by all providers sent an important message to hospital administrators that the project was important and required substantial support from hospital leaders.
Overall, the test appeared to be of appropriate difficulty. The finding that test scores improved after the Hemorrhage Project suggests that the test had appropriate content validity. The ideal item difficulty for a 4-response multiple-choice test for maximum discrimination potential is 0.62 [8]. Although questions with high item difficulty may not discriminate among test takers, they do indicate widespread knowledge of the concept being tested. High item discrimination values (values closer to 1) indicate a positive relationship between the correct answer to an item and total examination score. Our post-test's item discrimination values indicated that item discrimination was generally good (values between 0.30 and 0.39) to very good (values greater than 0.40) [8]. The Cronbach a value indicated an acceptable level of test reliability and internal consistency.
There are some limitations to our data and conclusions. Although the Maternal Mortality Review Committee attempted to standardize the core material taught in the education project, we could not assess the quality of the teaching, simulations, or debriefing sessions at individual hospitals. Thus, learning may have been uneven. We cannot verify that the test was administered in a consistent fashion to all examinees. The test was written by the workgroup, and modified after experts took the test. However, we did not formally test content, criterion, or construct validity of the examination. We chose an arbitrary ''passing'' rate of 88% because this was the median score of the maternal fetal medicine physicians who took the test. The proportion of maternal fetal medicine specialists who scored !88% on the post-test was only 70%. One expects this group of physicians to maintain expertise on the subject matter of obstetric hemorrhage and score better on the test. Thus, this finding suggests that some of the questions, although piloted before administration, may still have been confusing, or that the didactic interventions fell short of communicating some important information. The pre-and post-tests were identical; thus, some of the observed improvement might have been due to a practice effect.
We arbitrarily chose a post-project assessment time of 6 months. We do not know how fast knowledge decay occurs, or how often training should be repeated in order to maintain the knowledge and skills taught by the Project. At the recommendation of the MMRC, the Illinois Department of Health is requiring providers who care for obstetric patients to participate in obstetric hemorrhage training, including the benchmark assessment, every 2 years. However, our data suggest that providers who have less experience with postpartum hemorrhage, such as providers from Level 1 centers, or resident physicians, may benefit from more frequent refresher training. For example, resident physicians could participate in ''just-in-time'' refresher training before beginning a rotation on the obstetric service. A system of routine review of severe maternal mortality, as suggested by Kilpatrick et al., may identify providers or institutions that will benefit most from refresher training, thus limiting the need and cost of unnecessary training for experience clinicians [9].
An additional limitation of the study was that there was no control group that did not have exposure to the training intervention. Finally, the test scores are an intermediate outcome. The most important outcomes are related to a reduction of morbidity and mortality from obstetric hemorrhage.
Despite these limitations, the members of the State of Illinois Maternal Mortality Review Committee are highly encouraged by the results of the assessment. Over 12 000 providers in the State of Illinois participated in the Obstetric Hemorrhage Project. Anecdotally, many providers have communicated the usefulness of the project to committee members. The committee continues to monitor long-term outcomes, including important markers of morbidity (e.g. severe hemorrhage, blood product transfusion, postpartum hysterectomy), and maternal death from hemorrhage.