Identification of Non-Fatal Opioid Overdose Cases Using 9-1-1 Computer Assisted Dispatch and Prehospital Patient Clinical Record Variables

Abstract Background: The current epidemic of opioid overdoses in the United States necessitates a robust public health and clinical response. We described patterns of non-fatal opioid overdoses (NFOODs) in a small western region using data from the 9-1-1 Computer Assisted Dispatch (CAD) record and electronic Patient Clinical Records (ePCR) completed by EMS responders. We determined whether CAD and ePCR variables could identify NFOOD cases in 9-1-1 data for intervention and surveillance efforts. Methods: We conducted a retrospective analysis of 1 year of 9-1-1 emergency medical CAD and ePCR (including naloxone administration) data from the sole EMS provider in the response area. Cases were identified based on clinician review of the ePCR, and categorized as definitive NFOOD, probable NFOOD, or non-OOD. Sensitivity, specificity, positive and negative predictive values (PPV and NPV) of the most prevalent CAD and ePCR variables were calculated. We used a machine learning technique—Random-Forests (RF) modeling—to optimize our ability to accurately predict NFOOD cases within census blocks. Results: Of 37,960 9-1-1 calls, clinical review identified 158 NFOOD cases (0.4%), of which 123 (77.8%) were definitive and 35 (22.2%) were probable cases. Overall, 106 (67.1%) received naloxone from the EMS responder at the scene. As a predictor of NFOOD, naloxone administration by paramedics had 67.1% sensitivity, 99.6% specificity, 44% PPV, and 99.9% NPV. Using CAD variables alone achieved a sensitivity of 36.7% and specificity of 99.7%. Combining ePCR variables with CAD variables increased the diagnostic accuracy with the best RF model yielding 75.9% sensitivity, 99.9% specificity, 71.4% PPV, and 99.9% NPV. Conclusion: CAD problem type variables and naloxone administration, used alone or in combination, had sub-optimal predictive accuracy. However, a Random Forests modeling approach improved accuracy of identification, which could foster improved surveillance and intervention efforts. We identified the set of NFOODs that EMS encountered in a year and may be useful for future surveillance efforts.

While relatively reliable data on fatal OODs are available via federal and state-level mortality monitoring systems, accurately enumerating NFOODs has proven more difficult. Common sources of data for monitoring NFOODs include hospital admissions and emergency department (ED) visits (5). However, studies have raised concerns about the accuracy of using ED data for surveillance purposes: Rowe et al. examined ED visits from 2012 to 2014 in San Francisco, with the aim of validating the use of International Classification of Disease Ninth Revision (ICD-9) codes for identifying opioid overdose events (5). The use of opioid-poisoning ICD-9 codes to identify overdose cases had low sensitivity (25%); addition of clinically relevant ICD-9 codes (unspecified drug poisoning, general poisoning, opioid abuse, unspecified drug abuse) was only able to increase sensitivity to 56.8%, and the addition of clinically unrelated ICD-9 codes was required to increase sensitivity to 100% (5). This study suggests that using clinically relevant ICD-9 codes in hospital admissions and ED visits datasets may be insufficient for tracking NFOOD cases in the ED. In addition, hospital and ED data sources can only identify NFOOD cases that present to the ED, missing NFOODs that do not come to the attention of the emergency medical services (EMS) system (i.e., 9-1-1 is not called) or who refuse transport to the ED. In community-based research, only 23-63% of people who use drugs call 9-1-1 when they witness an overdose (6)(7)(8). Even when 9-1-1 is called, an estimated 12-15% of NFOODs that come to the attention of the EMS system refuse transport because of fear of withdrawal symptoms, harassment, discrimination, and arrest among other reasons (9)(10)(11) Studies early in the COVID-19 pandemic suggest that the rate of NFOOD patients refusing transport has increased (11,12), thereby further complicating efforts to enumerate NFOODs at a time when they may be increasing (13,14).
Another indicator commonly used for counting NFOODs is prehospital administration of naloxone by EMS personnel (15)(16)(17). Naloxone is an opioid antagonist used to reverse suspected OOD-induced respiratory depression. In addition to being the single most effective antidote to OOD-induced respiratory depression, naloxone is used as a diagnostic confirmation of NFOOD in some states. Because of its low risk profile, it may also be administered to patients with respiratory depression from other causes or to those with altered mental status of unknown origin, for example, in Ohio (18)(19)(20). Therefore, counts of naloxone administration alone can lead to over-counting of NFOOD cases (21,22). Additionally, NFOODs may be encountered by EMS providers that can be treated in the prehospital setting with respiratory support alone or without EMS interventions and may not require naloxone, thereby leading to under-counting of NFOOD cases (20,23,24). In one Rhode Island-based study, researchers used medical examiner data to identify factors associated with administration of naloxone as part of EMS resuscitation attempts finding patients had higher odds of receiving naloxone if they were at least 30 years old, males, and exhibited signs of drug injection and/or overt signs of overdose such as track marks and drug paraphernalia (25). Taken together, these data suggest that EMS-administered naloxone counts alone may overestimate or underestimate actual NFOOD cases, raising concerns about misclassification (15,17,(25)(26)(27). These findings led researchers in Rhode Island and Kentucky to include additional parameters such as narrative keywords, primary/secondary impressions, and positive response to naloxone administration in their algorithms to identify NFOOD cases in EMS data (10,28,29). However, these studies did not state the impact of additional parameters on the diagnostic accuracy of identifying NFOODs (10,28).
As part of a larger mixed methods project on the potential use of 9-1-1 dispatch data to identify NFOODs, and to determine the acceptability of deploying peer harm reduction intervention for the identified NFOODs (30), we quantitatively determined (1) the predictive accuracy of various 9-1-1 CAD and ePCR variables for identifying NFOODs, and (2) the accuracy of various indicators for predicting the number of NFOODs in specific geographic regions (e.g., census blocks). This undertaking was necessary because, at the time and to the best of our knowledge, no single "gold standard" for identifying NFOODs in 9-1-1 or EMS data existed, and there was interest by our research team and local partners in using these data for both surveillance and intervention efforts. To accomplish these aims, first, we arrived at the definitions for NFOODs using data from both electronic Patient Care Records (ePCR) and CAD data. Second, we described rates of NFOODs identified in the 9-1-1 data by day of the week, hours of the day, and demographic characteristics of the patients. Third, we assessed sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of selected variables from the 911 system. Fourth, we used a machine learning technique -Random Forests modeling-in an attempt to optimize our ability to accurately predict NFOOD cases within census blocks (30).

METHODS
Data for the current study were obtained for a mixed methods project (30). Based on our filter criteria, a third-party vendor created a composite quantitative dataset for our analysis which had additional variables that were not in ePCR but were present in CAD data. This was done by linking ePCR and CAD data using unique identifiers.
The quantitative component of the mixed methods project reported in this paper examined the feasibility of using 9-1-1 dispatch data to identify NFOODs. Our overarching goal was to optimize our ability to detect NFOODs early in the continuum of prehospital emergency care, and use this information to deploy peer harm reduction workers to the scene.

Background on 9-1-1 Systems Data
The study setting has a two-tiered response system for emergency medical calls. A 9-1-1 call is received at a primary Public Safety Answering Point (PSAP) where the call taker will determine if the person in need of service is requesting a police, medical or fire response. If a medical response is requested or needed, the caller is transferred to a secondary PSAP for Emergency Medical Dispatch (EMD) (31), a nationally standardized curriculum for handling medial 9-1-1 calls developed by National Highway Traffic Safety Administration (32). The secondary PSAP assigns the determinant code to the 9-1-1 call that is used in this analysis, provides post-dispatch and pre-arrival instructions to the caller, and communicates case information to the responding ambulance. Computer aided dispatch (CAD) TriTech software is used by the secondary PSAP, and the CAD field "problem type" represents the dispatcher's conclusion as to the nature of the call based on the information gathered on the phone. Upon arrival at the scene, the EMS responder documents additional information from the response in electronic patient care record (ePCR). The ePCR may be completed at the scene, but is often completed up to several hours after the response (33).

Study Setting
The study was conducted using de-identified CY 2016 9-1-1 data from mid-size county (population >460,000) in the Western United States that receives approximately 37,000 9-1-1 ambulance requests annually and has a single EMS provider. Data were received from FirstWatch, an organization that provides data analytics for EMS data. The dataset identified all medical emergency calls to 9-1-1 in the response area and comprised both CAD and ePCR variables. The Institutional Review Board (IRB) at the University of Nevada, Reno approved all study activities under protocol #1024876.

Measures
Information from 9-1-1 calls is documented in CAD variable fields. CAD variables used for this study include: date and time of call, CAD determinant (alphanumeric dispatch determinant code), CAD problem (problem type-CAD version), hour of day, day of week, age of patient, gender of patient, and CAD narrative text. For the purposes of this study, we searched the CAD narrative using a set of 127 opioid overdose-related keywords validated in research with mortality data (34), and a "CAD keyword" variable was populated with any text that matched that keyword list (e.g., oxycontin, narcotic, OOD, heroin), see methods supplement 1 https://figshare.com/ articles/online_resource/Supplementaryfile2_docx/ 16574798. Information about the clinical encounter is documented by EMS personnel in the ePCR. ePCR variables used for this study include: unique patient identifier (using non-identifiable information), chief complaint (what the patient says is wrong), primary impression (what the responder thinks is the problem), interventions (i.e., medication(s) administered, including naloxone), destination of transport, and vital signs relevant to the clinical identification of an NFOOD (level of consciousness [LOC], respiratory rate [RR], blood oxygen level), PCR nature of call (problem-type ePCR version). FirstLOC is the initial assessment of LOC while LastLOC is the second assessment of LOC. The ePCR also includes a space for text narrative, which was searched using the procedure described above and used to generate an "ePCR narrative keyword" variable.

Case Definition
NFOODs were identified through a hybrid manual and automated record review conducted by a physician (who is a former paramedic) and a second reviewer who is a current paramedic. The second reviewer reviewed a subset of the dataset, herein referred to as probable NFOODs. We had a third reviewer as the tiebreaker but there was no discordance in the review. To identify the set of possible NFOODs within the entire year's worth of 9-1-1 call data, a set of EMS agency specific NFOOD filters were created using FirstWatchV R software, the Zoll Data System software for ePCR and TriTech Inform CAD. A filter is a combination of variables and queries in the fields of the dataset. The filters were used to identify potential NFOODs within the larger set of emergency medical calls and, essentially functioned to rule out cases that could not possibly be considered a potential NFOOD. The filters were defined with decreasing levels of specificity.
Filter 1 identified all cases in which naloxone was administered and the ePCR contained documentation of improvement on RR, LOC, or pupil responsiveness. Filter 2 excluded cases identified in Filter 1 and identified any other cases in which naloxone administration was documented in the ePCR. Filter 3 excluded cases identified in Filters 1 and 2 and identified any cases in which a narrative keyword match was identified. Then, the clinical reviewer reviewed the set of cases identified by the filters, to determine whether cases met our definition of NFOOD. A case was determined to be a potential NFOOD if any of the following conditions were met: 1. within the documented narrative the clinician found any mention of the following: opioids on scene, paraphernalia, 2. witness on scene describes an OOD, 3. history of drug use, 4. administration of naloxone with improvement of LOC and/or RR, 5. improvement LOC and/or RR with stimulation (pain, voice), 6. improvement LOC and/or RR with rescue breathing and/or oxygen administration, 7. signs of asphyxia (bluish coloration).
The calls in which ePCR documented administration of naloxone and positive response to naloxone were classified as Definite NFOODs. Those without a clear confirmation through a documented positive response to naloxone in ePCR but which had a strong likelihood of being an opioid OD based on the clinician's judgment were classified as probable NFOODs. A call was determined to be a presumed non-OOD call if other clinical signs such as intoxication by other agent, signs of stroke, coronary, high/low blood sugar, high/low blood pressure, trauma convulsions, were found in the narrative and/or the vital sign section of the call.
The record reviewer manually reviewed all the cases identified in Filter 1 (n ¼ 106) and Filter 2 (n ¼ 107) and made a clinical confirmation of whether the case was a "definite NFOOD," "probable NFOOD," or "presumed non-OOD." Filter 3 yielded n ¼ 22,980 cases, which precluded a case-by-case manual review. Therefore, the reviewer further sorted this set of cases by using the EMS provider's documented impression of the call to narrow down the possible NFOOD calls along with the documentation of either an altered mental status (painful, unresponsive, verbal or not documented), a decrease in RR (RR less than 12), or an abnormality in pupil responsiveness (non-reactive, sluggish, or not documented). The field "provider impression" included the following entries: cardiac arrest, altered mental status, acute alcohol intoxication, alcohol abuse, drug reaction, drug withdrawal, fall NOS (not otherwise specified), hypoxemia, poisoning, respiratory arrest, respiratory distress, respiratory failure with hypoxia, shortness of breath, slurred speech, suicidal attempt, unconscious, unresponsive, acute respiratory failure, and Glasgow coma score. There were 2,037 cases that met those criteria, which were manually reviewed using the same process as described for calls that were identified through NFOOD filters (1) and (2). Additionally, manual review of 200 randomly selected calls which were not reviewed initially was performed to confirm that we had not inadvertently excluded any NFOODs using this sorting scheme. The rest of the calls identified by the filters that did not have any of the documented impressions were classified as non-OOD and no further review was performed.

Data Analysis
Data were analyzed using SAS 9.4 and R (35,36). Among the 9-1-1 calls, aggregate NFOOD (definite NFOOD þ probable NFOOD) cases per 1,000 calls by time of day and days of the week were computed and described. We describe demographic characteristics of definite and probable NFOOD and non-OOD cases.
For the purpose of computing the diagnostic accuracy parameters, definite and probable NFOODs were categorized as overdoses (NFOODs) while non-overdose cases and presumed non-OOD were categorized as non-opioid overdoses (non-OOD). Diagnostic accuracy parameters were determined for CAD variables, and combined CAD and ePCR variables .Then, we added selected CAD variables (CAD problem types). The CAD variables were selected based on clinical indication for naloxone administration (overdose/poisoning, breathing problem) and those associated with LOC (cardiac/respiratory arrest, unconscious/fainting, convulsions/seizures).
In Random Forests modeling (using the R Version 4.0.3 package "randomForest") (37), CAD variables (age of patient, gender of patient, time of day, day of week, CAD problem type), ePCR variables (chief complaint, nature of call, primary impression, patient's outcome, narrative keywords, pupillary reaction, level of consciousness, and number of naloxone doses), and naloxone administration were included to estimate the diagnostic accuracy parameters. Random Forests automatically detects nonlinearities and interactions among variables and has "tuning parameters" that tend to improve one diagnostic accuracy parameter (such as sensitivity) at the cost of another (such as specificity) when adjusted (37). For each set of tuning parameters, the number of predicted NFOODs was compared to the number of observed NFOODs in each census block, and the set with minimum average absolute error (absolute error is the absolute value of estimated number of NFOODs in the block by Random Forests minus the observed number of NFOODs in the block) was selected as the "best" Random Forests model. Additional (Methods) are available as supplemental material accompanying the online article https://figshare.com/articles/online_ resource/Supplementaryfile2_docx/16574798.  (Figures 1 and 2). NFOODs were prevalent among males and adults aged 24 À 64 years (Figure 3). Table 1 shows the diagnostic accuracy of selected CAD and ePCR variables. First, we examined the accuracy of variables available at the time of the 9-1-1 call. That is, variables from the CAD alone. The top half of Table 1 displays sensitivity, specificity, PPV, and NPV for the five most common CAD Problem Types assigned to NFOOD cases: cardiorespiratory arrest, overdose/poisoning, breathing problem, convulsion, and unconsciousness. The most sensitive indicator of NFOOD in the CAD Problem Type was cardiac/respiratory arrest (31.6%), followed by overdose/poisoning (26.6%), and unconscious/fainting (20.3%).

Our
Then, we examined the impact of adding variables available from the ePCR . That is, variables that only become available after paramedics respond to a call and complete a patient record. The second half of Table 1 displays sensitivity, specificity, PPV, and NPV for naloxone administration (from the ePCR data) and the CAD Problem Types when combined with naloxone administration. Overall, 241 ePCR records showed documentation of naloxone administration; 106 (44%) were identified as NFOOD cases and 135 (56%) were non-OOD cases. As a predictor of NFOOD, naloxone administration by EMS had 67.1% sensitivity, 99.6% specificity, 44% PPV, and 99.9% NPV. Adding EMS administered naloxone to the CAD Problem Type variables resulted in increases in all diagnostic accuracy parameters for all selected CAD problem type variables, including a doubling of the sensitivity for cardiac/respiratory arrest (67.1%). Notably, overdose/poisoning CAD Problem Type had only 26.6% sensitivity when used alone and 17.7% sensitivity when combined with naloxone administration from the ePCR. Table 2 shows results from the Random Forests modeling approach. The best Random Forests model using CAD variables alone achieved a sensitivity of 36.7% and specificity of 99.7%. The best RF model using ePCR variables reduced the number of false positives and increased PPV from 44% to 71.4%. Combining ePCR variables with CAD variables increased the diagnostic parameters with the best model yielding 75.9% sensitivity, 99.9% specificity, 71.4% PPV, and 99.9% NPV ( Table 2).
As a final step, we predicted the number of calls that could be expected per census block, using the three different RF models (CAD alone, naloxone alone, and CAD þ ePCR).
The "Best Model" uses four stages to determine if a call is an "NFOOD" or "non-overdose" based on the inputs of several variables. In the first stage, if FirstLOC is "Unresponsive" (or first pupil response "Non-reactive") and LastLOC is "Alert" (or last pupil response "Reactive"), then the Best Model predicts the call to be a NFOOD (there are 40 such calls in our sample, all of which are non-fatal overdoses). In the second stage, the Best Model predicts a call to be a non-overdose if the subject is under 16 years of age or has a    primary impression that contains any of the following words or phrases: "traumatic," "wound," "injury," "motor vehicle traffic," "alcohol," "pain," "diabetes," "anxiety," "collapse," and "fracture"; in our sample, there are 15,483 such calls, all of which are non-overdoses. In the third stage, naloxone administration and variables in both CAD and ePCR (128 total predictors) are used in a Random Forests model that has an estimated negative predictive value of 100%; this model is used to detect non-overdoses, and in our sample, 18,057 calls are predicted to be non-OODs after the third stage. For the calls that are undetermined after stage 3 (i.e., the calls that do not satisfy the conditions in stages 1 and 2 and are not predicted to be non-overdoses in stage 3), a Random Forests model using the same variables as in stage 3 (but using different tuning parameters than the model used in stage 3) is used to determine if each call is a NFOOD or non-overdose in the fourth and final stage. Our model depicts three potential scenarios for predicting NFOODs: Best model (ePCR and CAD); CAD only; naloxone only ( Figure 4). Figure 4 shows the accuracy and precision of predictions made across the 60 census blocks in our dataset when using naloxone only and other methods shown in Table 2, comparing CAD only to CAD þ ePCR (Best model). The height of each bar above "0" in Figure 4 represents the number of census blocks for which a perfect prediction was made (i.e., the number of NFOOD calls estimated by the model matched the number observed in the dataset); the heights of bars above negative values show the number of census blocks where more NFOOD calls were predicted by the model than observed in the dataset (for example, the height above "À1" represents the number of blocks where one fewer NFOOD call was observed than predicted, "À2" represents two fewer calls observed than predicted, etc.), and the heights of bars above positive values show the number of census blocks where fewer NFOOD calls were predicted than observed (for example, the height above "þ1" represents the number of blocks where one more NFOOD call was observed than predicted, "þ2" represents two more calls observed than predicted, etc.). Predicted NFOOD calls per census block using the Best Model (consisting of both CAD and ePCR variables) was closest to the observed number of calls, when compared to CAD alone or naloxone alone (Figure 4).

DISCUSSION
We undertook an analysis of 9-1-1 data from a single calendar year (2016) in a single Western US county, with the goal of optimizing our ability to identify and predict NFOOD cases. We identified 158 NFOOD cases within a set of 37,960 9-1-1 calls. The majority of NFOOD cases were in adults (38). The use of CAD problem type variables alone, that is, the variables available in the dispatch system at the time a 9-1-1 call is received, was not highly predictive of NFOOD cases, sensitivity being 37%. Notably, using only the CAD problem type "overdose/poisoning," which might seem like the most intuitive indicator of NFOOD cases, had low sensitivity alone (27%) and in combination with the naloxone administration variable from the ePCR (18%). This finding is somewhat intuitive, given that people who use drugs may fear a law enforcement response to a 9-1-1 call for help, and, therefore, may not identify the nature of the medical emergency to the dispatcher in an attempt to minimize the risk of criminal justice-related consequences (30,39). Our findings also support previous findings that naloxone is an imperfect indicator of NFOOD in EMS datasets (28). Using naloxone administration (as documented by EMS in the ePCR) as a single indicator had 67% sensitivity in our data, a finding which is again similar to that reported Wake County, North Carolina (57%) (15).
The Random Forests modeling with CAD and ePCR variables generated higher predictability (sensitivity, specificity, positive and negative predictive values) of NFOOD cases. In our best model, the combination of all CAD and ePCR variables resulted in a sensitivity of 75.9% and a specificity of 99.9%. This approach optimized the criteria for identifying NFOOD beyond primary/secondary impression, administration of naloxone by EMS, positive response to naloxone, and relevant narrative keywords (especially opioids, OOD, naloxone) (10,28).
The challenges of overdose surveillance data availability and quality are well recognized (38,40). These include significant delay in data availability, non-specificity of identification codes for unintentional and undetermined poisoning, data incompleteness, variation in NFOOD case definitions and population (28,38,40,41). At the same time, agencies charged with overdose response are seeking timely and accurate methods for using the available data to implement intervention efforts needed to address the ongoing overdose death crisis. To address this issue, our study had two guiding aims. First, we were interested in determining the predictive accuracy of variables available in real time in the CAD system where 9-1-1 dispatchers record information about emergency medical calls. If we were able to quickly and accurately predict which calls would be confirmed as NFOODs, this information could be used to dispatch interventionists or peer recovery support specialists along with EMS.
Post-overdose outreach has demonstrated promise in addressing the ongoing crisis of OOD deaths in the US (42)(43)(44) and dispatching an interventionist to the scene of an OOD could improve impact by ensuring contact with those patients who sign out against medical advice at the scene and/or refuse transport. Unfortunately, in our study, CAD variables alone demonstrated poor sensitivity and PPV, suggesting their limited utility for this purpose.
Second, we were interested in combining the CAD variables with the additional information available in the ePCR, which is completed by EMS personnel after the case is completed, to improve our ability to accurately predict NFOOD cases within census blocks. While decreasing the timeliness of identification (because ePCR data may not be available until hours or days after the call), this method could be useful for near-real-time surveillance purposes or for deploying peer support staff for community outreach and naloxone distribution. The most intuitive and best-performing variable within the ePCR for NFOOD case identification was naloxone administration by EMS, which demonstrated 67% sensitivity and 44% PPV. Accuracy declined when naloxone administration was combined with CAD variables. This finding suggests that using naloxone administration as a single variable may only identify about two thirds of NFOOD cases, leading to undercounting. Random Forest models in which all available variables are used to classify cases and non-cases could optimize identification of more NFOOD cases. Using all available CAD and ePCR variables, we were able to improve sensitivity and PPV to 75.9% and 71.4%, respectively. These Random Forests modeling procedures represent a first step toward improved utilization of 911 and ePCR data for near-real-time surveillance of NFOODs, which could lead to improved intervention efforts. While we had hoped our methodology would have the potential for developing near real-time surveillance for other emerging public health emergencies, and also for better surveillance of NFOOD cases during such emergencies (10,28), currently, our best model is not sufficiently sensitive to be used in this way. It might be worth considering the addition of a simple dichotomous indicator in the ePCR asking EMS personnel, "did this call involve an opioid-related overdose?" however, we also acknowledge that changes to ePCR systems may be difficult depending on the system used.
We also attempted to predict the number of NFOOD cases presenting within discrete geographical areas (i.e., census blocks), and found that our RF model did reasonably well in this regard. This type of spatial analysis could reveal areas of high NFOOD burden not discernible through other methods, particularly if overlaid with overdose mortality data or other indicators and could inform community outreach and naloxone distribution efforts.
Implications of our findings include the possibility of improving the use of 9-1-1 emergency medical dispatch data to contribute to more accurate surveillance of NFOODs and informing the timely deployment of intervention teams to the scene or the emergency department. Post-overdose outreach and intervention can provide assessment, counseling, and referral to substance use disorder treatment services, as well as opioid overdose education and naloxone distribution. Because many NFOOD patients who are treated with naloxone by EMS refuse to be transported to ED, where they could benefit from follow-up medical care, novel strategies for identifying NFOOD patients earlier in the continuum of care are needed. Thus, it is important to provide patients with information about the possible sequelae of OOD, take-home naloxone, and linkages to services when possible, even when they sign out against medical advice, so they can minimize their risk for fatal consequences in near future (9).

Limitations of the Study
The 9-1-1 data used for this study were from a single midsize county in the Western US and our findings may not be generalizable to other jurisdictions within the United States, due both to variations in local substance use morbidity and local protocols/procedures for dispatch and emergency response. Understanding the nuances from the different systems and challenges with documenting clinical responses are important in revising the performance of these models in other settings. Misclassification is also a concern. For example, emergency medical service responders may not have accurately documented features such as pupillary response to naloxone treatment. The list of keywords we used to identify NFOOD-related cases from narrative data have been used elsewhere for similar purposes (34), but it is also possible that the list was non-exhaustive, leading to some misclassification. In this retrospective study, the number of NFOOD we identified was relatively small, but was consistent with local patterns in ED admissions and OOD death data. Considering the increasing number of OOD in recent years, and the growing recognition of polypharmacy in driving those deaths, prospective validation of our findings will be helpful (10,28,29,45,46). Our modeling was limited to Random Forest, other machine learning techniques could yield higher sensitivity and warrant future study. Efforts are being made to more accurately enumerate NFOODs, and our research represents a first step in this direction. However, it will be important to replicate this approach in other settings. One primary contribution of the current work is to provide some data on the accuracy of using naloxone administration by EMS as an indicator of NFOOD, which has been used (and critiqued) in other settings (15,17,25,26).

CONCLUSION
In this study we identified 158 NFOODs among over 37,000 9-1-1 medical calls and examined the predictive accuracy of data from the CAD and ePCR for identifying them. CAD problem type variables and naloxone administration by EMS, used alone or in combination, had sub-optimal predictive accuracy, and would be inefficient to use for real time or near-real time identification of NFOODs. A Random Forests modeling approach improved accuracy of identification by 9% (from sensitivity of 67.1% to 75.9%), which could foster improved surveillance and intervention efforts.
An aspect of this work was presented at the 82nd College on Problems of Drug Dependence in June 2020, at the International Society for Disease Surveillance conference in 2019, and at the NAVIGATOR 2018 conference hosted by the International Academy of Emergency Dispatchers.