A comparison of two surveillance systems for deaths related to violent injury
- 1Oklahoma State Department of Health, Injury Prevention Service, Oklahoma City, Oklahoma and Centers for Disease Control and Prevention, Epidemiology Program Office, Division of Applied Public Health Training, Atlanta, Georgia, USA
- 2Oklahoma State Department of Health, Injury Prevention Service, Oklahoma City, Oklahoma, USA
- 3Office of the Chief Medical Examiner, Oklahoma City, Oklahoma, USA
- Correspondence to: Dr R Dawn Comstock Children’s Center for Injury Research and Policy, Ohio State University, College of Medicine, Department of Pediatrics, 700 Children’s Drive, Columbus, OH 43205, USA;
Objective: To compare violent injury death reporting by the statewide Medical Examiner and Vital Statistics Office surveillance systems in Oklahoma.
Methods: Using a standard study definition for violent injury death, the sensitivity and predictive value positive (PVP) of the Medical Examiner and Vital Statistics violent injury death reporting systems in Oklahoma in 2001 were evaluated.
Results: Altogether 776 violent injury deaths were identified (violent injury death rate: 22.4 per 100 000 population) including 519 (66.9%) suicides, 248 (32.0%) homicides, and nine (1.2%) unintentional firearm deaths. The Medical Examiner system over-reported homicides and the Vital Statistics system under-reported homicides and suicides and over-reported unintentional firearm injury deaths. When compared with the standard, the Medical Examiner and Vital Statistics systems had sensitivities of 99.2% and 90.7% (respectively) and PVPs of 95.0% and 99.1% for homicide, sensitivities of 99.2% and 93.1% and PVPs of 100% and 99.0% for suicide, and sensitivities of 100% and 100% and PVPs of 100% and 31.0% for unintentional firearm deaths.
Conclusions: Both the Vital Statistics and Medical Examiner systems contain valuable data and when combined can work synergistically to provide violent injury death information while also serving as quality control checks for each other. Preventable errors within both systems can be reduced by increasing training, addressing sources of human error, and expanding computer quality assurance programming. A standardized nationwide Medical Examiners’ coding system and a national violent death reporting system that merges multiple public health and criminal justice datasets would enhance violent injury surveillance and prevention efforts.
- CME-1, official Medical Examiner report
- ICD-10, International Classification of Diseases, 10th revision
- NVSS, National Vital Statistics System
- OSDH, Oklahoma State Department of Health
- PVP, predictive value positive
Violence is the leading cause of death for persons aged 15–44 years worldwide and accounted for an estimated 1 658 000 deaths in 2000; the age adjusted violence related injury death rate was 28.8 per 100 000.1 The United States has the highest violence related injury death rate of the developed countries.1 Defining violence as homicide or suicide only, in 2000, the age adjusted violence related injury death rate in the United States was 16.5 per 100 000.2 In 2001, violent injury deaths were the eighth leading cause of all deaths in Oklahoma and accounted for 30% of all injury deaths.2 When quality data are available to guide public health professionals, the predictability and preventability of violence can be determined.
In 1999, the Institute of Medicine Committee on Injury Prevention and Control recommended the development of a comprehensive fatal intentional injury surveillance system.3 In 2002, the World Health Organization called for enhanced capacity for collecting data on violence.1 Although the Centers for Disease Control and Prevention is piloting a population based, active surveillance system designed to obtain a census of all violent deaths and detailed information about the circumstances of these violent death events,3 a standardized, nationwide violent death reporting system does not yet exist in the United States. Data on violent deaths may currently be assembled from several sources. The Federal Bureau of Investigation maintains the Uniform Crime Reporting Program and annually produces Crime in the United States.4 This program provides some details on murder and non-negligent manslaughter (United States law defined criminal homicide as opposed to International Classification of Diseases, 10th revision (ICD-10) defined homicide); however, these data may be incomplete due to local level agencies voluntary reporting.5 Additionally, suicides are not included. The National Vital Statistics System (NVSS) is a standardized mortality reporting system universally collected in the United States6; however, the information provided about the circumstances of injury are often limited. Medical Examiners or coroners charged with determining the cause (mechanism) and manner (intent) of death provide more detailed information including alcohol and drug involvement, type of weapon used, and circumstances resulting in death,7 but no standardized nationwide Medical Examiner/coroner system exists in the United States and methodology and reporting systems vary from state to state.
Oklahoma has a statewide, centralized Medical Examiner system. All violent injury deaths in Oklahoma must be reported (OS63 § 931–954) to the Office of the Chief Medical Examiner, and the Medical Examiner working closely with the law enforcement system conducts a medicolegal investigation of the circumstances resulting in death and files an official Medical Examiner report (CME-1). In addition, the Medical Examiner completes and signs the sections of the death certificate indicating the circumstances, including cause and manner, of death. Funeral homes complete the demographic information sections of the death certificate in consultation with family/friends of the deceased. Completed death certificates are forwarded to the Oklahoma State Department of Health (OSDH) Vital Statistics Office where they are scanned into a database and trained data entry personnel enter the cause and manner of death into the database. The data are then sent to the National Center for Health Statistics for death coding, utilizing software and nosologists trained in ICD-10 rules.8 These coded data are returned to OSDH and made available to epidemiologists through the OSDH intranet; de-identified data are also accessible at National Center for Health Statistics and becomes a part of the NVSS (fig 1).
The purpose of this study was to compare violent injury death reporting by the statewide Medical Examiner and the OSDH Vital Statistics Office surveillance systems in Oklahoma.
Computerized datasets of all injury deaths in 2001 from the Medical Examiner and Vital Statistics Office were linked using the Automatch probabilistic software matching program.9 Matching was performed in a sequential block system using name, date of death, sex, county of death, age, race, and state of residence as matching variables. Manner of death was reported using the manner of death indicated on the CME-1 in the Medical Examiner system and ICD-10 coding in the Vital Statistics system.
We created a “standard definition of a violent injury death” based predominantly on the definition utilized by the Centers for Disease Control and Prevention in the National Violent Death Reporting System currently being piloted in the United States.3 All deaths in the Vital Statistics and Medical Examiner systems meeting the criteria to be coded suicide (ICD-10 X60-X84, Y87.0), homicide (ICD-10 X85-Y08, Y87.1), legal intervention (ICD-10 Y35, Y89.0), unintentional firearm death (ICD-10 W32–34, Y86) or terrorism deaths (U01, U03) were considered to be violent injury deaths. Our definition of violent injury death did not include events of undetermined intent (ICD-10 Y10-Y34, Y87.2, Y89.9). Only Oklahoma residents who died in 2001 were included in the study.
We compared the manner of death reported by the Medical Examiner and Vital Statistics systems to the standard manner of death designated for each record to determine if each surveillance system had properly coded the manner of death. The manner of death reported by both systems was designated as the standard manner of death when the Medical Examiner and Vital Statistics records were concordant. When the Medical Examiner and Vital Statistics databases had a discordant manner of death (for example, unintentional v homicide), we reviewed the hard copies of the Medical Examiner report and Vital Statistics death certificate and designated the standard manner of death in one of two ways. If we found an obvious error (for example, miscode of a natural death as an injury, a missing death record, etc) the manner of death reported by the system without the error was designated as the standard manner of death. For the remaining discordant records, we submitted the hard copy of the death certificate to the National Center for Health Statistics for blinded ICD-10 recoding by an experienced nosologist who also specified ICD-10 coding rules; the resultant ICD-10 code was designated as the standard manner of death. Miscoded records included those: (1) coded as violent injury deaths that did not meet the study definition of violent injury death (for example, natural deaths miscoded as unintentional firearm deaths); (2) meeting the study definition but not coded as violent injury deaths (for example, suicide poisoning deaths miscoded as accidental poisonings); and (3) coded as violent injury deaths but miscoded with respect to the proper homicide, suicide, or unintentional firearm designation (for example, homicide deaths that were miscoded as unintentional firearm deaths).
We calculated each surveillance system’s sensitivity and predictive value positive (PVP) for each subcategory of violent injury death. Sensitivity was defined as the proportion of true violent injury death events identified as violent injury deaths by the surveillance system (true positives divided by sum of the true positives and false negatives) and PVP was defined as the proportion of violent injury deaths identified by the surveillance system that were true violent injury death events (true positives divided by sum of the true positives and false positives). Finally, we systematically reviewed the movement of data through both systems to identify potential sources of error leading to miscoded manner of death or to discordance in matching the records from the two systems.
Probabilistic linkage resulted in 776 deaths that met the study definition of violent injury death. This included 741 deaths meeting the definition that had records in both the Medical Examiner and Vital Statistics system and were identified as violent injury deaths by one or both of the surveillance systems, 29 Medical Examiner death records meeting the definition that did not have matching Vital Statistics records, and six Vital Statistics death records meeting the definition that did not have matching Medical Examiner records. In addition, there were 22 records coded as violent injury deaths that did not meet the study definition, including 13 Medical Examiner records coded transportation homicides that ICD-10 coding rules recoded accidental and nine Vital Statistics records coded as violent injury deaths that did not meet the standard definition (seven natural deaths and two accidental injury deaths). The Medical Examiner system over-reported homicides and the Vital Statistics system under-reported homicides and suicides and over-reported unintentional firearm injury deaths (fig 2).
The 776 violent injury deaths represented 31% of injury deaths and 2% of all 2001 Oklahoma deaths (violent injury death rate 22.4/100 000 population). Violent injury deaths included 519 (66.9%) suicides, 248 (32.0%) homicides, and nine (1.2%) unintentional firearm deaths; there were no terrorism deaths (fig 2).
Manner of death coding discrepancies occurred for 90 deaths with 78.9% of the coding discrepancies occurring in the Vital Statistics system (table 1). Several reasons for coding discrepancies were identified including: (1) coding disagreements between the Medical Examiner medicolegal coding and the Vital Statistics ICD-10 coding (for example, transportation deaths coded homicide by Medical Examiner and unintentional according to ICD-10 rules); (2) deaths which were reported to one system but not the other; (3) records with manner of death miscoded by one of the two systems (for example, natural deaths miscoded as unintentional firearm deaths); (4) deaths that existed in both databases but were missing any ICD-10 code in the Vital Statistics database; and (5) Vital Statistics coding irregularities due to improperly completed death certificates.
Sensitivity and PVP
Sensitivity and PVP of the two systems differed by manner of death (table 2). Sensitivity rates were higher in the Medical Examiner system for homicides and suicides. PVP rates were higher in the Vital Statistics system for homicides and substantially lower for unintentional firearm deaths. It should be noted that death records that were properly identified as violent injury deaths but miscoded with respect to homicide, suicide, or unintentional firearm designation will appear in more than one 2 × 2 table (for example, a firearm suicide miscoded as an unintentional firearm death will appear as a false negative in the suicide 2 × 2 table and as a false positive in the unintentional firearm 2 × 2 table). The unintentional firearm deaths over-reported by Vital Statistics included six firearm homicides, eight firearm suicides, one undetermined firearm death, and five natural deaths miscoded as unintentional firearm deaths during ICD-10 coding.
Each coding discrepancy affecting the sensitivity and PVP of the two systems was investigated. Specific sources of discordant manner of death coding were discovered including: (1) OSDH data entry errors (keypunch errors, misreads from poorly completed forms, fatigue/inattention of data entry staff, etc) resulting, for example, in the wrong entry of the manner of death or the wrong death certificate information being entered for a certificate number; (2) pre-coding abbreviation errors that resulted in an “mm” designation intended to indicate a malignant melanoma being misinterpreted as an indication of ammunition leading to a natural death being miscoded as a firearm death; (3) death certificate amendments to the manner or cause of death not submitted to OSDH for cases “pending” or not updated in Vital Statistics computer data files when received resulting in incomplete death coding; (4) ICD-10 coding rules that change the Medical Examiner determination of manner of death resulting in coding differences; (5) human errors by National Center for Health Statistics nosologists that lead to incorrect coding (applying the wrong ICD-10 code); (6) errors by Medical Examiners in completing death certificates that lead to incorrect coding (placing a natural disease process such as atherosclerotic heart disease as the immediate cause of death and the homicide by assault in the “other significant medical conditions” field on the death certificate resulting in a death being coded as a natural death); and (7) failure/inability of software to catch obvious data entry errors resulting, for example, in a report of an infant committing suicide by firearm.
We believe this is the first study to conduct a direct comparison of the sensitivity and PVP of violent injury death reporting by statewide Medical Examiner and Vital Statistics Office surveillance systems. Since the Vital Statistics system is the standardized, universal death surveillance system in the United States and Vital Statistics mortality data are used frequently to monitor trends, to drive development of prevention programs and policy, and to evaluate public health interventions, we used the ICD-10 coding rules when determining the standard manner of death designation in this study. Several previous studies have directly compared the mortality data available from Medical Examiner and Vital Statistics surveillance systems in general10 and for a wide range of public health mortality concerns including drowning,11 traumatic injuries,12 unintentional and intentional injuries,13 non-injury deaths,14 cocaine overdoses,15,16 sudden and unexpected deaths due to natural causes,17 autopsied deaths,18 carbon monoxide poisonings,19 motorcycle crashes,20 injury-at-work deaths,21 all-terrain vehicle crashes,22 and poisonings23 with varied results. Similarly, other researchers have evaluated the quality of data produced by Medical Examiner24–34 or Vital Statistics25,35–48 surveillance systems, often concluding that the quality of data would be improved through standardization of definitions and coding systems and through enhanced training of the Medical Examiners, coroners, physicians, and nosologists responsible for determining and recording manner of death. However, even with these limitations, both systems have been found to be widely useful.
The sensitivity of the Oklahoma Medical Examiner system was over 99% for all violent deaths. The Medical Examiner system codes deaths caused by the legal system’s determination of “negligent homicide” (for example, a pedestrian hit by a driver under the influence) as homicide while the ICD-10 rules state “when homicide is reported in the manner of death box with transport accidents, code the external cause as accidental unless there is a statement on the certificate that clearly establishes that an intentional act of homicide occurred (for example, decedent run over by vehicle several times in parking lot)”.8 Thus, in this study, transportation deaths noted as homicides by the Medical Examiner were considered miscoded in the Medical Examiner system if they were coded accidental deaths by the Vital Statistics system; however, a review of the CME-1 narratives found two were intentional and would have been coded homicide in the Vital Statistics system had there been statements clearly establishing intentionality on the death certificate. Additionally, two homicides and two suicides were coded natural deaths because the primary cause of death noted on the death certificate indicated natural disease (for example, atherosclerotic heart disease) while injuries were only noted in the section “other significant medical conditions not directly contributing to death”. This suggests Medical Examiner training in completing death certificates could improve the quality of Vital Statistics coding of death data.
While the sensitivity of the Vital Statistics system was over 90% for violent injury deaths in Oklahoma, we found several steps in the processing of death data in Oklahoma where error was introduced. Although each source of error identified in this study accounted for only a small number of discrepancies between the Medical Examiner and Vital Statistics system, they affected the sensitivity and PVP of the Vital Statistics system. The accuracy of violent injury death reporting can be increased by (1) minimizing human data entry error by utilizing automated death certificate scanning systems, revising forms to facilitate logical data entry, and rotating work tasks/incorporating frequent breaks to keep manual data entry staff alert; (2) providing detailed training on violent injury death coding to nosologists and providing detailed training on completing death certificates to Medical Examiners and coroners; (3) updating pre-coding and coding software quality assurance programs so obvious data entry or logic errors are caught and addressed; and (4) routinely evaluating the flow of information in both hard and electronic forms in existing violent injury death surveillance systems to identify and address potential sources of error.
Although a standardized, nationwide violent death reporting system does not yet exist in the United States, the Centers for Disease Control and Prevention are piloting a population based, active surveillance system designed to obtain a census of all violent deaths and detailed information about the circumstances of these violent death events.
The Medical Examiner system over-reported homicides and the Vital Statistics system under-reported homicides and suicides and over-reported unintentional firearm injury deaths.
The Medical Examiner and Vital Statistics surveillance systems contain valuable data, can work synergistically to provide violent injury death information, and can serve as quality control checks for each other.
The diversity in state based death investigation systems in the United States (Medical Examiner, coroner, or combination Medical Examiner and coroner based) limits our ability to generalize findings based on these data to other state’s data or to national data.
One potential limitation in our study was our decision to classify injury deaths as either violent injury deaths or not violent injury deaths when the Medical Examiner and Vital Statistics systems agreed. We did not review all injury death records which the Medical Examiner and Vital Statistics systems agreed were not violent injury deaths to determine false negative rates. Thus, we did not estimate specificity. Additionally, we chose to exclude deaths from undetermined intent from our standard definition of violent injury death. In 2001, 20 Oklahoma deaths were coded with undetermined intent (19 concordantly coded); thus, their inclusion would have had minimal impact on our findings. If added to the 776 violent injury deaths identified in this study, these 20 deaths would have represented 2.5% of all violent injury deaths. While we did not formally review discrepancies in cause of death reporting (for example, hanging, stabbing, etc), preliminary analysis of the data suggests fewer than 3% of causes of death reported by the Medical Examiner and Vital Statistics were discordant. Finally, the diversity in death investigation systems in the United States (Medical Examiner based, coroner based, and combination Medical Examiner and coroner based; statewide, centralized Medical Examiner systems are only available in 14 states) limits our ability to generalize findings based on this data to other states’ data or to national data.49
While identifying and attempting to address discrepancies between the Medical Examiner and Vital Statistics violent injury death surveillance systems will improve the quality of violent injury death data in Oklahoma, an unavoidable number of discrepancies will persist between these two systems because they use different approaches to coding (medicolegal and ICD-10). However, when linked, these two systems provide a much more complete picture of violent injury death events than is attainable when either system is the sole source of data. Additionally, the Vital Statistics and Medical Examiner systems work synergistically, and can serve as quality control checks for each other by merging the datasets and investigating discrepancies.
Linking datasets from multiple sources will result in more complete case ascertainment than using one source alone, thus providing higher quality data.17,18,21–23,31,44,49–54 For example, merging criminal justice datasets with public health datasets could enable access to information on the perpetrators in homicides thus leading to more substantive analytic risk factor analysis and ultimately to more effective targeted prevention efforts. However, as information from multiple sources is merged, issues will arise concerning which data source is right regarding categorization of manner of death. For example, as discussed by Barber et al, differences in manner of death reported for firearm deaths by police department reports and Medical Examiner reports likely reflect differences in definition of terms rather than inaccuracies.55 Continuing to expand the pilot National Violent Death Reporting System3 would improve violent injury surveillance and enhance knowledge of violence related risk factors necessary to guide the development and evaluation of prevention programs in the United States. Adoption of a standardized nationwide Medical Examiners’ coding system and a nationwide violent death reporting system based on a combination of multiple public health and criminal justice datasets is needed. Future studies should investigate errors and differences in cause of death reporting between the two systems.
We acknowledge Sue Bordeaux and John Burks, Sr, Oklahoma State Department of Health Vital Statistics Office for assistance in the evaluation of the Vital Statistics system and review of the manuscript. We also acknowledge nosologists from the National Center for Health Statistics for assistance in determining the accuracy of manner of death reporting. This report is partially funded by the Centers for Disease Control and Prevention (CDC) grant # U17/CCU617756. The content of this report is solely the responsibility of the authors and does not necessarily represent the official view of the CDC.