Brief report: Comparison of methods to identify Iraq and Afghanistan war veterans using Department of Veterans Affairs administrative data.
Post-traumatic stress disorder (Surveys)
|Publication:||Name: Journal of Rehabilitation Research & Development Publisher: Department of Veterans Affairs Audience: Academic Format: Magazine/Journal Subject: Health Copyright: COPYRIGHT 2010 Department of Veterans Affairs ISSN: 0748-7711|
|Issue:||Date: Dec 30, 2010 Source Volume: 47 Source Issue: 8|
|Topic:||Computer Subject: Algorithm|
|Product:||Product Code: E198380 Veterans|
|Geographic:||Geographic Scope: Iraq; Afghanistan; Minnesota Geographic Code: 7IRAQ Iraq; 9AFGH Afghanistan; 1U4MN Minnesota|
More than 2 million servicemembers have served in Operation Iraqi Freedom (OIF) and Operation Enduring Freedom (OEF) and nearly 1.2 million have been discharged as veterans. Veterans from these conflicts have experienced multiple deployments and unique combat environments [1-3], along with high rates of psychiatric disturbance and physical injury [4-7]. Recognizing the importance of the healthcare needs and demands of OIF/ OEF veterans, the Veterans Health Administration (VHA) has solicited research focused on them [8-9].
The Department of Veterans Affairs (VA) health services use SAS data sets found in the Austin Information Technology Center (AITC) contain national VHA-provided healthcare information on veterans, which researchers use to identify patient populations [10-11]. Using these data sets, VA researchers can identify periods of service (POS) for veterans who have served from the Spanish-American War to the gulf war. However, VA researchers cannot search the VA health services use SAS data sets by POS to identify OIF/OEF veterans because Congress has not designated OIF/OEF as a POS. Other methods are therefore needed to identify this priority group.
The VHA Support Service Center OIF/OEF Roster is arguably the gold standard for identification of OIF/OEF veterans [12-14]. The OIF/OEF Roster is cataloged by the Department of Defense's Defense Manpower Data Center (DMDC) and has been provided to VHA's Environmental Epidemiology Service (EES) since September 2003. This comprehensive database includes identifying information on VA-registered OIF/OEF veterans with military discharges starting October 1, 2001, who were (1) physically located within the OIF/OEF combat zones or areas of operation or (2) identified by their service branch as directly supporting the OIF/OEF mission outside designated combat zones. ** This data set is not available to researchers without funded studies. Furthermore, the approval process, although appropriate given the need to ensure veteran privacy and data security, can take several months. Another method for identifying OIF/OEF veterans is self-reported POS on surveys or during interviews. This method, however, cannot be used for selection of OIF/OEF veterans before data collection.
In this article, we describe results from two studies that compare an algorithm we developed to identify OIF/ OEF VA users that makes use of variables available in the Outpatient Visit (SF) file of the VA health services use database. We compared this OIF/OEF Algorithm with the OIF/OEF Roster (study 1) and then with veterans' self-report surveys (study 2). We describe the positive predictive value (PPV) and sensitivity of our OIF/ OEF Algorithm and also present differences between veterans we identified using our OIF/OEF Algorithm against those we failed to identify according to the OIF/ OEF Roster and self-report survey. Findings will help investigators determine the most appropriate approach for identifying OIF/OEF veterans for their research.
Samples and Study Design
Study 1 included two samples: (1) a national sample of OIF/OEF veterans who used VA healthcare from fiscal year (FY) 2004 to FY2007 selected on the basis of the OIF/OEF Algorithm (outlined in the Figure) for a study examining OIF/OEF veteran reintegration problems and treatment interests and (2) OIF/OEF veterans included in the OIF/ OEF Roster with military discharges through December 2007 who had VA health services use data from FY2004 to FY2007. Study 2 included a sample of veterans who had at least one diagnosis of posttraumatic stress disorder (PTSD) in VA health services use data during FY2008 or FY2009 for a study of PTSD treatment participation. In study 2, we compared those who self-reported OIF/OEF POS on the completed survey with those we identified as OIF/OEF veterans using the OIF/OEF Algorithm.
For study 1, we used VA health services use data to obtain demographic information and health services use information, including psychiatric diagnoses and use of a VA facility in the past 2 years. For study 2, we obtained age from VA health services use data. We used the self-report survey for all other demographic information and POS. The survey included a question about which conflicts the survey recipient served in. We classified survey respondents who indicated they had participated in OEF (2001-present), OIF (2003-present), or the Global War on Terrorism (2004-present) as self-reported OIF/OEF veterans.
OIF/OEF Algorithm Based on VA Health Services Use Data: Design and Data Source
OIF/OEF veterans are, by definition, combat veterans discharged after October 1, 2001, and eligible for VA services. Before the National Defense Authorization Act of 2008 was enacted on January 28, 2008, the VA granted OIF/OEF combat veterans discharged or released from Active Duty on or after January 28, 2003, eligibility to enroll for VA healthcare for 2 years from the date of discharge or release. That end date has subsequently been expanded to 5 years .
To identify OIF/OEF veterans, we restricted the following three variables in the SF file: (1) combat eligibility flag, (2) end date of VA healthcare eligibility, and (3) POS. It should be noted that the variable indicating the end date of VA healthcare eligibility will exist only if the combat eligibility flag occurs in a veteran's record. We defined the end date of VA healthcare eligibility differently in study 1 and study 2. Because the researchers prioritized surveying OIF veterans who would have been discharged after October 2003, we specified selection of those veterans with an eligibility end date after October 1, 2005, for study 1. For study 2, we used an earlier VA eligibility end date (October 1, 2003) to encompass a larger proportion of OEF veterans, since the war began in 2001. The Figure depicts the OIF/OEF Algorithm.
The gold standards were the OIF/OEF Roster and self-report survey for study 1 and study 2, respectively. We performed analyses with SAS 9.1 software (SAS Incorporation Inc; Cary, North Carolina).
For studies 1 and 2, we calculated sensitivity and PPV. For study 2, we also calculated specificity and negative predictive value (NPV). However, given that the OIF/OEF Roster only includes OIF/OEF veterans, we were unable to calculate specificity or NPV for study 1. To compare demographic characteristics for study 1 and study 2, we used Pearson chi-square tests for categorical variables and two-sample t-tests for continuous variables. For study 1, we also used the nonparametric Wilcoxon rank-sum test for number of visits because of the significant positive skew in its distribution.
Study 1: OIF/OEF Algorithm Versus VHA Support Service Center OIF/OEF Roster
Of the 181,612 potential OIF/OEF veterans we identified using of OIF/OEF Algorithm, we confirmed 92 percent (PPV, n = 167,110) as OIF/OEF when compared with the OIF/OEF Roster. However, our OIF/OEF Algorithm failed to identify 119,758 OIF/OEF veterans. The sensitivity was 58 percent (167,110/286,868) and the false negative rate was 42 percent. There were differences between those OIF/ OEF veterans identified using our OIF/OEF Algorithm and those OIF/OEF veterans we failed to identify (Table 1). The small differences in age and sex reached statistical significance because of the large sample size. The differences in rate of psychiatric diagnoses and healthcare use, however, appear meaningful.
We failed to identify most (n = 88,974, 74%) of the 119,758 OIF/OEF veterans using our OIF/OEF Algorithm because they were not classified as "combat" veterans in the SF file. The remaining 30,784 (26%) did not have a VA healthcare eligibility date or POS that fit our restrictions.
Study 2: OIF/OEF Algorithm Versus OIF/OEF Veteran Self-Report
Of the 7,952 veterans surveyed, 5,207 returned study material for a response rate of 65.5 percent. Among the 4,563 who self-reported a POS on the survey, 1,229 of those we identified using the OIF/OEF Algorithm also self-reported OIF/OEF POS. However, of the 4,563 veterans that self-reported POS, we failed to identify 331 (7.3%) OIF/OEF veterans and misidentified 24 (0.5%). Our OIF/OEF Algorithm had a PPV of 0.98, NPV of 0.90, sensitivity of 0.79, and specificity of 0.99 (Table 2). Thus, assigning OIF/OEF veteran status using our OIF/ OEF Algorithm had a higher false negative rate (21%) than a false positive rate (1%), as detailed in Table 2.
Table 3 presents differences between those we identified and failed to identify using our OIF/OEF Algorithm. OIF/OEF veterans not identified using our OIF/ OEF Algorithm were more likely than the identified OIF/OEF veterans to report having served in Afghanistan or the Global War on Terrorism but less likely to report having served in Iraq.
For those 331 OIF/OEF veterans that our OEF/OIF Algorithm failed to identify, 267 (80.6%) were not classified as "combat" veterans in the SF file. The remaining 64 did not have a POS that fit the criterion we used for that variable.
The PPV of our OIF/OEF Algorithm against the OIF/ OEF Roster was 92 percent (167,110/181,612) and, as detailed in Table 2, the PPV of the self-reported survey was 98 percent. The PPV of our OIF/OEF Algorithm was acceptable for both studies. However, the false negative rate could be problematic depending on the sampling goals. Those OIF/OEF veterans identified by the OIF/OEF Algorithm (study 1) were more likely to have a mental health diagnosis and more VA appointments than those not identified. Therefore, using an algorithm based on VA health services use data to select OIF/OEF veterans may result in a nonrepresentative sample of OIF/OEF VA users. It is unknown whether veterans not identified using our OIF/ OEF Algorithm have similar healthcare needs and are simply receiving healthcare services outside of the VA. Another possibility is that reduced VA appointment frequency may result in decreased opportunities to update combat status and diagnose all of a veteran's medical conditions.
Study 2 suggests that the OIF/OEF Algorithm produces few false positives when the population of interest is OIF/OEF veterans diagnosed with PTSD within the VA. We would expect this group of veterans to use healthcare services more routinely and therefore have an accurate combat status in VA health services use data. Our OIF/OEF Algorithm could be a useful tool for developing a sampling frame if no other method to identify OIF/OEF veterans is available. However, as in study 1, we missed those without a combat indicator.
Starting in FY2008, a new method for OIF/OEF veteran identification was released. There is now an OIF/OEF flag available to researchers in the National Data Extracts of the Decision Support System (DSS), located in the AITC system. This DSS OIF/OEF veteran flag is created using the DMDC data from the OIF/OEF Roster that is sent to EES and does not rely on the combat indicator that we found to be unreliable. Additionally, using the DSS OIF/OEF veteran flag to select a cohort of OIF/OEF veterans requires only one data variable restriction as opposed to three different variable restrictions that we used in the OIF/OEF Algorithm. This method is limited to veterans who have used the VA healthcare system from FY2008 onward.
Our OIF/OEF Algorithm allows for accurate OIF/OEF veteran identification of a subgroup of veterans who are classified by combat eligibility status in VA health services use data. However, it fails to identify nearly 42 percent of OIF/OEF veterans on the OIF/OEF Roster and 21 percent who self-reported OIF/OEF veteran status. Therefore, our OIF/OEF Algorithm would not be appropriate if a researcher wanted to sample from the entire OIF/OEF veteran population that uses VA services, but it might be useful to identify the subgroup of OIF/OEF veterans who use the VA more frequently and have mental health diagnoses such as PTSD.
Abbreviations: AITC = Austin Information Technology Center, DMDC = Defense Manpower Data Center, DSS = Decision Support System, EES = Environmental Epidemiology Service, FY = fiscal year, NPV = negative predictive value, OEF = Operation Enduring Freedom, OIF = Operation Iraqi Freedom, POS = period of service, PPV = positive predictive value, PTSD = posttraumatic stress disorder, SF = outpatient visit, VA = Department of Veterans Affairs, VHA = Veterans Health Administration.
Study concept and design: N. Sayer, M. Spoont.
Acquisition of data: A. Bangerter, A. Cutting, N. Sayer, M. Spoont.
Analysis and interpretation of data: B. Clothier, A. Gravely, A. Cutting, A. Bangerter, N. Sayer, M. Spoont.
Drafting of manuscript: A. Bangerter, N. Sayer.
Critical revision of manuscript for important intellectual content: N. Sayer, A. Bangerter, A. Gravely, A. Cutting, M. Spoont, B. Clothier.
Statistical analysis: B. Clothier, A. Gravely, A. Cutting, A. Bangerter.
Obtained funding: N. Sayer, M. Spoont.
Administrative, technical, or material support: A. Bangerter, A. Cutting, A. Gravely.
Study supervision: N. Sayer, M. Spoont.
Financial Disclosures: The authors have declared that no competing interests exist.
Funding/Support: This material was based on work supported by the VA Health Services Research and Development Service (grants IAC 06-266 and RRP 07-315).
Additional Contributions: The views expressed in this article are those of the author(s) and do not necessarily represent the position or policy views of the VA or any other government agency.
Institutional Review: The data for this work came from two separately funded projects approved by the Subcommittee on Human Studies, Minneapolis VA Medical Center, Minneapolis, Minnesota.
[1.] Hosek J, Kavanagh J, Miller L. How deployments affect service members. Santa Monica (CA): RAND; 2006.
[2.] Hoge CW, Castro CA, Messer SC, McGurk D, Cotting DI, Koffman RL. Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care. N Engl J Med. 2004; 351(1):13-22. [PMID: 15229303] DOI:10.1056/NEJMoa040603
[3.] Tanielian TL, Jaycox L; RAND Corporation. Invisible wounds of war: Psychological and cognitive injuries, their consequences, and services to assist recovery. Santa Monica (CA): RAND; 2008.
[4.] Okie S. Traumatic brain injury in the war zone. N Engl J Med. 2005;352(20):2043-47. [PMID: 15901856] DOI:10.1056/NEJMp058102
[5.] Gawande A. Casualties of war-Military care for the wounded from Iraq and Afghanistan. N Engl J Med. 2004; 351(24):2471-75. [PMID: 15590948] DOI:10.1056/NEJMp048317
[6.] Seal KH, Metzler TJ, Gima KS, Bertenthal D, Maguen S, Marmar CR. Trends and risk factors for mental health diagnoses among Iraq and Afghanistan veterans using Department of Veterans Affairs Health Care, 2002-2008. Am J Public Health. 2009;99(9):1651-58. [PMID: 19608954] DOI:10.2105/AJPH.2008.150284
[7.] Seal KH, Bertenthal D, Maguen S, Gima K, Chu A, Marmar CR. Getting beyond "Don't ask; don't tell": An evaluation of US Veterans Administration postdeployment mental health screening of veterans returning from Iraq and Afghanistan. Am J Public Health. 2008;98(4):714-20. [PMID: 18309130] DOI:10.2105/AJPH.2007.115519
[8.] Office of Research and Development program announcement: Quality Enhancement Research Initiative (QUERI) special solicitation for projects: Implementing research into practice to improve care delivery. Washington (DC): Department of Veterans Affairs; 2005.
[9.] Afghanistan & Iraq [Internet]. Washington (DC): Department of Veterans Affairs; 2009 [cited 2009 Aug 3]. Available from: http://www.research.va.gov/resources/pubs/docs/OIF-OEFbrochure.pdf.
[10.] Rosenheck RA, Fontana AF. Recent trends in VA treatment of post-traumatic stress disorder and other mental disorders. Health Aff (Millwood). 2007;26(6):1720-27. [PMID: 17978391] DOI:10.1377/hlthaff.26.6.1720
[11.] Jia H, Zheng YE, Cowper DC, Stansbury JP, Wu SS, Vogel WB, Duncan PW, Reker DM. Race/ethnicity: Who is counting what? J Rehabil Res Dev. 2006;43(4):475-84. [PMID: 17123187] DOI:10.1682/JRRD.2005.05.0086
[12.] Seal KH, Bertenthal D, Miner CR, Sen S, Marmar C. Bringing the war back home: Mental health disorders among 103,788 US veterans returning from Iraq and Afghanistan seen at Department of Veterans Affairs facilities. Arch Intern Med. 2007;167(5):476-82. [PMID: 17353495] DOI:10.1001/archinte.167.5.476
[13.] Cohen BE, Gima K, Bertenthal D, Kim S, Marmar CR, Seal KH. Mental health diagnoses and utilization of VA non-mental health medical services among returning Iraq and Afghanistan veterans. J Gen Intern Med. 2010;25(1): 18-24. [PMID: 19787409] DOI:10.1007/s11606-009-1117-3
[14.] Haskell SG, Brandt CA, Krebs EE, Skanderson M, Kerns RD, Goulet JL. Pain among veterans of Operations Enduring Freedom and Iraqi Freedom: Do women and men differ? Pain Med. 2009;10(7):1167-73. [PMID: 19818028] DOI:10.1111/j.1526-4637.2009.00714.x
[15.] National Defense Authorization Act of 2008, HR 4986, 110th Cong. 2nd Session (2008).
Submitted for publication August 7, 2009. Accepted in revised form July 6, 2010.
This article and any supplementary material should be cited as follows:
Bangerter A, Gravely A, Cutting A, Clothier B, Spoont M, Sayer N. Brief report: Comparison of methods to identify Iraq and Afghanistan war veterans using Department of Veterans Affairs administrative data. J Rehabil Res Dev. 2010;47(8):815-22.
Ann Bangerter, BS; (1) * Amy Gravely, MA; (1) Andrea Cutting, MA; (1) Barb Clothier, MA, MS; (1) Michele Spoont, PhD; (1-3) Nina Sayer, PhD, LP (1-2,4)
(1) Center for Chronic Disease Outcomes Research, Minneapolis Department of Veterans Affairs Medical Center, Minneapolis, MN; Departments of (2) Psychology, (3) Psychiatry, and (4) Medicine, University of Minnesota, Twin Cities Campus, Minneapolis, MN
* Address all correspondence to Ann Bangerter, BS; Center for Chronic Disease Outcomes Research, VA Medical Center, One Veterans Drive (152/2E), Minneapolis, MN 55417; 612467-1384; fax: 612-727-5699. Email: firstname.lastname@example.org
** Access to the OIF/OEF Roster description is provided on a Web site restricted to VA employees. Please contact the corresponding author for an electronic copy.
Table 1. Comparison of Operation Iraqi Freedom/Operation Enduring Freedom (OIF/OEF) veterans selected using OIF/OEF Algorithm versus OIF/ OEF veterans not selected. Confirmed OIF/OEF Veteran * Selected Not Selected Characteristic (n = 167,110) (n = 119,758) Male, n (%) 147,057 (88) 104,190 (87) Reservist, n (%) 90,240 (54) 55,089 (46) Diagnosis in VA Records, ([dagger]) n (%) Anxiety 17,463 (10.5) 8,168 (6.82) PTSD 54,528 (32.6) 18,371 (15.3) Mood Disorder 44,669 (26.7) 17,964 (15.0) Psychosis 1,855 (1.1) 886 (0.7) MH Visit Recorded, 18,766 (11.2) 9,233 (7.7) ([double dagger]) n (%) Number of VA Appoint- 10 2 ments ([dagger]) (median) Age (mean [+ or -] SD) 33.2 [+ or -] 9.0 36 [+ or -] 9.8 Characteristic p-Value Male, n (%) <0.001 Reservist, n (%) <0.001 Diagnosis in VA Records, ([dagger]) n (%) Anxiety <0.001 PTSD <0.001 Mood Disorder <0.001 Psychosis <0.001 MH Visit Recorded, <0.001 ([double dagger]) n (%) Number of VA Appoint- <0.001 ments ([dagger]) (median) Age (mean [+ or -] SD) <0.001 OIF/OEF veteran status confirmed using Veterans Health Administration Support Service Center OIF/OEF Roster. ([dagger]) Based on at least one ICD-9-CM code in VA health services use data 2 years before survey. ICD-9-CM codes used for diagnoses are as follows: Anxiety = 300.0, 300.2, 300.3, 308, 309. PTSD = 309.81. Mood Disorder = 296, 300.4, 311. Psychosis = 295, 297, 298. ([double dagger]) Based on clinical visit to any MH Clinic Stop (500 [less than or equal to] clinic stop < 600) at medical center or associated VA community-based outpatient clinic 1 year before survey. ICD-9-CM = International Classification of Diseases-9th Revision-Clinical Modification, MH = mental health, PTSD = posttraumatic stress disorder, SD = standard deviation, VA = Department of Veterans Affairs. Table 2. Sensitivity and specificity of Operation Iraqi Freedom/Operation Enduring Freedom (OIF/OEF) Algorithm as compared with self- reported survey data. Selected Using OIF/OEF Veteran * Predictive Value (%) OIF/OEF Yes No Algorithm Yes TP = 1,229 FP = 24 Positive = 98 No FN = 331 TN = 2,979 Negative = 90 Sensitivity (%) 79 -- -- Specificity (%) -- 99 -- OIF/OEF status confirmed using self-reported survey data. Note: Sensitivity = TP / (TP + FN) = 1,229 / (1,229 + 331) = 79%. Specificity = TN / (FP + TN) = 2,979 / (24 + 2,979) = 99%. Positive predictive value = TP / (TP + FP) = 1,229 / (1,229 + 24) = 98%. Negative predictive value = TN / (FN + TN) = 2,979 / (331 + 2,2979) = 90%. FP rate ([alpha]) = FP / (FP + TN) = 24 / (24 + 2,979) = 0.8% = 1--specificity. FN rate ([beta]) = FN / (TP + FN) = 331 / (1229 + 331) = 21.0% = 1--specificity. FN = false negative, FP = false positive, TN = true negative, TP = true positive. Table 3. Comparison of Operation Iraqi Freedom/Operation Enduring Freedom (OIF/OEF) veterans selected using OIF/OEF Algorithm versus OIF/ OEF veterans not selected. Confirmed OIF/OEF Veteran * Characteristic Selected Not Selected (n = 1,229) (n = 331) Male, n (%) 932 (76) 242 (73) Conflicts, n (%) OIF (2003-present) 1,090 (89) 270 (82) OEF (2001-present) 217 (18) 92 (28) Global War on Terror- 169 (14) 75 (23) ism (2004-present) Race/Ethnicity, n (%) Caucasian 563 (46) 176 (53) African American 196 (16) 78 (24) Asian American 79 (7) 11 (3) Hispanic (any) 415 (34) 70 (21) Income, n (%): >$50,000 174 (14) 77 (23) Education, n (%): High 283 (23) 59 (18) school only Disabled, n (%) 506 (41) 170 (51) Marital Status, n (%) Single (never married) 338 (28) 78 (24) Married/Partnered 678 (55) 181 (55) Divorced/Widowed 207 (17) 71 (22) Employed (full- or part- 750 (61) 203 (61) time), n (%) Age ([dagger]) 32.5 [+ or -] 8.9 36.8 [+ or -] 11.1 (mean [+ or -] SD) Characteristic p-Value Male, n (%) 0.31 Conflicts, n (%) OIF (2003-present) 0.001 OEF (2001-present) <0.001 Global War on Terror- <0.001 ism (2004-present) Race/Ethnicity, n (%) Caucasian 0.02 African American 0.001 Asian American 0.031 Hispanic (any) <0.001 Income, n (%): >$50,000 0.001 Education, n (%): High 0.042 school only Disabled, n (%) <0.001 Marital Status, n (%) Single (never married) 0.09 Married/Partnered 0.09 Divorced/Widowed 0.09 Employed (full- or part- 0.97 time), n (%) Age ([dagger]) <0.001 (mean [+ or -] SD) * OIF/OEF status confirmed using self-report survey. ([dagger]) Age information is from Department of Veterans Affairs health services use data; all other variables are from survey data. SD = standard deviation.
|Gale Copyright:||Copyright 2010 Gale, Cengage Learning. All rights reserved.|