Computer Literacy and the Accuracy of Substance Use Reporting in an ACASI Survey

In recent years, audio computer-assisted self-interviews (ACASI) have been demonstrated to increase and presumably improve the quality of drug reporting in epidemiologic research. Surprisingly little research is available, however, regarding the potential limitations of this technique. For example, it is unclear what effects computer literacy may have on the validity of substance use information collected via ACASI. Respondents with limited computer skills may become distracted by the automated technology, requiring the devotion of considerable cognitive effort to the navigation of unfamiliar computer equipment and software that would otherwise be available for use to more carefully process and answer survey questions. In this study, we report findings from a community ACASI survey conducted in Chicago, which are used to address this problem. Using multiple indicators of computer literacy, a covariance structure model was developed to test the hypothesis that persons with low computer literacy skills may report drug use with less accuracy. Biological assays were used to evaluate 30-day cocaine use reporting accuracy. Model findings confirmed a positive relationship between computer literacy and the accuracy of cocaine use reports. Future research should investigate strategies for improving the usability of self-administrated computer reporting systems for persons with little direct computer experience.


Introduction
Over the past several decades, considerable concern has been expressed regarding the quality of estimates of sensitive social behaviors such as drug use that have been traditionally obtained via selfreports during survey interviews (Babor, Brown, & DelBoca, 1990;Harrison & Hughes, 1997;Rouse, Kozel, & Richards, 1985). In response, a body of research has developed to address this problem, both to identify potential sources of measurement error and to develop new strategies for minimizing it (Aquilino, 1994;Fendrich & Johnson, 2001;Kennet & Gfroerer, 2005; Office of Applied Studies, 2001;Turner, Lessler, & Gfroerer, 1992). Perhaps the most important innovation in this regard has been the development of audio computer-assisted self-interviews (ACASI), a methodology designed specifically in recognition of the importance of privacy and respondent comfort when reporting on sensitive topics such as drug use behaviors (O'Reilly, Hubbard, Lessler, Biemer, & Turner, 1994;Turner et al., 1998). For surveys of substance use and other sensitive topics, ACASI has arguably become the methodological standard. The impact this methodology has had is reflected in the National Survey of Drug Use and Health (NSDUH; previously known as the National Household Survey on Drug Abuse), which was redesigned approximately 10 years ago to incorporate ACASI technology into its premier nationwide, recurring drug use monitoring effort.
Surprisingly little research is available, however, regarding the potential limitations of ACASI technology. In particular, it is unclear what effects computer literacy may have on the validity of substance use reports collected via ACASI. Persons with limited computer skills, for example, may become distracted by the automated technology, requiring the devotion of additional cognitive effort to the navigation of unfamiliar computer equipment and software that would otherwise be available for use to more carefully answer survey questions (Couper & Rowe, 1996;Johnson, O'Rourke, Burris, & Warnecke, 2005). Currently, there is no research available that has investigated the effects of computer literacy on ACASI reporting of drug use. The purpose of this study is to evaluate the hypothesis that survey respondents with less computer literacy may be less able to provide accurate reports of current drug use behavior.

Method Sample
Data used for this study came from a survey of English-speaking adults who resided in the City of Chicago, which was originally conducted as part of a methodological study (Fendrich, Johnson, Wislar, Hubbell, & Spiehler, 2004). The survey was fielded from June 2001 through January 2002. Residents between the ages of 18 and 40 were randomly selected to participate in a household drug use survey using a multistage area probability design (Levy & Lemeshow, 1991). At stage 1, census tracts in Chicago were randomly selected. At stage 2, one block was randomly selected from within each sampled tract. At stage 3, every household on the sampled block was screened for eligibility. At stage 4, one 18-to 40-year-old adult was selected at random from within each eligible household. Households initially were sent an advance letter and then visited by a trained interviewer from the University of Illinois at Chicago Survey Research Laboratory. To provide respondents with a confidential environment in which to respond, the drug use portion of the survey was administered on laptop computers using ACASI technology. The substance use questions were similar to those used in the NSDUH, as was a brief, standardized tutorial regarding use of the ACASI system that respondents were provided with before beginning to answer questions. The survey averaged 57.5 min in length (SD ¼ 22.4) and primarily focused on substance use and treatment experiences. The study was approved by the University of Illinois at Chicago Institutional Review Board (UIC IRB#1998-0550).
A total of 627 interviews were completed. We used American Association for Public Opinion Research (2000) definitions for response rates (formula #3) and cooperation rates (formula #1). According to these definitions, the response rate is the number of completed interviews divided by the eligible sample, and the cooperation rate is the number of completed interviews divided by the sum of the number of completed interviews and the number of refusals. Note that because those in the eligible sample include potential subjects who were never contacted by the interviewers despite repeated attempts, the response rate tends to be lower than the cooperation rate. The overall response and cooperation rates for this study were 40% and 74%, respectively. These rates reflect the challenges of conducting survey interviews in urban environments, where response rates tend to be lower for many reasons (Groves & Couper, 1998). When restricted-access, high-rise apartment buildings are excluded from consideration, the comparable response and cooperation rates were 51% and 75%, respectively.

Drug Use Assessment
Following all drug use reporting, a separate consent to participate in three drug-testing procedures was requested from subjects: hair testing, saliva testing, and urine testing. Note that respondents were not informed of the drug-testing procedures until after the drug survey portion of the study had been completed. The order of presentation for testing was random, and subjects were not asked to participate in the next procedure until they had decided upon participation on the previous procedure. Hair samples were taken by the interviewers, who were instructed to cut approximately 60 strands of hair from the head of potential subjects. Subjects judged by interviewers as having insufficient hair length (approximately less than one half inch in length) were deemed ineligible to participate and were not recruited into the procedure. For saliva testing, subjects were asked to spit into a plastic tube presented to the subject by the interviewer. For urine testing, subjects were asked to privately urinate in a plastic cup provided to the subject by the interviewer. Among survey respondents, the participation rates for each drug test were 90.4% for saliva, 76.3% for urine, 66.9% for hair, and 91.6% for providing at least one specimen. Detailed analyses of respondent participation in these drug-testing procedures are reported elsewhere .
All biological specimens of sufficient quantity were assayed for the indications of cocaine and several other substances by U.S. Drug Testing Laboratories in Des Plaines, Illinois. Specimens that screened positive were confirmed using gas chromatography/mass spectrometry. Because of concerns about differential participation in hair testing by race/ethnicity and by gender  and additional concerns about potential racial bias in hair testing (Wolff et al., 1999), this study examines urine and saliva test results only.
Although the vast majority of the 627 subjects (90%) self-administered the substance use questions via ACASI, subjects also could opt to have their questions administered by the interviewer. Of the 563 respondents who used ACASI, urine and saliva specimens sufficient to be assayed were available for 430. These cases serve as the sample to be analyzed in this report.

Measures
Within each sample, self-reports of cocaine use in the past month were evaluated against saliva and urine drug tests in combination (i.e., whether a positive result was indicated by either test). There was a 95.0% concordance rate between the findings of the saliva and urine tests. Of the two tests, the urine tests identified a greater number of persons as users (6.94%), compared to the saliva samples, which identified 3.45% of the sample as users. Subject reports were coded as concordant if the past-month use was reported and either test was positive or if the past-month use was not reported and both tests were negative. Otherwise, subject reports were coded as discordant.
Four self-report items were used to assess computer literacy. Three were measured on a 7-point scale with only the endpoints labeled. These items included the following: (1) ''On a scale from 1 to 7, where 1 is none and 7 is a lot, how much experience have you had with computers?'', (2) ''On a scale of 1 to 7, where 1 is very uncomfortable and 7 is very comfortable, how comfortable are you with computers?'', and (3) ''On a scale from 1 to 7, where 1 is very poor and 7 is excellent, how would you rate your reading ability?''. The fourth measure was a dichotomous indicator of whether the respondent had a computer at home. To validate the usefulness of these measures as indicators of computer literacy, we examined the associations between each and the respondents' willingness to answer questions via ACASI technology (vs. a traditional interviewer-assisted survey). Indeed, those agreeing to respond via ACASI scored significantly higher (p < .05) on all four measures (results available from authors), suggesting that each discriminated between those who were and were not willing to use a computer during the interview. Presumably, those electing not to use a computer might be expected to be less computer literate.
Several additional measures were used as covariates, given their known associations with drug use reporting (Substance Abuse and Mental Health Services Administration, 2007). These included age, gender, race/ethnicity, and socioeconomic status. Three indicators were available to assess the socioeconomic status, including ordinal measures of education and income, as well as the indicator of home computer ownership listed above, which is also serving as a measure of computer literacy. One of the advantages of the analytic strategy used in this study (see below) is that it affords the opportunity to recognize that some measures are partial indicators of multiple constructs. In this study, we conceptualize home computer ownership as an indicator of both computer literacy and socioeconomic status.

Analyses
The relationship between computer literacy and drug use reporting accuracy was examined using covariance structure modeling (Hayduk, 1987), also known as structural equation modeling (SEM). Structural equation models represent the underlying structure among variables in terms of latent variables represented by multiple indicators, and regression paths connecting the latent variables. Covariance structure analysis estimates the population parameters in these models using a sample of covariances. The PRELIS software package was used to construct the covariance matrix used in these analyses, and the LISREL 8.51 software was used to conduct the covariance structure analyses (Jöreskog & Sörbom, 1993a, b). Our analyses proceeded in two phases. First, following preliminary variable screening, a measurement model was constructed using confirmatory factor analysis to determine whether the four measures of computer literacy and the three indicators of socioeconomic status each served as useful indicators of their respective latent constructs. Second, a structural model was fit to evaluate the independent relationships of these two latent constructs and indicators of age, gender, and race/ethnicity, with the accuracy of cocaine use self-reports.

Results
The demographic composition of the sample analyzed is summarized in Table 1. In addition to presenting gender, race, age, education, and income distributions, this table indicates that the concordance of past 30-day cocaine use reporting was 94.2% (91.6% were concordant for nondrug use; 2.6% were concordant for drug use). Of those that were discordant, 5.6% failed to report past 30-day cocaine use but tested positive, and 0.2% reported cocaine use but tested negative. Table 2 summarizes the distributions of the four indicators of computer literacy in column 1. The associations between each literacy indicator and drug use reporting concordance also are presented in columns 3-4. Two of the four literacy indicators (computer at home and experience with computers) were significantly associated with reporting concordance. For all four indicators, literacy values were greater among those respondents who provided concordant drug use reports.
The confirmatory factor analysis was next specified to verify that our proposed indicators of computer literacy and socioeconomic status functioned in a manner consistent with each representing one of the two latent variables of interest. The measurement model, shown in Table 3, confirmed the four-indicator model of computer literacy and the three-indicator model of socioeconomic status, as each indicator was significantly associated with its hypothesized latent variable. This measurement model provided a good fit to the observed data (model w 2 ¼ 7.18, df ¼ 5, ns; RMSEA ¼ 0.03; CFI ¼ 1.00; GFI ¼ 1.00) and also revealed a strong association between these two latent constructs (standardized phi ¼ 0.67, p < .0001).
Using these latent measures, we next examined a structural model that evaluated the association between computer literacy and drug use report concordance while adjusting for the effects on concordance of socioeconomic status, gender, age, and race. This model, presented in Table 4, confirmed the bivariate findings (reported in Table 2) that suggested a positive association between computer literacy and the accuracy of drug use reporting. Socioeconomic status also was positively associated with reporting concordance, and white respondents were found to provide more concordant drug use reports. Gender and age were not associated with drug use report concordance.

Discussion
Consistent with our working hypothesis, an independent association was found between computer literacy and the accuracy of self-reported drug use behavior among adults in a community survey. This is an important finding given the now widespread application of ACASI technology for the Observed variables Self-rated computer experience 1.00 a Self-rated reading ability 0.26*** (0.03) Self-rated comfort with computers 0.55*** (0.07) Have a computer at home 0.14*** (0.02) 0.91*** (0.07) Income 1.00 a Education 1.33*** (0.11) Model fit statistics Model chi-square (df) 7.18 (5)   collection of self-reported drug use behavior. ACASI was originally developed for the purpose of minimizing the social desirability demands and reporting errors associated with the collection of data relevant to sensitive and/or stigmatizing social behaviors (Turner et al., 1998). The evidence reported here suggests that the introduction of this self-administered computer technology into the survey interview process may additionally have the unintended consequence of reducing reporting quality for persons less familiar with it. Although not examined here, similar problems may exist with other survey-based modes of data collection that rely all or in part on respondent ability to complete computer-assisted self-interview (CASI) questionnaires, such as text-CASI (Couper, Tourangeau and Marvin, 2009), video-CASI, and telephone audio-CASI (Harmon et al., 2009). We note that the effects of computer literacy on reporting errors in this study were found within a sample of younger adults aged 18-40. Because the prevalence of illicit drug use is much higher among those in younger adult age cohorts (Substance Abuse and Mental Health Services Administration, 2007), these findings are particularly meaningful, as it suggests that computer illiteracy may represent an important source of measurement error within that segment of the general population at increased risk for illicit substance use behavior. We also observe that younger adults are generally assumed to be universally computer literate, an assumption that is not supported by the data reported here.
It is also important to note that our analyses specifically removed those respondentsapproximately 10% of the sample-who agreed to participate in the survey but elected not to self-administer the questionnaire using the ACASI system. Although a relatively small percentage overall, these respondents represent an important subgroup who were uncomfortable with this technology and were willing to express those concerns to their interviewer. Our findings further suggest that there are additional respondents who may also be reluctant to divulge similar discomfort to an interviewer and will instead attempt to respond using unfamiliar technology, thereby threatening the quality of their self-reports.
One potential solution that researchers using ACASI might consider would be to include brief screening questions to identify those respondents at elevated risk for having difficulty navigating the computer technology (i.e., those with less computer literacy) to provide them with additional assistance and/or support in completing this task. Doing so while preserving respondent privacy, one of ACASI's primary objectives, will be critical. Future researchers should develop experiments to identify the effectiveness of such strategies. Although several measures have been developed (Bunz, 2004;Hargittai, 2005Hargittai, , 2009, additional research to improve methods for the rapid, reliable assessment of computer and digital literacy also should be encouraged. To echo this last point, an important limitation of this study is the assessment of computer literacy, as the measure constructed for use here was post hoc in nature. However, there is currently no consensus and little empirical evidence regarding how best to measure this construct (Hargittai, 2009). Although the measure we used does exhibit reasonable face and construct validity, we acknowledge that additional work will be necessary to confirm our findings using measures specifically designed for that purpose. In addition, this study identifies only one factor that may inhibit ACASI's effectiveness. Other factors, including visual and hearing impairments, should be considered as well.
It is also possible that the measure of computer literacy examined is so strongly associated with general reading ability and/or general educational achievement that it is one of these measures, rather than computer literacy per se, that is responsible for differential drug use reporting quality. Indeed, our latent measures of computer literacy and socioeconomic status, which both share home computer ownership as an observed indicator, are moderately correlated with one another. As Table  3 indicates, however, each of these latent constructs also have independent associations with drug use report concordance, suggesting that computer literacy does in fact have an independent influence on reporting behavior.
Several strengths of this study also should be recognized. First, the collection of survey data from a random community sample enhances the external validity of this research. Second, although imperfect (Cone, 1997;Wolff et al., 1999), the use of biologic assays to confirm self-report accuracy substantially enhances the internal validity of this research beyond studies based solely on selfreport data. In addition, although these data were collected approximately 7 years ago, the observed relationships among the variables examined remain valid. We nonetheless encourage replication.

Declaration of Conflicting Interests
The authors declared no potential conflicts of interests with respect to the authorship and/or publication of this article.