Consistent Differential Discrimination Model Estimation

ABSTRACT A novel factor-analytic model—the differential discrimination model—for assessing individual differences in scale use has been recently introduced, together with a three-stage estimation approach for model fitting. Unfortunately, the second-stage estimator and, as a consequence, the third-stage estimator of this procedure are not consistent. In this article we show that (a) the differential discrimination model can be expressed in a structural equation model framework, and (b) consistent and simultaneous estimation of all model parameters can be achieved using standard SEM software.


Introduction
If individuals differ in their scale use when answering questionnaire items for assessing a specific trait, then these differences have to be considered a systematic influence on the item responses. If not accounted for by a model for item calibration, the model is-at least to some degree-misspecified, and the validity of the conclusions based on the model may be negatively affected.
For a continuous item response format, a standard linear factor model can be used for item analysis (Mellenbergh, 1994). In this context, a continuous item response is defined by a check mark on a line segment between a lower and upper scale endpoint. The numerical value of that response is the distance between the lower endpoint and the mark. Check marks below the scale midpoint are often regarded as disagreement and marks above as agreement (e.g., Samejima, 1973).
Using standard linear factor analysis to model these data assumes that all respondents use the scale in the same way. However, this is not necessarily true. A new approach to measure individual differences in scale use is the differential discrimination model (DDM) suggested by Ferrando (2014). The DDM can be regarded as a linear factor model for continuous responses containing a latent variable representing the trait of interest and an additional latent variable accounting for individual response style. This second latent variable enters the model in a multiplicative way. Such a multiplicative latent variable has been used also for modeling response style in ordinal data (e.g., Jin & Wang, 2014;Johnson, 2003).
If this multiplicative latent variable is above average, individuals tend to prefer extreme responses; if it is CONTACT Dirk Lubbe Dirk.Lubbe@psychol.uni-giessen.de Justus Liebig University Giessen, Otto-Behaghel-Strasse  F,  Giessen, Germany. Supplemental material for this article can be found on the publisher's website.
below average, responses closer to the scale midpoint are endorsed. Jin and Wang (2014) refer to these possibilities as extreme and mild response styles, respectively. Because this multiplicative variable can capture both tendencies, it will be referred to simply as "response style. " The three-stage procedure proposed by Ferrando (2014) for estimating the model parameters, while computationally simple, does not yield consistent parameter estimates. Clearly, consistency is a minimal requirement for any "reasonable" estimator. Generally speaking, it requires the estimator to approach the true value of the parameter in probability as the sample size increases. More formally, a sequence of estimatorsθ n is consistently estimating the true value θ 0 if for all ε > 0, Pr(|θ n − θ 0 | > ε) → 0 as n → ∞. For example, by the law of large numbers, the sample meanx n consistently estimates the population mean μ. By integrating the DDM into a general structural equation model framework (e.g., Bentler & Weeks, 1980;Muthén, 1984;Jöreskog, 1973), we show that consistent estimation can be achieved using one's preferred SEM software package, (e.g., MPLUS, AMOS, LISREL, and SAS PROC CALIS).
This article is organized as follows. First, we explain the DDM and relate the model to the SEM framework. The consistency of the estimation within this SEM-based approach is discussed and illustrated with a small-scale simulation study. We then give a numerical application by fitting the DDM to an artificial data example. Next, the inconsistency of the previously proposed three-stage estimation procedure is demonstrated. Finally, we give a short discussion of the results. As supplementary material we provide MPLUS code with which the DDM has been fitted to the data example.

Differential discrimination model
The DDM is used for analyzing responses given by n individuals to each of p items. These responses are restricted to an interval between a lower and upper scale endpoint, having a midpoint denoted by c. The scale endpoints are valid scale values. For example, if the responses are restricted to the unit interval, the scale midpoint is c = 0.5. Specifically, the model for the observed response of the ith individual to the jth item is where θ i is the trait the items are supposed to measure and α i is the response style variable. Because response style enters the model multiplicatively, its value is restricted to the positive real numbers. For model identification, the constraints E(θ i ) = 0, Var(θ i ) = 1, and E(α i ) = 1 are imposed on the latent variables. The model contains the following parameters: p loadings γ j , p difficulties β j , and p unique variances Var(e i j ) = u j . In addition, the response style variance is an important parameter, which will be denoted by σ 2 α . Thus, the DDM has a total of 3p + 1 parameters. A large value of σ 2 α indicates that individuals differ considerably in their use of the response scale. Values of α larger than 1.0 correspond to individuals who tend to give answers far from the scale midpoint. By contrast, individuals with values of α smaller than 1.0 tend to give answers close to the scale midpoint.
To illustrate, we show expected responses of two individuals having the same value of 0.5 for θ but different response styles. Specifically, we use α 1 = 1.2 for person 1 and α 2 = 0.8 for person 2. The expected responses to three items are shown in Figure 1 assuming (a) a scale midpoint of 5.0; (b) difficulties of β 1 = −1.0, β 2 = 0.0, β 3 = 1.0; and (c) loadings of γ 1 = 1.0, γ 2 = 1.0, γ 3 = 1.5. As can be seen from Figure 1; all responses of person 1, having the larger α-value, deviate more from the scale midpoint of 5.0.
Note that the γ j parameters are unstandardized loadings; that is, they are not correlations between the observed responses and the factor. Nevertheless, for simplicity, we refer to them as loadings. In terms of distributional assumptions, the model requires the latent variables θ i and α i as well as the vector of zero-mean residuals (e i1 , . . . , e ip ) to be mutually independent.

Relating the DDM to the SEM framework
The notation we use for the structural equation model is as follows. Let x i be the (p × 1) vector of observed item responses, ν and e i be (p × 1) vectors of item intercepts and zero-mean residuals, and η i be a (2 × 1) vector of latent variables, not necessarily having zero means. Moreover, let be the (p × 2) matrix of factor loadings, be the (2 × 2) diagonal matrix of factor variances, and Var(e i ) = U be the (p × p) diagonal matrix of unique variances. Using this notation, the DDM can be represented as the following standard linear factor model: The implied mean vector and variance-covariance matrix are Var To show how the DDM can be related to the SEM framework, we proceed in two steps. In the first step, we introduce the key-concept "scaled trait" and calculate the observed variable means implied by the model. Second, we calculate the variance-covariance matrix of the latent variables. While response style enters the model multiplicatively, replacing trait with scaled trait allows one to retain additivity of the latent variables' contributions on the right side of the factor model (see [5] in the following).
To begin with the first step, let scaled trait be defined as τ i = α i θ i . Thus, the response style variable, α i , moderates the trait inasmuch as it increases or decreases its value depending on whether a person's α i value is below or above its mean of 1.0. In terms of scaled trait, the model becomes where we have used δ j = −γ j β j . Next, we define Using these expressions in (2) yields the matrix representation of (5). Calculating the unconditional mean of x i j yields for j = 1, . . . , p. This follows because (a) τ i and e i j have zero means and α i has a mean of 1.0 and (b) τ i , α i , and e i j are mutually independent. Arranging the variable means as a column vector, and using E(η) = (0 1) , yields (3). Note that the ν-vector is not equal to the item means, but contains item intercepts, equal across items, fixed at the scale midpoint.
In the second step, we calculate (a) the variance of the scaled trait and (b) the covariance between scaled trait and response style. For calculating Var(τ i ), we use the formula for the variance of the product of two independent random variables (Goodman, 1960). This yields α . In addition, a zero covariance between τ i and α i follows because . From the assumed independence between θ i and α i and the identification constraint E(θ Collecting these results in the latent variables' covariance matrix yields Thus, the DDM implies one constraint on the latent variable variances. Given and as defined in the preceding, together with U , the variances and covariances of the observed variables are given by (4). Given p variable means as well as p(p + 1)/2 nonredundant elements in the sample covariance matrix, the DDM has (p 2 − 3p − 2)/2 degrees of freedom. To obtain degrees of freedom greater than zero requires at least four observed variables.
In summary, rewriting the DDM as a general structural equation model requires only minor modifications. First, we introduce scaled trait as product of response style and trait. Second, rather than having difficulty parameters, the structural equation formulation requires δ-parameters, which are simply the difference between the unconditional means of the jth variable and the scale midpoint. If we define μ j as the expected value of x i j , then δ j = μ j − c. The difficulty parameters β j of (1) can be expressed as β j = (c − μ j )/γ j = −δ j /γ j . Both types of parameters express the deviations between the variable means and the scale midpoint in closely related metrics. If the model is fitted as a structural equation model, the difficulty estimates can be calculated readily from estimates of δ j and γ j .
It is remarkable that several of the equations given in the preceding were already contained in Ferrando (2014). In particular, he recognized that the covariance structure implied by the DDM can be conceived of as the covariance structure of a standard two-factor model with uncorrelated factors (see p. 339). Clearly, the concept of a scaled trait is implicit in this representation. However, he did not give the linear model in terms of scaled trait and response style from which this covariance structure follows (see our Equation [5]).

Distributional assumptions and consistent estimation
So far, only general distributional assumptions have been made. In fact, specific distributional assumptions, such as normally distributed θ i or α i are not needed to achieve consistent parameter estimates. If the model from which the mean and covariance structures are derived holds, then consistent estimation can be obtained by fitting a structural equation model that correctly accounts for this structure. This follows from the general properties of the discrepancy function approach to model fitting (Browne & Arminger, 1995, p. 188). This approach is implemented in MPLUS, AMOS, LISREL, and SAS PROC CALIS. Therefore, virtually all popular structural equation software packages can be used to achieve consistent parameter estimates.
For inferential purposes, such as judging model fit and calculating standard errors, multivariate normality of the observed variables is usually required. Given the linear structure of the model, multivariate normality follows if the latent variables themselves are normally distributed. However, this is a questionable assumption for both response style and scaled trait. First, response style is constrained to be positive having a mean of 1.0. If its standard deviation is large, a skewed response style distribution is expected. Second, scaled trait τ i is the product of two other random variables, specifically, α i and θ i , and, for this reason, may not be normally distributed.
We illustrate the performance of the SEM-based estimation procedure with the help of a small-scale simulation study. To keep the number of reported parameter estimates small, responses to only five items were simulated under the model. The true parameters are γ j = 0.1 and u j = 0.04 for j = 1, . . . , 5. Furthermore, we use δ 1 = 0.0, δ 2 = 0.1, δ 3 = 0.2, δ 4 = 0.3, δ 5 = 0.4. The response style variance was set to σ 2 α = 0.25. The latent variable θ was sampled from a standard normal distribution, and a lognormal distribution was used for sampling α values. Simulations were conducted for sample sizes of 100, 250, and 500. For each of the three sample sizes, the simulation was replicated 1,000 times. Model fitting was done in MPLUS using maximum likelihood estimation. The averaged σ 2 α estimates are 0.198 for n = 100, 0.235 for n = 250, and 0.243 for n = 500. All other parameter estimates are displayed in Table 1.
Obviously, all estimates approach their true values as sample size increases, illustrating the consistency of the SEM-based estimation approach. While the δ estimates appear to have no bias even for samples of size 100, the u and γ estimates show some bias in this case. For a sample size of 250, the bias is negligible. With only five items and a sample size of 100, the model estimation did not converge in about 20% of the simulations. For n = 250 and n = 500 nonconvergences were reduced to 3% and 0%, respectively. These nonconvergence rates decreased as the number of items is increased. Clearly, five items per individual do carry only a limited amount of information on response style values. Thus, for small sample sizes, using more than five items is recommended.

Example
To illustrate the model, we use an artificial data set containing the responses of 500 individuals to 5 items. The responses are assumed to lie on a scale ranging from 1 to 9, with a scale midpoint of 5. From the raw data, we have calculated the following vector of meansx and the variance-covariance matrix S:  Table 2.
The δ j parameters are very close to the difference between the item means and c. However, they do not match perfectly. This is expected because models for means and covariances will not generally fit the sample means exactly (e.g., Browne & Arminger, 1995, p. 193). To illustrate, for item 3 we obtain δ 3 = −0.751, whereas the differencex 3 − c = 4.260 − 5 equals −0.740. The item loadings γ j show some variation across items, ranging from 0.569 to 1.018. The unique variances u j indicate that items 1 and 4 are explained best by the model. MPLUS code for fitting the model is given in the Appendix. Note that the robust standard errors obtained from the so-called sandwich estimator reported in the preceding require raw data. They cannot be reproduced from the sample moments. In this data example, the nonrobust standard errors (not reported) are very similar to the robust standard errors. For completeness, we also give the true model parameters from which the raw data were generated. These values are γ 1 = 0.6; γ 2 = 0.8; γ 3 = 0.9; γ 4 = 0.6; γ 5 = 1.0; δ 1 = −1.4; δ 2 = 0.5; δ 3 = −0.6; δ 4 = 1.4; δ 5 = 0.4; u 1 = 0.09; u 2 = 0.36; u 3 = 0.25; u 4 = 0.09; and u 5 = 0.25. Ferrando (2014) proposed the following three-stage procedure for fitting the DDM. In the first stage, the delta parameters are estimated. For this purpose, the responses are centered by calculating y ij = x ij − c. Thus, the scale midpoint of the centered responses equals zero. Then, their means are calculated. Because the expected value of y ij is equal to δ j (see [6]), each sample meanȳ j is estimating δ j for j = 1, . . . , p. In the second stage, the sample covariances s jk = i (y ij −ȳ j )(y ik −ȳ k )/(n − 1) are calculated and the response style variance is estimated using the following formula (Ferrando, 2014, eq. 14, p. 395).

Inconsistency of the three-stage estimation procedure
Based onσ 2 α , a reduced covariance matrix is obtained whose elements are for j = k. In the third stage, the loadings γ j and unique variances u j are estimated by an unweighted least-squares (ULS) factor analysis of the reduced covariance matrix (Jöreskog, 2003).
While the first-stage estimates are consistent, the estimates at the second and third stages are not. The inconsistency of the response style variance estimate follows from the so-called continuous mapping theorem (e.g., van der Vaart, 1998, p. 7). This theorem implies that continuous functions of consistent estimators are themselves consistent. It is well known that the sample means and sample covariances consistently estimate the corresponding population parameters if they exist. Thus,ȳ j and s jk consistently estimate the δ j and σ jk parameters, respectively, where σ jk refers to the elements of the variance-covariance matrix in (4). To determine the value against whichσ 2 α converges in probability, we replace the estimates on the right hand of (7) with the true parameter values. Specifically, theȳ j are replaced with δ j and the s jk are replaced with Simplifying the resulting expression yields Of course, this formula requires that j =k (δ j δ k ) 2 > 0, which is true if at least two δ parameters are different from zero. Because (9) yields the value against whichσ 2 α converges in probability, we refer to it as the "target value" of σ 2 α . Clearly, it can be seen from (9) thatσ 2 α will estimate σ 2 α consistently, if and only if j =k Therefore,σ 2 α is not generally a consistent estimate of the response style variance.
To demonstrate that only special parameter combinations will fulfill this condition, we give a numerical example. Assuming four items having equal γ parameters, Equation (10) simplifies to γ 2 j =k δ j δ k = 0. In this case, δ 1 = δ 2 = δ 3 = 1 requires δ 4 = −1 for this condition to be fulfilled. For these parameter values, σ 2 α would be consistently estimated in the second stage of the three-stage approach. While one could produce additional examples for which (10) holds, they would all require special parameter combinations. Because our SEM-based estimation procedure is consistent under any set of parameters, it is superior to the three-stage approach.
Formula (9) can be used also to investigate why the simulation studies reported by Ferrando (2014) did not indicate the inconsistency of his second-stage estimator. In these simulation studies, the following parameter values were used: γ j = 0.1 and u j = 0.04 for j = 1, . . . , 30. For the δ parameters, only their range is reported. Therefore, we select δ parameters equally spaced within this range. Specifically, we use δ 1 = 0.27, . . . , δ 30 = −0.27. While all parameter values were constant across simulation studies, Ferrando selected for σ 2 α values of 0 for the first, 0.25 for the second, and 0.75 for the third (continuous responses) simulation study. Table 3 shows the results of (9) for these parameter values as well as the average estimates of σ 2 α reported by Ferrando (2014) in his Table 1. Given the sampling error necessarily involved in any simulation study, the small difference between target and true values in Table 3, make the inconsistency ofσ 2 α virtually impossible to detect. Clearly, Ferrando's results confirm the target values obtained from (9) and vice versa.
Of course, larger differences are expected under different parameter settings. We illustrate this with just one example. Decreasing the number of items from 30 to 10, while leaving all other parameters unchanged, yields for σ 2 α = 0.75 a target value of 0.678. Thus, simulation studies based on only ten items would show the inconsistency ofσ 2 α more clearly. Because the estimate of the second stage is required for calculating the reduced covariance matrix (8) analyzed in the third stage, the estimates of γ j and u j are expected to be inconsistent as well. We demonstrate this by computer simulation considering the γ parameters only.
We simulated responses using the exact same parameter values as in our simulation study reported previously with which we illustrated the consistency of the SEMbased procedure. Again, these values are γ j = 0.1 and u j = 0.04 for j = 1, . . . , 5 and δ 1 = 0.0, δ 2 = 0.1, δ 3 = 0.2, δ 4 = 0.3, δ 5 = 0.4. The response style variance was set to σ 2 α = 0.25. Based on these values, (9) yields a target value for σ 2 α of 0.41, which is very different from the true value of σ 2 α = 0.25. Thus, considerable bias can be expected. To demonstrate the large sample properties of the third-stage estimators, simulations were conducted for sample sizes of 500, 1,000, and 5,000. Each simulation setting was replicated 1,000 times. Again, the latent variables were sampled from a standard normal distribution for θ and a lognormal distribution for α. The results are displayed in Table 4.
Apparently, the estimates deviate considerably from the true values and do not approach the true γ parameters as sample size increases. This suggests that the third-stage estimates are inconsistent as well.

Discussion
In our opinion, the potential for applying the DDM is large because the item responses do not have to be strictly continuous. In fact, as outlined by Ferrando (2014) and Mellenbergh (1994), models for continuous responses may also be useful for analyzing graded responses if the number of categories is large. As our data example shows, the model can be applied to summary and raw data alike. No preprocessing of the raw data or summary data is required, as would be necessary if the responses are assumed to lie in the unit interval. What is required is a clearly defined scale midpoint. Beyond item calibration, the DDM can be used for factor score estimation. The scores produced by structural equation software represent scaled trait and response style. Because scaled trait is a product of two latent variables, its interpretation is complex. On the other hand, θ i can be regarded as trait estimate unaffected by response style and may be preferred for diagnostic purposes. If trait θ i rather than scaled trait τ i is desired, this is easily calculated as θ i = τ i /α i .

Article information
Conflict of Interest Disclosures: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.
Ethical Principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.
Funding: This work was not supported by any funding agency.
Role of the Funders/Sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.