Chapter 1

Unidimensionality, Agreement and Concordance Probability

The evaluation and comparison of various methods often arise in medical research. For example, the evaluation of reproducibility of a new measurement technique often needs a comparison with the established technique, and image interpretation is often read by two or more observers. In this chapter, we provide a review of the measures of agreement and association, describe the statistical models underlying the Cronbach’s alpha coefficient (CAC) and the backward reliability curve (BRC), the kappa coefficient, and present a general approach based on the concept of concordance probability. In particular, we illustrate the relationship between the concordance probability and various existing measures of agreement and association, namely Kendall’s τ , Somer’s D, area under receiver operating characteristic (ROC) curve and Harrell’s c-index. In addition, we review the estimation of concordance probability and present its large sample properties. Recent developments in the analysis of right censored data are also presented.

1.1. Introduction

The evaluation and comparison of various methods often arise in medical research. For example, the evaluation of reproducibility of a measurement technique often needs a comparison with the established technique, and the interpretation of a computerized tomography (CT) or magnetic resonance imaging (MRI) scan is often read by two or more observers. There is considerable literature on the measure of agreement (see [CHO 04], [BAR 07], [WAT 10], [SHO 04] and [LIN 10]). The methods vary with different types of measurement, i.e. continuous or categorical measurements. When the response variable is continuous, there are several intuitive approaches, namely comparison of means, Cronbach’s coefficient alpha (CAC), various correlation coefficients and the test of slope being 1 in a simple linear regression, as well as alternative methods, the limits of agreement [BLA 86, BLA 99], the concordance correlation coefficient [LIN 89], mean squared deviation and total deviation index [LIN 00], and coverage probability approach [LIN 02]. When the response variable is categorical, kappa statistic, Somer’s D-statistic and logistic regression are commonly used. When one measure is binary and the other measure is continuous, the methods of the ROC curve and logistic regression approach are often applied. These methods are related to typical concordance correlation between repeated measurements through an underlying linear or nonlinear parametric model. Recently developed concordance probability is a non-parametric approach. The concordance probability is commonly used as a measure of discriminatory power and predictive accuracy of statistical models. We show that the concordance probability also provides a unified measure of agreement for different types of measurement.

In this chapter, we present a review of the statistical models underlying the CAC and the BRC in section 1.2, and the kappa coefficient in section 1.3. In section 1.4, we introduce the concordance probability and describe its relationship with Kendall’s τ , Somer’s D and area of ROC curve of sensitivity and 1–specificity for different cutoffs. In section 1.5, we review the estimation of concordance probability and present its large sample properties. In section 1.6, we present recent developments on how to use the concordance probability to assess the agreement among different measures. We present the extension of the approach to the right censored data in section 1.7 and conclude with some discussion in section 1.8.

1.2. From reliability to unidimensionality: CAC and curve

1.2.1. Classical unidimensional models for measurement

Latent variable models involve a set of observable variables A = {X1, X2, …, Xk} and a latent (unobservable) variable θ of dimension d ≤ k. In such models, the dimensionality of A is captured by the dimension of θ, the value of d. When d = 1, the dimensionality of set A is called unidimensional.

In a health-related quality of life (HrQoL) study, measurements are taken with an instrument: the questionnaire, which consists of questions (or items). In such cases, the Xij represents the random response of the jth question by the ith subject and the Xj denotes the random variable generating responses to the jth question.

The parallel model is a classical latent variable model describing the unidimensionality of a set A = {X1, X2, …, Xk} of quantitative observable variables. Let Xij be the measurement of subject i, given by a variable Xj, i = 1, …, n, j = 1, …k, then:

[1.1]

where τij is the unknown true measurement corresponding to the observed measurement Xij and εij a measurement error. The model is called a parallel model if the τij can be divided as:

where βj is an unknown fixed parameter (non-random) representing the effect of the jth variable, and θi is an unknown random parameter effect of the ith subject.

It is generally assumed that θi has zero mean and unknown standard deviation σθ . It should be noted that the zero-mean assumption is an arbitrary identifiability constraint with consequence on the interpretation of the parameter: its value must be interpreted comparatively to the mean population value. In HrQoL setting, θi is the true latent HrQoL that the clinician or health scientist wants to measure and analyze. It is a zero mean individual random part of all observed subject responses Xij, the same whatever the variable Xj (in practice, a question j of an HrQoL questionnaire). It is also generally assumed that εij are independent random errors with zero mean and standard deviation σ corresponding to the additional measurement error. Moreover, the true measure and the error are assumed to be uncorrelated, i.e. cov(θi, εij) = 0. This model is known as the parallel model, because the regression lines relating any observed item Xj, j = 1…, k, and the true unique latent measure θi are parallel.

Model [1.1] can be obtained in an alternative way through modeling the conditional moments of the observed responses. Specifically, the conditional mean of Xij can be specified as:

[1.2]

where βj, j = 1, …, k, are fixed effects and θi, i = 1, …, n, are independent random effects with zero mean and standard deviation σθ. The conditional variance of Xij is specified as:

[1.3]

Assumptions [1.2] and [1.3] are classical in experimental design. The model defines relationships between different kinds of variable: the observed score Xij, the true score τij and the measurement error εij. It is significant to make some remarks about the assumptions underlying this model. The random part of the true measure given by response by the ith individual does not vary with the question number j as the θi does not depend on j, j = 1, …, k. The model is unidimensional in the sense that the random part of all observed variables (questions Xj) is generated by the common unobserved variable (θi). More precisely, let X*ij = Xij – βj be the calibrated version of the response to the jth item by the ith subject, then models [1.2] and [1.3] can be rewritten as:

[1.4]

along with the same assumptions on β and θ and the conditional variance model [1.3].

When both θi and εij are normally distributed, then we have the so-called conditional independence property: whatever j and j′, two observed items Xj and Xj′ are independent conditional to the latent θi.

1.2.2. Reliability of an instrument: CAC

A measurement instrument yields values that we call the observed measure. The reliability ρ of an instrument is defined as the ratio of two variances of the true over the observed measure. Under the parallel model, we can show that the reliability of any variable Xj (as an instrument to measure the true value) is given by:

[1.5]

This coefficient is also known as the intra-class coefficient. The reliability coefficient, ρ, can easily be interpreted as a correlation coefficient between the true measure and the observed measure. When the parallel model is assumed, the reliability of the sum of k variables is:

[1.6]

This formula is known as the Spearman–Brown formula [BRO 10, SPE 10].

The Spearman–Brown formula shows a simple relationship between and k, the number of variables. It is easy to see that is an increasing function of k.

The maximum likelihood estimator of , under the parallel...