Empirical Estimates Of Reliability

Concept of ReliabilityThe concept of reliability is of the consistency orprecision of a measureWeight exampleReliability varies along a continuum, measures arereliable to a greater or lesser extentNot an all or nothing qualityNewsom, Psy 521/621 Univariate Quantitative Methods1

Concept of ReliabilityThe opposite of consistency and precision isvariability due to random measurement errorReliability is lack of random measurement errorRandom error is unexplained variation that is notsystematicIf variability is random, there will be someoverestimates and some underestimatesOn average estimate is accurateNewsom, Psy 521/621 Univariate Quantitative Methods2

3Concept of ReliabilityTarget asurement/validity and reliability.htmlNewsom, Psy 521/621 Univariate Quantitative Methods

4Theoretical FoundationsClassical Test Theory (CTT)Observed ScoreXo Note: many texts use X T ENewsom, Psy 521/621 Univariate Quantitative MethodsTrueScoreXt Error Xe

Theoretical FoundationsReliability is the proportion of the observed score22variance, so , that is due to the true score, st2The smaller the error variance, se ,the greaterproportion that is due to true score variance andthe higher the reliabilityIf proportion is 1.0, then no error variance – perfectreliabilityIf proportion is 0.0, then all error variance – noreliability and all noiseNewsom, Psy 521/621 Univariate Quantitative Methods5

Theoretical Foundations6TrueReliability True Errorst2Rxx 2 2st sest2 2soNote: your text uses Rxx as the symbol for reliability but most texts use ρxx (rho) or rxxNewsom, Psy 521/621 Univariate Quantitative Methods

AttenuationMeasurement error attenuates correlationsImagine if a score was only random errorIf observed scores are a function of true scores andmeasurement error, degree of error will cloud estimationof the relationship between two variablesExample: child’s age and reading abilityNewsom, Psy 521/621 Univariate Quantitative Methods7

8AttenuationRemember that measurement error will increase thevariance of the observed score, so the denominatorin the correlation coefficient will be largerThis makes the estimate of the correlation smaller inmagnituderxy( X X )(Y Y ) ( X X ) (Y Y )Newsom, Psy 521/621 Univariate Quantitative Methods22cxysx s y

Attenuationrxo yo rxt yt Rxx Ryyrxo yo is the correlation estimated from the data(between observed scores), rxt yt is the correlationbetween the true scores (if we could know them),Rxx and Ryy are the reliabilities of the two measuresNewsom, Psy 521/621 Univariate Quantitative Methods9

AttenuationExample 1: say the reliability for my guess at the age is.6 and the measurement of reading ability is .5 andthat the true score correlation is .4rxo yo rxt yt Rxx Ryy .4 (.5 )( .3 .4 (.548) .21.6 ) .4 When the true score correlation is .4, the estimatedcorrelation is .21—a substantial underestimate—almost half the value!Newsom, Psy 521/621 Univariate Quantitative Methods10

AttenuationExample 2: say the reliability for my guess at the age is.9 and the measurement of reading ability is .9 andthat the true score correlation is .4rxo yo rxt yt Rxx Ryy .4 (.9 )( .81 .4 (.9 ) .36.9 ) .4 When the true score correlation is .4, the estimatedcorrelation is .36—not nearly as badNewsom, Psy 521/621 Univariate Quantitative Methods11

12MeansRemember that random measurement error sometimes leadsto overestimates and sometimes leads to underestimatesOn average the estimate will be accuratefVariability around the averageAverageweightNewsom, Psy 521/621 Univariate Quantitative Methods

13MeansComparing meanst X1 X 2s12 s22 n1 n2If X1 and X2 observed scores have larger variance (s12 and s22)than their true score counterparts, then the denominator willbe larger and the t will be smaller, so less likely to be significantNewsom, Psy 521/621 Univariate Quantitative Methods

14MeansComparing meansd xo X o1 X o 2so21 so222Also seen in the estimate of the effect size, which gives themagnitude of the group difference (where o1 and o2 subscriptsindicate observed values for group one and two)Newsom, Psy 521/621 Univariate Quantitative Methods

Estimating ReliabilityTest-retest reliabilityRepeat the test two or more times to see how similar themeasurements areCalculate the correlation between the measurement occasionsProblem is that in the interval between the measurementoccasions the attribute may have changedSmall time interval needed in between measurements withoutcontamination from recallNewsom, Psy 521/621 Univariate Quantitative Methods15

Estimating ReliabilityParallel testsTwo tests are parallel if their true scores are the same and theyhave the same standard deviationTheoretical notion, because it is not possible to know withabsolute certainty that two tests are exactly parallelNewsom, Psy 521/621 Univariate Quantitative Methods16

Estimating ReliabilityAlternative forms reliabilityIf we could create two parallel or alternative forms of ameasure, we could estimate reliability of the measurewithout repeated measurementse.g., standardized tests, like the SAT and GRE, use alternative testformsNewsom, Psy 521/621 Univariate Quantitative Methods17

Estimating ReliabilitySplit-half reliabilityCan develop a larger test and correlate twohalvesProblem is how best to split up the teste.g., what if the first half and second half differ?Newsom, Psy 521/621 Univariate Quantitative Methods18

Estimating ReliabilityDomain sampling theory (model)What if we considered a set of items from a test to be from alarger pool (domain, population) of items from the same testWe could think of every item as a small parallel test, a testlet orsubtestNewsom, Psy 521/621 Univariate Quantitative Methods19

Estimating ReliabilityDomain sampling theory (model)If we view each item as good representations of the true scoreand each as a random selected item from a domain orpopulation of possible items, then we can relax theassumption that each test is strictly parallelInstead we only need to think of them as on average equallyrepresenting the domainNewsom, Psy 521/621 Univariate Quantitative Methods20

Estimating ReliabilityInternal reliabilityThe domain sampling idea allows us to use the correlationsamong items to gauge the reliability of a measureThis is the basis of internal reliability, such as the type ofreliability assessed by Cronbach’s alphaNewsom, Psy 521/621 Univariate Quantitative Methods21

Cronbach’s AlphaPreliminary steps Generate descriptive statistics, including means, standarddeviations (and/or variances, skewness and kurtosis) Obtain frequency tables and histograms Check for errors in entry, coding, etc. Variables do not need to be normally distributed, but when theyare highly skewed or kurtotic or they respondents have not usedthe full range of values, you may want to consider the wording ofthat item. Check correlations to confirm scoring direction is correct andpotentially eliminate items that are supposed to correlate that donotNewsom, Psy 521/621 Univariate Quantitative Methods22

23Cronbach’s AlphaCronbach’s alpha (Cronbach, 1951) is an estimate ofinternal reliability (sometimes called the “consistencycoefficient”)Conceptually based on the proportion of true score tototal observed score variancest2Rxx st2 se2Newsom, Psy 521/621 Univariate Quantitative Methodsst2so2

24Cronbach’s AlphaIf we can estimate the proportion of the observed scorevariance that is due to measurement error, then we canestimate reliabilityse2Proportion error 1 2soCronbach’s alpha (α) raw score form is:2k siα 1 2k 1 sX k number of items, si2 is the variance for each item, and sx2 is the variance forthe composite scale score (as a sum of the items)Newsom, Psy 521/621 Univariate Quantitative Methods

25Cronbach’s AlphaThe domain sampling model conceptualizes the items (testlets orsubtests) as retests, so that the average correlation betweenthese subtests is a measure of reliabilityCronbach’s alpha in the standardized form is:α krii ′1 ( k 1) rii ′rii ′ is the average correlation among all pairs of items, and k is thenumber of itemsNewsom, Psy 521/621 Univariate Quantitative Methods

Cronbach’s Alpha26The standardized coefficient alpha is the alpha for the set ofitems after they have been standardized (converted to zscores) and will be equal or higher than the raw score versionRaw score alpha assumes the variances of the of the items areequal, and if they are not, the raw score estimate will besmaller than the standardized estimateUsually similar, but when items are on very different scales (e.g.,some 5-point and some 9-point scales), the difference may belargerNewsom, Psy 521/621 Univariate Quantitative Methods

Cronbach’s AlphaComposites scores calculated by the sum or mean tend toweight items with larger variances more heavilyStandardizing items before computing the composite willequally weight them, because variances are all equal to 1In most applications, researchers do not bother to dostandardize items, sometimes because the original metric islost (e.g., average of items on a 7-point no longer between 1and 7, but are z-score values instead)Newsom, Psy 521/621 Univariate Quantitative Methods27

Cronbach’s AlphaWhat is an acceptable alpha? Exceeding .70 is widelymentioned as a cutoff for acceptable reliability, but what is“acceptable” or “good” depends heavily of theconsequences of using a measure with some certain level ofreliability.Many scales with an alpha of .70 can be improved, however.And this value has been grossly over applied and over stated.Newsom, Psy 521/621 Univariate Quantitative Methods28

Cronbach’s Alpha29The .70 criteria is commonly attributed to Nunnally (1978), a highly regardedpsychometrician, but using.70 as a standard was clearly not his intention:what a satisfactory level of reliability is depends on how a measure is being used. In the earlystages of research . . . one saves time and energy by working with instruments that have onlymodest reliability, for which purpose reliabilities of .70 or higher will suffice. . . . In contrast to thestandards in basic research, in many applied settings a reliability of .80 is not nearly high enough.In basic research, the concern is with the size of correlations and with the differences in means fordifferent experimental treatments, for which purposes a reliability of .80 for the differentmeasures is adequate. In many applied problems, a great deal hinges on the exact score made bya person on a test. . . . In such instances it is frightening to think that any measurement error ispermitted. Even with a reliability of .90, the standard error of measurement is almost one-third aslarge as the standard deviation of the test scores. In those applied settings where importantdecisions are made with respect to specific test scores, a reliability of .90 is the minimum thatshould be tolerated, and a reliability of .95 should be considered the desirable standard. (pp. 245246)Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill. Quote also given by: Lance, C. E., Butts, M. M., &Michels, L. C. (2006). The sources of four commonly reported cutoff criteria what did they really say?. Organizational researchmethods, 9(2), 202-220.Newsom, Psy 521/621 Univariate Quantitative Methods

Kuder-Richardson 20 (KR20)The KR20 (Kuder & Richardson, 1937) is a special case ofCronbach’s alpha when the items are binary (e.g., yes/no orcorrect/incorrect)It is equivalent to the raw score form of Cronbach’s alpha, socomputation of α for a set of binary items will give the sameresult as the KR20Newsom, Psy 521/621 Univariate Quantitative Methods30