Estimating reliability of schoollevel scores usin

Authors: MinJeong Jeon Guemin Lee JeongWon Hwang SangJin Kang

Publish Date: 2009/05/06

Volume: 10, Issue: 2, Pages: 149-158

PDF Link

Abstract

The purpose of this study was to investigate the methods of estimating the reliability of schoollevel scores using generalizability theory and multilevel models Two approaches ‘student within schools’ and ‘students within schools and subject areas’ were conceptualized and implemented in this study Four methods resulting from the combination of these two approaches with generalizability theory and multilevel models were compared for both balanced and unbalanced data The generalizability theory and multilevel models for the ‘students within schools’ approach produced the same variance components and reliability estimates for the balanced data while failing to do so for the unbalanced data The different results from the two models can be explained by the fact that they administer different procedures in estimating the variance components used in turn to estimate reliability Among the estimation methods investigated in this study the generalizability theory model with the ‘students nested within schools crossed with subject areas’ design produced the lowest reliability estimates Fully nested designs such as studentsschools or subject areasstudentsschools would not have any significant impact on reliability estimates of schoollevel scores Both methods provide very similar reliability estimates of schoollevel scoresSchool performance assessment programs have been implemented for the purpose of evaluating and monitoring the quality of school systems in many countries Various achievement tests have been commonly used as a primary indicator in assessing school performance In general achievement test scores of students are aggregated into schoollevel scores such as school mean scores or PAACs percentages of students at or above cutscores Aggregated schoollevel scores obtained from student scores have been examined in many previous studies of various fields to investigate topics related to school quality and educational policies Hill and Hurley 1984 Ingelhart 1977 1985a 1985b Rohrschneider 1988 Dalton 1984 Sabatier et al 1987 Wright et al 1985 Brennan 2001a b Kane a Staiger 2002Before schoollevel scores are used it is necessary to examine their fitness from the perspectives of reliability and validity This confirmation is critical to making accurate substantial inferences based on those scores Dunbar et al 1991 Gao et al 1994 Linn et al 1991 It is required for researchers who use schoollevel scores to gather and provide information regarding the quality of those measuresUnfortunately many previous studies have reported individuallevel reliability estimates such as Cronbach’s alpha even though aggregated schoollevel scores were used Jones and Norrander 1996 These studies failed to recognize the fact that the reliability estimates for individuallevel scores differed from those for schoollevel scores This might lead to the misinterpretation or misuse of scores resulting from the application of inappropriate levels of score consistency In addition many researchers believe that schoollevel scores are more reliable than individuallevel scores However this kind of conventional opinion on the reliability of schoollevel scores does not necessarily hold true For example if the number of persons within groups goes to infinite it is reasonable to assume that error variance for persons is likely to be larger than error variance for groups However for small number of persons within groups this is not necessarily true Brennan 1995 2001a b More precise investigation of the reliability of schoollevel scores should be conducted before those scores are used as primary measuresIn this study several methods of estimating the reliability of schoollevel scores were conceptualized using multilevel and generalizability theory models Generalizability theory has been commonly used for this purpose Brennan 1995 Gao et al 1994 Jones and Norrander 1996 O’Brien 1991 as it enables investigators to explore reliabilities for various circumstances by fixing or randomizing measurement conditions Brennan 2001a b Though multilevel models have not been frequently used for this purpose they do offer many advantages in examining the relationships among individuallevel and schoollevel measures Raudenbush and Bryk 1986 Teddlie and Reynolds 2000 Snijders and Bosker 1999 provided several multilevel model procedures for estimating the reliability of schoollevel scores utilizing data involving individuals nested within schools However there are relatively few multilevel model studies that address this issue

Keywords: