Quotation Rusch, Thomas, Lowry, Paul B., Mair, Patrick, Treiblmaier, Horst. 2016. Breaking Free from the Limitations of Classical Test Theory: Developing and Measuring Information Systems Scales Using Item Response Theory. Information & Management, 54 (2), 189-203.




Information systems (IS) research frequently uses survey data to measure the interplay between technological systems and human beings. Researchers have developed sophisticated procedures to build and validate multi-item scales that measure latent constructs. The vast majority of IS studies uses classical test theory (CTT), but this approach suffers from three major theoretical shortcomings: (1) it assumes a linear relationship between the latent variable and observed scores, which rarely represents the empirical reality of behavioral constructs; (2) the true score can either not be estimated directly or only by making assumptions that are difficult to be met; and (3) parameters such as reliability, discrimination, location, or factor loadings depend on the sample being used. To address these issues, we present item response theory (IRT) as a collection of viable alternatives for measuring continuous latent variables by means of categorical indicators (i.e., measurement variables). IRT offers several advantages: (1) it assumes nonlinear relationships; (2) it allows more appropriate estimation of the true score; (3) it can estimate item parameters independently of the sample being used; (4) it allows the researcher to select items that are in accordance with a desired model; and (5) it applies and generalizes concepts such as reliability and internal consistency, and thus allows researchers to derive more information about the measurement process. We use a CTT approach as well as Rasch models (a special class of IRT models) to demonstrate how a scale for measuring hedonic aspects of websites is developed under both approaches. The results illustrate how IRT can be successfully applied in IS research and provide better scale results than CTT. We conclude by explaining the most appropriate circumstances for applying IRT, as well as the limitations of IRT.


Press 'enter' for creating the tag

Publication's profile

Status of publication Published
Affiliation WU
Type of publication Journal article
Journal Information & Management
Citation Index SSCI
WU Journalrating 2009 A
WU-Journal-Rating new INF-A, STRAT-B, WH-B
Language English
Title Breaking Free from the Limitations of Classical Test Theory: Developing and Measuring Information Systems Scales Using Item Response Theory
Volume 54
Number 2
Year 2016
Page from 189
Page to 203
Reviewed? Y
URL http://www.sciencedirect.com/science/article/pii/S0378720616300611
DOI http://dx.doi.org/10.1016/j.im.2016.06.005
Open Access Y


Rusch, Thomas (Details)
Lowry, Paul B. (City University Hong Kong, China)
Mair, Patrick (Department of Psychology, Harvard University, United States/USA)
Treiblmaier, Horst (FH Steyr, Austria)
Competence Center for Empirical Research Methods WE (Details)
Research areas (Ă–STAT Classification 'Statistik Austria')
1162 Statistics (Details)
5509 Psychological methodology (Details)
5701 Applied statistics (Details)
5704 Social statistics (Details)
5912 Social sciences (interdisciplinary) (Details)
Google Scholar: Search