Cronbach’s alpha - what makes it really good? Some advice for planning and criticizing psychological questionnaires

Tomasz Rak

a:1:{s:5:"pl_PL";s:46:"Uniwersytet Papieski Jana Pawła II w Krakowie";}

Szymon Wrześniowski

Uniwersytet Papieski Jana Pawła II w Krakowie


Abstract

Whatever Cronbach’s alpha measures – it’s not internal consistency, commonly misunderstood in psychology as the average strength of relationships within questionnaire items. In this article, we explore the reasons why the understanding of alpha as internal consistency is particularly flawed, and focus on how alpha inflation works in a practical way. Using the simulation method, we determine the precise (common) influence of the number of respondents, the range of measurement (Likert) scales, the number of questions in the questionnaire and the average correlation of items on the alpha level. The results confirm alpha-level inflation due to a greater number of questions: alpha gets a satisfactory level even with minimal internal consistency if there are many questions in the questionnaire. We suggest that the reliability of weak psychological tools is overestimated because of presented rapid alpha inflation. Number of subjects and the range of the scale had no influence on alpha.


Keywords:

reliability, alpha coefficient, alpha inflation, internal consistency, simulation


Anselmi, P., Colledani, D., & Robusto, E. (2019). A comparison of classical and modern measures of internal consistency. Frontiers in Psychology, 10, Article 2714. https://doi.org/10.3389/fpsyg.2019.02714
Crossref   Google Scholar

Bajpai, S., & Bajpai, R. (2014). Goodness of measurement: Reliability and validity. International Journal of Medical Science and Public Health, 3(2), 112‒115. https://doi.org/10.5455/ijmsph.2013.191120133
Crossref   Google Scholar

Barbaranelli, C., Lee, C. S., Vellone, E., & Riegel, B. (2015). The problem with Cronbach's alpha: comment on Sijtsma and van der Ark (2015). Nursing Research, 64(2), 140‒145. https://doi.org/10.1097/NNR.0000000000000079
Crossref   Google Scholar

Bentler, P. M. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74(1), 137‒143. https://doi.org/10.1007/s11336-008-9100-1
Crossref   Google Scholar

Bland, J. M., & Altman, D. G. (1997). Statistics notes: Cronbach's alpha. British Medical Journal, 314(7080), Article 572. https://doi.org/10.1136/bmj.314.7080.572
Crossref   Google Scholar

Bonett, D. G., & Wright, T. A. (2015). Cronbach's alpha reliability: Interval estimation, hypothesis testing, and sample size planning. Journal of Organizational Behavior, 36(1), 3‒15. https://doi.org/10.1002/job.1960
Crossref   Google Scholar

Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71(3), 425‒440. https://doi.org/10.1007/s11336-006-1447-6
Crossref   Google Scholar

Borsboom, D., & Mellenbergh, G. J. (2002). True scores, latent variables, and constructs: A comment on Schmidt and Hunter. Intelligence, 30(6), 505‒514. https://doi.org/10.1016/S0160-2896(02)00082-X
Crossref   Google Scholar

Brannick, M. T. (1995). Critical comments on applying covariance structure modeling. Journal of Organizational Behavior, 16(3), 201‒213. https://doi.org/10.1002/job.4030160303
Crossref   Google Scholar

Breckler, S. J. (1990). Applications of covariance structure modeling in psychology: Cause for concern? Psychological Bulletin, 107(2), 260‒273. https://doi.org/10.1037/0033-2909.107.2.260
Crossref   Google Scholar

Bujang, M. A., Omar, E. D., & Baharum, N. A. (2018). A review on sample size determination for Cronbach’s alpha test: a simple guide for researchers. The Malaysian Journal Of Medical Sciences: MJMS, 25(6), 85‒99. https://doi.org/10.21315/mjms2018.25.6.9
Crossref   Google Scholar

Chan, E. K. (2014). Standards and guidelines for validation practices: Development and evaluation of measurement instruments. In Validity and validation in social, behavioral, and health sciences (pp. 9‒24). Springer.
Crossref   Google Scholar

Chang, L. (1994). A psychometric evaluation of 4-point and 6-point Likert-type scales in relation to reliability and validity. Applied Psychological Measurement, 18(3), 205‒215. https://doi.org/10.1177/014662169401800302
Crossref   Google Scholar

Charter, R. A. (1999). Sample size requirements for precise estimates of reliability, generalizability, and validity coefficients. Journal of Clinical and Experimental Neuropsychology, 21(4), 559‒566. https://doi.org/10.1007/978-3-319-07794-9_2
Crossref   Google Scholar

Cho, E. (2022). The accuracy of reliability coefficients: A reanalysis of existing simulations. Psychological Methods. Online first. https://doi.org/10.1037/met0000475
Crossref   Google Scholar

Cho, E., & Kim, S. (2015). Cronbach’s coefficient alpha: Well known but poorly understood. Organizational Research Methods, 18(2), 207‒230. https://doi.org/10.1177/1094428114555994
Crossref   Google Scholar

Comrey, A. L., & Lee, H. B. (1992). A first course in factor analysis (2nd ed.). Erlbaum.   Google Scholar

de Vet, H. C., Mokkink, L. B., Mosmuller, D. G., & Terwee, C. B. (2017). Spearman–Brown prophecy formula and Cronbach's alpha: different faces of reliability and opportunities for new applications. Journal of Clinical Epidemiology, 85, 45‒49. https://doi.org/10.1016/j.jclinepi.2017.01.013
Crossref   Google Scholar

DeVellis, R. F. (2006). Classical test theory. Medical Care, 44(11), S50‒S59. https://doi.org/10.1097/01.mlr.0000245426.10853.30
Crossref   Google Scholar

Dimov, I. T. (2008). Monte Carlo methods for applied scientists. World Scientific. https://doi.org/10.1142/9789812779892
Crossref   Google Scholar

Duhachek, A., Coughlan, A. T., & Iacobucci, D. (2005). Results on the standard error of the coefficient alpha index of reliability. Marketing Science, 24(2), 294‒301. https://doi.org/10.1287/mksc.1040.0097
Crossref   Google Scholar

Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105(3), 399‒412. https://doi.org/10.1111/bjop.12046
Crossref   Google Scholar

Dunn, W. L., & Shultis, J. K. (2011). Exploring Monte Carlo methods. Elsevier. https://doi.org/10.1016/B978-0-444-51575-9.00007-5
Crossref   Google Scholar

Eisinga, R., Grotenhuis, M. T., & Pelzer, B. (2013). The reliability of a two-item scale: Pearson, Cronbach, or Spearman-Brown? International Journal of Public Health, 58(4), 637‒642. https://doi.org/10.1007/s00038-012-0416-3
Crossref   Google Scholar

Flake, J. K., Pek, J., & Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370‒378. https://doi.org/10.1177/1948550617693063
Crossref   Google Scholar

Flora, D. B. (2020). Your coefficient alpha is probably wrong, but which coefficient omega is right? A tutorial on using R to obtain better reliability estimates. Advances in Methods and Practices in Psychological Science, 3(4), 484‒501. https://doi.org/10.1177/2515245920951747
Crossref   Google Scholar

Gignac, G. E., Bates, T. C., & Jang, K. L. (2007). Implications relevant to CFA model misfit, reliability, and the five-factor model as measured by the NEO-FFI. Personality and Individual Differences, 43(5), 1051‒1062. https://doi.org/10.1016/j.paid.2007.02.024
Crossref   Google Scholar

Golafshani, N. (2003). Understanding reliability and validity in qualitative research. The Qualitative Report, 8(4), 597‒607.   Google Scholar

Green, S. B., & Yang, Y. (2009). Commentary on coefficient alpha: A cautionary tale. Psychometrika, 74(1), 121‒135. https://doi.org/10.1007/s11336-008-9098-4
Crossref   Google Scholar

Gruszczyńska, E. (2012). Kwestionariusz Samooceny Zysków i Strat – polska adaptacja Cor-Evaluation Se Hobfolla i jej podstawowe właściwości psychometryczne [Profit and Loss Self-Assessment Questionnaire – Polish adaptation of Hobfoll's Cor-Evaluation Se and its basic psychometric properties]. In E. Bielawska-Batorowicz & B. Dudek (Eds.), Teoria zachowania zasobow Stevana E. Hobfolla. Polskie doświadczenia [Stevan E. Hobfoll's theory of conservation of resources. Polish experience]. Wydawnictwo Uniwersytetu Łódzkiego.   Google Scholar

Guidroz, A. M., Yankelevich, M., Barger, P., Gillespie, M. A., & Zickar, M. J. (2009). Practical considerations for creating and using organizational survey norms: Lessons from two long-term projects. Consulting Psychology Journal: Practice and Research, 61(2), 85‒102. https://doi.org/10.1037/a0015969
Crossref   Google Scholar

Hayes, A. F., & Coutts, J. J. (2020). Use omega rather than Cronbach’s alpha for estimating reliability. But… Communication Methods and Measures, 14(1), 1‒24. https://doi.org/10.1080/19312458.2020.1718629
Crossref   Google Scholar

Henson, R. K. (2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34(3), 177‒189. https://doi.org/10.1080/07481756.2002.12069034
Crossref   Google Scholar

Hoyt, W. T., Warbasse, R. E., & Chu, E. Y. (2006). Construct validation in counseling psychology research. The Counseling Psychologist, 34(6), 769‒805. https://doi.org/10.1177/0011000006287389
Crossref   Google Scholar

Jolliffe, I. T., & Cadima, J. (2016). Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065), Article 20150202. https://doi.org/10.1098/rsta.2015.0202
Crossref   Google Scholar

Kalkbrenner, M. T. (2023). Alpha, Omega, and H internal consistency reliability estimates: Reviewing these options and when to use them. Counseling Outcome Research and Evaluation, 14(1), 77‒88. https://doi.org/10.1080/21501378.2021.1940118
Crossref   Google Scholar

Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1‒73. https://doi.org/10.1111/jedm.12000
Crossref   Google Scholar

Leung, S. O. (2011). A comparison of psychometric properties and normality in 4-, 5-, 6-, and 11-point Likert scales. Journal of Social Service Research, 37(4), 412‒421. https://doi.org/10.1080/01488376.2011.580697
Crossref   Google Scholar

Li, H., Rosenthal, R., & Rubin, D. B. (1996). Reliability of measurement in psychology: From Spearman-Brown to maximal reliability. Psychological Methods, 1(1), 98‒107. https://doi.org/10.1037/1082-989X.1.1.98
Crossref   Google Scholar

Lucke, J. F. (2005). The α and the ω of congeneric test theory: An extension of reliability and internal consistency to heterogeneous tests. Applied Psychological Measurement, 29(1), 65‒81. https://doi.org/10.1177/0146621604270882
Crossref   Google Scholar

Macey, W. H., & Eldridge, L. D. (2006). National norms versus consortium data: What do they tell us. In A. I. Kraut (Ed.), Getting action from organizational surveys: New concepts, technologies, and applications (pp. 352‒376). Jossey-Bass.   Google Scholar

Matell, M. S., & Jacoby, J. (1972). Is there an optimal number of alternatives for Likert-scale items? Effects of testing time and scale properties. Journal of Applied Psychology, 56(6), 506‒509. https://doi.org/10.1037/h0033601
Crossref   Google Scholar

McCrae, R. R., Kurtz, J. E., Yamagata, S., & Terracciano, A. (2011). Internal consistency, retest reliability, and their implications for personality scale validity. Personality and Social Psychology Review, 15(1), 28‒50. https://doi.org/10.1177/1088868310366253
Crossref   Google Scholar

McDonald, R. P. (2013). Test theory: A unified treatment. Psychology Press. https://doi.org/10.4324/9781410601087
Crossref   Google Scholar

McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412‒433. https://doi.org/10.1037/met0000144
Crossref   Google Scholar

Metsämuuronen, J. (2022). The effect of various simultaneous sources of mechanical error in the estimators of correlation causing deflation in reliability: Seeking the best options of correlation for deflation-corrected reliability. Behaviormetrika, 49(1), 91‒130. https://doi.org/10.1007/s41237-022-00158-y
Crossref   Google Scholar

Pastore, M., & Lombardi, L. (2014). The impact of faking on Cronbach’s alpha for dichotomous and ordered rating scores. Quality & Quantity, 48(3), 1191‒1211. https://doi.org/10.1007/s11135-013-9829-1
Crossref   Google Scholar

Ponterotto, J. G., & Ruckdeschel, D. E. (2007). An overview of coefficient alpha and a reliability matrix for estimating adequacy of internal consistency coefficients with psychological research measures. Perceptual and Motor Skills, 105(3), 997‒1014. https://doi.org/10.2466/pms.105.3.997-1014
Crossref   Google Scholar

Preston, C. C., & Colman, A. M. (2000). Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104(1), 1‒15. https://doi.org/10.1016/S0001-6918(99)00050-5
Crossref   Google Scholar

Raykov, T., & Marcoulides, G. A. (2011). Introduction to psychometric theory. Routledge. https://doi.org/10.4324/9780203841624
Crossref   Google Scholar

Revelle, W., & Condon, D. M. (2019). Reliability from α to ω: A tutorial. Psychological Assessment, 31(12), 1395‒1411. https://doi.org/10.1037/pas0000754
Crossref   Google Scholar

Šerbetar, I., & Sedlar, I. (2016). Assessing reliability of a multi-dimensional scale by coefficient alpha. Journal of Elementary Education, 9(1/2), 189‒196.   Google Scholar

Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107‒120. https://doi.org/10.1007/s11336-008-9101-0
Crossref   Google Scholar

Sijtsma, K. (2020). Measurement Models for Psychological Attributes: Classical Test Theory, Factor Analysis, Item Response Theory, and Latent Class Models. CRC Press. https://doi.org/10.1201/9780429112447-2
Crossref   Google Scholar

Sijtsma, K., & Pfadt, J. M. (2021). Part II: On the use, the misuse, and the very limited usefulness of Cronbach’s alpha: Discussing lower bounds and correlated errors. Psychometrika, 86(4), 843‒860. https://doi.org/10.1007/s11336-021-09789-8
Crossref   Google Scholar

Streiner, D. L. (2003). Starting at the beginning: an introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80(1), 99‒103. https://doi.org/10.1207/S15327752JPA8001_18
Crossref   Google Scholar

Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Allyn & Bacon.   Google Scholar

Taber, K. S. (2018). The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273‒1296. https://doi.org/10.1007/s11165-016-9602-2
Crossref   Google Scholar

Taherdoost, H. (2022). What is the best response scale for survey and questionnaire design; review of different lengths of rating scale / attitude scale / Likert scale. International Journal of Academic Research in Management, 8(1), 1‒10.   Google Scholar

Tang, W., Cui, Y., & Babenko, O. (2014). Internal consistency: Do we really know what it is and how to assess it. Journal of Psychology and Behavioral Science, 2(2), 205‒220.   Google Scholar

Ten Berge, J. M., & Sočan, G. (2004). The greatest lower bound to the reliability of a test and the hypothesis of unidimensionality. Psychometrika, 69(4), 613‒625. https://doi.org/10.1007/BF02289858
Crossref   Google Scholar

Thall, P. F., & Vail, S. C. (1990). Some covariance models for longitudinal count data with overdispersion. Biometrics, 46(3), 657‒671. https://doi.org/10.2307/2532086
Crossref   Google Scholar

Thigpen, N. N., Kappenman, E. S., & Keil, A. (2017). Assessing the internal consistency of the event‐related potential: An example analysis. Psychophysiology, 54(1), 123‒138. https://doi.org/10.1111/psyp.12629
Crossref   Google Scholar

Thompson, B. (2002). Score reliability: Contemporary thinking on reliability issues (1st ed.). Sage Publications, Inc. https://doi.org/10.4135/9781412985789.n1
Crossref   Google Scholar

Trizano-Hermosilla, I., & Alvarado, J. M. (2016). Best alternatives to Cronbach's alpha reliability in realistic conditions: congeneric and asymmetrical measurements. Frontiers in Psychology, 7, Article 769. https://doi.org/10.3389/fpsyg.2016.00769
Crossref   Google Scholar

Vaske, J. J., Beaman, J., & Sponarski, C. C. (2017). Rethinking internal consistency in Cronbach's alpha. Leisure Sciences, 39(2), 163‒173. https://doi.org/10.1080/01490400.2015.1127189
Crossref   Google Scholar

Vehkalahti, K., Puntanen, S., Tarkkonen, L. (2006). Estimation of reliability: a better alternative for Cronbach's alpha. Department of Mathematics and Statistics, University of Helsinki.   Google Scholar

Zumbo, B. D., & Chan, E. K. (2014). Validity and validation in social, behavioral, and health sciences. Social Indicators Research Series, Vol. 54. Springer. https://doi.org/10.1007/978-3-319-07794-9
Crossref   Google Scholar

Download


Published
2023-12-31

Cited by

Rak, T., & Wrześniowski, S. (2023). Cronbach’s alpha - what makes it really good? Some advice for planning and criticizing psychological questionnaires. The Review of Psychology, 66(4), 151–167. https://doi.org/10.31648/przegldpsychologiczny.9467

Tomasz Rak 
a:1:{s:5:"pl_PL";s:46:"Uniwersytet Papieski Jana Pawła II w Krakowie";}
Szymon Wrześniowski 
Uniwersytet Papieski Jana Pawła II w Krakowie