Estimating the reliability coefficient of tests in presence of missing values

  1. Cuesta Izquierdo, Marcelino 1
  2. Fonseca Pedrero, Eduardo 2
  1. 1 Universidad de Oviedo
    info

    Universidad de Oviedo

    Oviedo, España

    ROR https://ror.org/006gksa02

  2. 2 Universidad de La Rioja
    info

    Universidad de La Rioja

    Logroño, España

    ROR https://ror.org/0553yr311

Journal:
Psicothema

ISSN: 0214-9915

Year of publication: 2014

Volume: 26

Issue: 4

Pages: 516-523

Type: Article

DOI: 10.7334/PSICOTHEMA2014.98 DIALNET GOOGLE SCHOLAR lock_openOpen access editor

More publications in: Psicothema

Institutional repository: lock_openOpen access Editor

Abstract

Background: The problem of missing values at the item level is common in studies using educational and psychological tests. The aim of the present work is to explore how the estimation of reliability is affected by missing values. Method: Using real data, we simulated missing values in accordance with a "missing at random mechanism". Four factors were manipulated with the aim of checking their effect on the estimation of the reliability of the instrument: missing data mechanism, percentage of missing data in the database, sample size, and procedure employed for the treatment of missing values. Results: The results show that the quality of estimations depends on the interaction of various factors. The general tendency is that the estimations are worse when the sample size is small and the percentage of missing values increases. Listwise is the worst procedure for treatment of the missing data in the simulated conditions. Conclusions: It is concluded that with a small percentage of missing values one can obtain estimations that are acceptable from a practical point of view with all the procedures employed, except Listwise.

Bibliographic References

  • Allison, P.D. (2002). Missing data. Thousand Oaks, CA: Sage Publications.
  • Baraldi, A.N., & Enders, C.K. (2010). An introduction to modern missing data analyses. Journal of School Psychology, 48, 5-37.
  • Bernaards, C.A., & Sijtsma, K. (2005). Bias in factor loadings from questionnaire data with imputed item scores. Journal of Statistical Computation and Simulation, 75(1), 13-23.
  • Botella, J. (2002). Potencia de pruebas alternativas para dos muestras relacionadas con datos perdidos [Power of alternative tests for two paired samples with missing data]. Psicothema, 14(1), 174-180.
  • Carpita, M., & Manisera, M. (2011). On the imputation of missing data in surveys with Likert-type scales. Journal of Classification, 28(1), 93-112.
  • Cuesta, M., Fonseca-Pedrero, E., Vallejo, G., & Muñiz, J. (2013). Datos perdidos y propiedades psicométricas en los tests de personalidad [Missing data and psychometric properties of personality tests]. Anales de Psicología, 29(1), 285-292.
  • Enders, C.K. (2003). Using the expectation maximization algorithm to estimate coefficient alpha for scales with item-level missing data. Psychological Methods, 8(3), 322-337.
  • Enders, C.K. (2004). The impact of missing data on sample reliability estimates: Implications for reliability reporting practices. Educational and Psychological Measurement, 64(3), 419-436.
  • Enders, C.K. (2010). Applied missing data analysis. New York: The Guilford Press.
  • Fernández-Alonso, R., Suárez-Alvarez, J., & Muñiz, J. (2012). Imputación de datos perdidos en las evaluaciones diagnósticas educativas [Imputation methods for missing data in educational diagnostic evaluation]. Psicothema, 24(1), 167-175.
  • Fonseca-Pedrero, E., Muñiz, J., Lemos, S., Paíno, M., & Villazón, U. (2010). Esquizo-Q. Cuestionario Oviedo para la evaluación de la esquizotipia [ESQUIZO-Q. Oviedo Schizotypy Assessment Questionnaire]. Madrid: TEA Ediciones.
  • Fonseca-Pedrero, E., Lemos-Giráldez, S., Paíno, M., Sierra-Baigrie, S., Santarén-Rosell, M., & Muñiz, J. (2011). Internal structure and reliability of the Oviedo Schizotypy Assessment Questionnaire (ESQUIZO-Q). International Journal of Clinical and Health Psychology, 11, 385-402.
  • Fonseca-Pedrero, E., Paíno, M., Lemos-Girádez, S., Sierra-Baigrie, S., Ordóñez, N., & Muñiz, J. (2011). Early psychopathological features in Spanish adolescents. Psicothema, 23, 87-93.
  • Gmel, G. (2001). Imputation of missing values in the case of a multiple item instrument measuring alcohol consumption. Statistics in Medicine, 20, 2369-2381.
  • Graham, J.W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549-576.
  • Gottschall, A.C., West, S.G., & Enders, C.K. (2012). A comparison of item level and scale-level multiple imputation for questionnaire batteries. Multivariate Behavioral Research, 47, 1-25.
  • Howell, D.G. (2008). The analysis of missing data. In Outhwaite, W., & Turner, S. (Eds), Handbook of Social Science Methodology. London: Sage.
  • Little, R.J.A., & Rubin, D.B. (1987). Statistical analysis with missing data. New York: Wiley.
  • López-Pina, J.A., Sánchez-Meca, J., & López-Lopez, J.A. (2012). Métodos para promediar coeficientes alfa en los estudios de generalización de la fiabilidad [Methods for averaging alpha coefficients in reliability generalization studies]. Psicothema, 24(1), 161-166.
  • Mcdonald, R.A., Thurston, P.W., & Nelson, M.R. (2000). A Monte Carlo study of missing item methods. Organizational Research Methods, 3(1), 71-92.
  • Oren, C., Kennet-Cohen, T., Turvall, E., & Allalouf, A. (2014). Demonstrating the validity of three general scores of PET in predicting higher education achievement in Israel. Psicothema, 26(1), 117-126.
  • Raaijmakers, Q.A.W. (1999). Effectiveness of different missing data treatments in surveys with Likert-Type data: Introducing the relative mean substitution approach. Educational and Psychological Measurement, 59(5), 725-748.
  • Ríos, J., & Wells, C. (2014). Validity evidence based on internal structure. Psicothema, 26(1), 108-116.
  • Roth, P.L. (1994). Missing data: A conceptual review for applied psychologists. Personnel Psychology, 47, 537-560.
  • Roth, P.L., Switzer, F.S., & Switzer, D.M. (1999). Missing data in multiple item scales: A Monte Carlo analysis of missing data techniques. Organizational Research Methods, 2(3), 211-232.
  • Rubin, D.B. (1976). Inference and missing data. Biometrika, 63, 581-592.
  • Rubin, D.B. (1987). Multiple imputation for nonresponse in surveys. New York: Wiley.
  • Schafer, J.J., & Graham, J.W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147-177.
  • Shrive, F., Stuart, H., Quan, H., & Ghali, W.A. (2006). Dealing with missing data in a multi-question depression scale: A comparison of imputation methods. BMC Medical Research Methodology, 6:57 (http://www. biomedcentral.com/1471-2288-6-57) (download on 25/01/2011).
  • Sijtsma, K., & van der Ark, L.A. (2003). Investigation and treatment of missing item scores in test and questionnaire data. Multivariate Behavioral Research, 38(4), 505-528.
  • Vallejo, G., Fernández, P., Livacic-Rojas, P., & Tuero-Herrero, E. (2011). Comparison of modern methods for analyzing unbalanced repeated measures data. Multivariate Behavioral Research, 46(6), 900-937.
  • Van der Ark, L.A., & Vermunt, J.K. (2010). New developments in missing data analysis (editorial). Methodology, 6(1), 1-2.
  • Van Ginkel, J.R., Sijtsma, K., van der Ark, L.A., & Vermunt, J.K. (2010). Incidence of missing item scores in personality measurement, and simple item-score imputation. Methodology, 6(1), 17-30.
  • Van Ginkel, J.R., van der Ark, L.A., & Sijtsma, K. (2007a). Multiple imputation of item scores in test and questionnaire data, and influence on psychometric results. Multivariate Behavioral Research, 42(2), 387-414.
  • Van Ginkel, J.R., van der Ark, L.A., & Sijtsma, K. (2007b). Multiple imputation of item scores when test data are factorially complex. British Journal of Mathematical and Statistical Psychology, 60(2), 315-337.
  • Von Hippel, P.T. (2004). Biases in SPSS 12.0 missing value analysis. The American Statistician, 58(2), 160-164.
Data source: Dialnet