Ad hoc procedure for optimising agreement between observational records

  1. Javier Arana 1
  2. Daniel Lapresa 2
  3. M. Teresa Anguera 3
  4. Belén Garzón
  1. 1 Universidad Internacional de La Rioja
    info

    Universidad Internacional de La Rioja

    Logroño, España

    ROR https://ror.org/029gnnp81

  2. 2 Universidad de La Rioja
    info

    Universidad de La Rioja

    Logroño, España

    ROR https://ror.org/0553yr311

  3. 3 Universitat de Barcelona
    info

    Universitat de Barcelona

    Barcelona, España

    ROR https://ror.org/021018s57

Show affiliations +
Journal:
Anales de psicología

ISSN: 0212-9728,  1695-2294

Year of publication: 2016

Volume: 32

Issue: 2

Pages: 589-595

Type: Article

DOI: 10.6018/ANALESPS.32.2.213551 DIALNET GOOGLE SCHOLAR

More publications in: Anales de psicología

Abstract

Observational studies in the field of sport are complicated by the added difficulty of having to analyse multiple, complex events or behaviours that may last just a fraction of a second. In this study, we analyse three aspects related to the reliability of data collected in such a study. The first aim was to analyse and compare the reliability of data sets assessed quantitatively (calculation of kappa statistic) and qualitatively (consensus agreement method). The second aim was to describe how, by ensuring the alignment of events, we calculated the kappa statistic for the order parameter using SDIS-GSEQ software (version 5.1) for data sets containing different numbers of sequences. The third objective was to describe a new consultative procedure designed to remove the confusion generated by discordant data sets and improve the reliability of the data. The procedure is called “consultative” because it involves the participation of a new observer who is responsible for consulting the existing observations and deciding on the definitive result

Bibliographic References

  • Altman, D.G. (1991). Practical statistics for medical research. London: Chapman and Hall.
  • Anguera, M.T. (1979). Observational Typology. Quality & Quantity. European-American Journal of Methodology, 13(6), 449-484.
  • Anguera, M.T. (1988). Observación en la escuela. Barcelona: Grao.
  • Anguera, M.T. (1989). Innovaciones en la metodología de evalua-ción de programas. Anales de Psicología, 5, 13-42.
  • Anguera, M.T. (1990). Metodología observacional. En J. Arnau, M.T. Anguera y J. Gómez (Eds.), Metodología de la investigación en Ciencias del Comportamiento (pp. 125-238). Murcia: Universidad de Murcia.
  • Anguera, M.T. (2003). La observación. En C. Moreno (Ed.), Evaluación psicológica. Concepto, proceso y aplicación en las áreas del desarrollo y de la inteligencia (pp. 271-308). Madrid: Sanz y Torres.
  • Anguera, M.T., Blanco-Villaseñor, A., Hernández-Mendo, A. & Losada, J.L. (2011). Diseños observacionales: Ajuste y aplicación en psicología del deporte. Cuadernos de Psicología del Deporte, 11(2), 63-76.
  • Anguera, M.T., Blanco-Villaseñor, A. & Losada, J.L. (2001). Dise-ños observacionales, cuestión clave en el proceso de la metodo-logía observacional. Metodología de las Ciencias del Comportamiento, 3(2), 135-160.
  • Anguera, M.T., Blanco-Villaseñor, A., Losada, J. & Hernández-Mendo, A. (2000). La metodología observacional en el deporte: Conceptos básicos. Lecturas: Educación Física y Deportes, 5(24). Retrieved from http://www.efdeportes.com/efd24b/obs.htm
  • Anguera, M.T. & Hernández-Mendo, A. (2013). Observational methodology in sport sciences. E-balonmano.com: Journal of Sport Science, 9(3), 135-160.
  • Anguera, M.T., Magnusson, M.S. & Jonsson, G.K. (2007). Instrumentos no estándar. Avances en medición, 5(1), 63-82.
  • Bakeman, R. (1978). Untangling streams of behavior: Sequential analysis of observation data. In G.P. Sackett (Ed.), Observing Behavior, Vol. 2: Data collection and analysis methods (pp. 63-78). Baltimore: University of Park Press.
  • Bakeman, R., McArthur, D. & Quera, V. (1996). Detecting group differences in sequential association using sampled permuta-tions: Log odds, kappa, and phi compared. Behavior Research Methods, Instruments, and Computers, 28(3), 446-457. Doi: 10.3758/BF03200524
  • Bakeman, R. & Quera, V. (1992) SDIS: A sequential data inter-change standard. Behavior Research Methods, Instruments & Computers, 24(4), 554-559. Doi: 10.3758/BF03203604
  • Bakeman, R. & Quera, V. (1995). Analyzing interaction: Sequential analysis with SDIS and GSEQ. Cambridge: Cambridge University Press.
  • Bakeman, R. & Quera, V. (2001). Using GSEQ with SPSS. Metodo-logía de las Ciencias del Comportamiento, 3(2), 195-214.
  • Bakeman, R. & Quera, V. (2007). Reference manual GSEQ: 4.1.3. Retrieved from http://www.ub.es/comporta/sg.htm
  • Bakeman, R. & Quera, V. (2011). Sequential analysis and observational methods for the behavioral sciences. Cambridge: Cambridge University Press.
  • Benavente, A., Ato, M. & López, J.J. (2006). Procedimientos para detectar y medir el sesgo entre observadores. Anales de Psicología, 22(1), 161-167.
  • Blanco-Villaseñor, A. (1993). Fiabilidad, precisión, validez y generalizabilidad de los diseños observacionales. En M.T. Anguera (Ed.), Metodología observacional en la investigación psicológica, Vol. 2: Fundamentación (pp. 151-261). Barcelona: PPU.
  • Bland, J.M. & Altman, D.G. (1986). Statistical methods for as-sessing agreement between two methods of clinical measure-ment. The Lancet, i, 307-310.
  • Bland, J.M. & Altman, D.G. (1990). A note on the use of the intraclass correlation coefficient in the evaluation of agreement between two methods of measurement. Computers in Biology and Medicine, 20, 337-340.
  • Castellano, J., Hernández-Mendo, A., Morales, V. & Anguera, M.T. (2007). Optimizing a probabilistic model of the development of play in soccer. Quality & Quantity. International Journal of Methodology, 41(1), 93-104. Doi: 10.1007/s11135-005-3148-0
  • Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
  • Garganta, J. (2007). Modelaçao táctica em jugos desportivos: A desejável cumplicidade entre pesquisa, treino e competiçao. Re-vista Portuguesa de Ciências do Desporto, 7(1), 13.
  • Hernández-Mendo, A. & Anguera, M.T. (2002). Behavioral struc-ture in sociomotor sports: Roller-Hockey. Quality & Quantity. International Journal of Methodology, 36(4), 347-378. Doi: 10.1023/A:1020905411739
  • Herrero, M.L. (1992). Posibilidades de la metodología observacio-nal en el estudio analítico de conductas en el aula: Aplicación en escolares con problemas de comportamiento. Anales de Psicología, 8(1-2), 149-155.
  • Lapresa, D., Anguera, M.T., Alsasua, R., Arana, J. & Garzón, B. (2013). Comparative analysis of T-patterns using real time data and simulated data by assignment of conventional durations: The construction of efficacy in children's basketball. Internation-al Journal of Performance Analysis in Sport, 13(2), 321-339.
  • Lapresa, D., Arana, J., Anguera, M.T. & Garzón, B. (2013). Com-parative analysis of the sequentiality using SDIS-GSEQ and THEME: A concrete example in soccer. Journal of Sport Sciences, 31(15), 1687-1695. Doi: 10.1080/02640414.2013.796061
  • Lapresa, D., Ibañez, R., Arana, J., Garzón, B. & Amatria, M. (2011). Spatial and temporal analysis of karate kumite moves: comparative study of the senior and 12-13 year old groups. International Journal of Performance Analysis in Sport, 11(1), 57-70.
  • Nevill, A.M. & Atkinson, G. (1997). Assessing agreement between measurements recorded on a ratio scale in sports medicine and sports science. British Journal of Sports Medicine, 31(4), 314-318. Doi: 10.1136/bjsm.31.4.314
  • O´Donoghue, P. (2007). Reliability issues in performance analysis. International Journal of Performance Analysis in Sport, 7(1), 35-48.
  • Quera, V., Bakeman, R. & Gnisci, A. (2007). Observer agreement for event sequences: Methods and software for sequence alignment and reliability estimates. Behavior Research Methods, 39(1), 39-49. Doi: 10.3758/BF03192842
  • Robinson, G. & O´Donoghue, P. (2007). A weighted kappa statis-tic for reliability testing in performance analysis of sport. International Journal of Performance Analysis in Sport, 7(1), 12-19.
Data source: Dialnet