Cognitive Interview in the Search for Validity Evidence Based on Response Processes

Authors

DOI:

https://doi.org/10.1590/1982-4327e3336

Keywords:

Item content (test), Test construction, Test validity, Psychological testing

Abstract

Cognitive interviews can provide validity evidence for instruments based on item response processes; however, use of focus groups still prevails in Brazilian literature. Moreover, semantic analysis has only been considered when searching for validity evidence based on test content. This paper presents a proposal for qualitative data analysis based on cognitive interviewing, thus providing researchers with a protocol that enables best practices in carrying out this technique, and consolidating it in the Brazilian literature as an option to search for validity evidence based on item response processes. To conclude, we present some criticisms regarding current procedures for validity evidence based on test content and discuss some possibilities.

Downloads

Download data is not yet available.

Author Biographies

  • Ana Gabriela Rocha Araújo, Universidade Federal do Rio Grande do Sul

    Ana Gabriela Rocha Araújo is a Ph.D. candidate of the Instituto de Psicologia at the Universidade Federal do Rio Grande do Sul, Porto Alegre-RS, Brazil.

  • Cláudio Simon Hutz, Universidade Federal do Rio Grande do Sul

    Cláudio Simon Hutz is a Professor of the Instituto de Psicologia at the Universidade Federal do Rio Grande do Sul, Porto Alegre-RS, Brazil.

  • Cleonice Alves Bosa, Universidade Federal do Rio Grande do Sul

    Cleonice Alves Bosa is a Professor of the Instituto de Psicologia at the Universidade Federal do Rio Grande do Sul, Porto Alegre-RS, Brazil.

References

Almanasreh, E., Moles, R., & Chen, T. F. (2019). Evaluation of methods used for estimating content validity. Research in Social and Administrative Pharmacy, 15(2), 214-221. https://doi.org/10.1016/j.sapharm.2018.03.066

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (Eds.). (2014). The standards for educational and psychological testing. AERA.

American Psychological Association. (2020). Publication manual of the American Psychological Association (7th ed.). APA.

Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149

Borsa, J. C., & Seize, M. M. (2017). Construção e adaptação de instrumentos psicológicos: Dois caminhos possíveis [Construction and adaptation of psychological instruments: Two possible paths]. In B. Damasio & J. C. Borsa (Eds.), Manual de desenvolvimento de instrumentos psicológicos [Psychological instrument development manual] (pp. 15-38). Vetor.

Clark, L. A., & Watson, D. (2019). Constructing validity: New developments in creating objective measuring instruments. Psychological Assessment, 31(12), 1412-1427. https://doi.org/10.1037/pas000626

Ellis, A. (1946). The validity of personality questionnaires. Psychological Bulletin, 43(5), 385-440. https://doi.org/10.1037/h0055483

Fryer, L. K., & Nakao, K. (2020). The future of survey self-report: An experiment contrasting likert, VAS, slide, and swipe touch interfaces. Frontline Learning Research, 8(3), 10-25. https://doi.org/10.14786/flr.v8i3.501

Hubley, A. M. (2021). Response processes validity evidence : Understanding the meaning of scores from psychological measures. In P. Graf & D. J. A. Dozois (Eds.), Handbook on the state of the art in applied psychology (pp. 413-434). Wiley Blackwell.

Meadows, K. (2021). Cognitive interviewing methodologies. Clinical Nursing Research, 30(4), 375-379. https://doi.org/10.1177/10547738211014099

Miller, K., Willson, S., Chepp, V., & Ryan, J. M. (2014). Analysis. In K. Miller, S. Willson, V. Chepp, & J. L. Padilla (Eds.), Cognitive interviewing methodology (pp. 35-50). Wiley.

Pacico, J. C. (2015). Como é feito um teste? Produção de itens [How is a test built? Item production]. In C. S. Hutz, D. R. Bandeira, & C. M. Trentini (Orgs.), Psicometria [Psychometrics] (pp. 55-70). Artmed.

Padilla, J.-L., & Benítez, I. (2014). Validity evidence based on response processes. Psicothema, 26(1), 136-144. https://doi.org/10.7334/psicothema2013.259

Pasquali, L. (1998). Princípios de elaboração de escalas psicológicas [Principles of elaboration of psychological scales]. Revista de Psiquiatria Clínica, 25(5), 206-213. http://ppget.ifam.edu.br/wp-content/uploads/2017/12/Principios-de-elaboracao-de-escalas-psicologicas.pdf

Peterson, C. H., Peterson, N. A., & Powell, K. G. (2017). Cognitive interviewing for item development: Validity evidence based on content and response processes. Measurement and Evaluation in Counseling and Development, 50(4), 217-223. https://doi.org/10.1080/07481756.2017.1339564

Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93-105. https://doi.org/10.1037//0003-066x.54.2.93

Tourangeau, R. (1984). Cognitive sciences and survey methods. In T. B. Jabine, M. L. Straf, J. M. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73-100). The National Academies.

Tourangeau, R., & Hanover, L. (2018). The survey response process from a cognitive viewpoint. Quality Assurance in Education, 26(2), 169-181. https://doi.org/10.1108/qae-06-2017-0034

Tourangeau, R., Maitland, A., Steiger, D., & Yan, T. (2019). A framework for making decisions about question evaluation methods. In P. Beatty, D. Collins, L. Kaye, J. L. Padilla, G. Willis, & A. Wilmot (Eds.), Advances in questionnaire design, development, evaluation and testing (pp. 47-73). Wiley.

Urbina, S. (2014). Essentials of psychological testing (2nd ed.). Wiley.

Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. SAGE.

Willis, G. B. (2018). Cognitive interviewing in survey design: State of the science and future directions. In D. Vannette & J. Krosnick (Eds.), The Palgrave handbook of survey research (pp. 103-107). Palgrave Macmillan.

Willson, S., & Miller, K. (2014). Data collection. In K. Miller, V. Chepp, S. Willson, & J. L. Padilla (Eds.), Cognitive interviewing methodology (pp. 15-33). Wiley.

Wolcott, M. D., & Lobczowski, N. G. (2021). Using cognitive interviews and think-aloud protocols to understand thought processes. Currents in Pharmacy Teaching & Learning, 13(2), 181-188. https://doi.org/10.1016/J.cptl.2020.09.005

Downloads

Published

2024-05-04

Issue

Vol. 33 (2023)

Section

Thematic Dossier “Methodological Challenges in Psychology: Contributions to Academic Practice and Training”

License

Copyright (c) 2024 Paidéia (Ribeirão Preto)

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Regarding the availability of contents, Paideia adopts the Creative Commons License, CC-BY. With this licence anyone is allowed to copy and redistribute the material in any medium or format, as well as to remix, transform, and create from the material for any purpose, even commercial, giving the proper copyright credits to the journal, providing a link to the licence and indicating if changes have been made.

Partial reproduction of other publications

Quotations of more than 500 words, reproductions of one or more figures, tables or other illustrations must have written permission from the copyright holder of the original work for the reproduction specified in the Paidéia journal. Permission should be addressed to the author of the submitted manuscript. Secondarily obtained rights will not be transferred under any circumstance. 

How to Cite

Cruz, A. C. A. da, Araújo, A. G. R., Hutz, C. S., & Bosa, C. A. (2024). Cognitive Interview in the Search for Validity Evidence Based on Response Processes. Paidéia (Ribeirão Preto), 33, e3336. https://doi.org/10.1590/1982-4327e3336