Methodological considerations for survey research: Validity, reliability, and quantitative analysis




Survey, validity, reliability, measurement invariance, Cognitive Translation Studies


As translation and interpreting studies continue to develop cognitive theories of translator and interpreter behavior and processing, there has been increased emphasis on research methods and data collection methodologies to glean new insights into the translation process. This article presents a critical review of survey research methods in Cognitive Translation Studies and argues for their inclusion as a means of better understanding translator and interpreter attitudes, behaviors, perceptions, and values. The article begins with a reflection on measurement and the need for alignment with theoretical frameworks and constructs; then it reviews important considerations when developing theoretically-grounded, empirically-based survey instruments, namely, validity, reliability, measurement invariance, and quantitative analysis. The article concludes with a call for additional methodological reflection on developing and using survey instruments.


Agbo, A. A. (2010). Cronbach’s alpha: Review of limitations and associated recommendations. Journal of Psychology in Africa, 20(2), 233–239.

Alves, F. (Ed.). (2003). Triangulating translation: Perspectives in process-oriented research. John Benjamins.

Alves, F. (2015). Translation process research at the interface: Paradigmatic, theoretical, and methodological issues in dialogue with cognitive science, expertise studies, and psycholinguistics. In A. Ferreira & J. W. Schwieter (Eds.), Psycholinguistic and cognitive inquiries into translation and interpreting (pp. 17–40). John Benjamins.

Alves, F., & Hurtado Albir, A. (2017). Evolution, challenges, and perspectives for research on cognitive aspects of translation. In J. W. Schwieter & A. Ferreira (Eds.), The handbook of translation and cognition (pp. 537–554). Wiley.

Alvstad, C., Hild, A., & Tiselius, E. (2011). Methods and strategies of process research: Integrative approaches in translation studies. In C. Alvstad, A. Hild, & E. Tiselius (Eds.), Methods and strategies of process research (pp. 1–9). John Benjamins.

American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1999). Standards for educational and psychological testing. American Educational Research Association.

Angelelli, C. V. (2004). Revisiting the interpreter’s role: A study of conference, court, and medical interpreters in Canada, Mexico, and the United States. John Benjamins.

Angelelli, C. V., & Baer, B. J. (Eds.). (2016). Researching translation and interpreting. Routledge.

Balling, L. W. (2008). A brief introduction to regression designs and mixed-effects modelling by a recent convert. Copenhagen Studies in Language, 36, 175–192.

Bandalos, D. L. (2018). Measurement theory and applications for the social sciences. Guilford Press.

Beaujean, A. A. (2014). Latent variable modeling using R: A step-by-step guide. Routledge.

Behr, D., & Shishido, K. (2016). The translation of measurement instruments for cross-cultural surveys. In C. Wolf, D. Joye, T. W. Smith, & Y.-C. Fu (Eds.), The SAGE handbook of survey methodology (pp. 269–287). SAGE.

Bernardini, S. (2001). Think-aloud protocols in translation research: Achievements, limits, future prospects. Target, 13(2), 241–263.

Bolaños-Medina, A. (2014). Self-efficacy in translation. Translation and Interpreting Studies, 9(2), 197–218.

Bollen, K. A. (2002). Latent variables in psychology and the social sciences. Annual Review of Psychology, 53(1), 605–634.

Bontempo, K., & Napier, J. (2011). Evaluating emotional stability as a predictor of interpreter competence and aptitude for interpreting. Interpreting, 13(1), 85–105.

Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press.

Boyle, G. J., Saklofske, D. H., & Matthews, G. (Eds.). (2015). Measures of personality and social psychological constructs. Academic Press.

Brown, J. D. (1996). Testing in language programs. Prentice Hall.

Buchanan, E. A., & Hvizdak, E. E. (2009). Online survey tools: Ethical and methodological concerns of human research ethics committees. Journal of Empirical Research on Human Research Ethics, 4(2), 37–48.

Carifio, J., & Perla, R. J. (2007). Ten common misunderstandings, misconceptions, persistent myths and urban legends about Likert scales and Likert response formats and their antidotes. Journal of Social Sciences, 3(3), 106–116. 10.3844/jssp.2007.106.116

Carifio, J., & Perla, R. J. (2008). Resolving the 50-year debate around using and misusing Likert scales. Medical Education, 42(12), 1150–1152.

Chan, E. K. H. (2014). Standards and guidelines for validation practices: Development and evaluation of measurement instruments. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 9–24). Springer.

Chesterman, A. (2009). The name and nature of translator studies. Hermes, 22(42), 13–22.

Choi, B. C. K., & Pak, A. W. P. (2005). A catalog of biases in questionnaires. Preventing Chronic Disease, 2(1), A13.

Clark, A. (1996). Being there: Putting brain, body, and world together again. MIT Press.

Cole, D. A., & Preacher, K. J. (2014). Manifest variable path analysis: Potentially serious and misleading consequences due to uncorrected measurement error. Psychological Methods, 19(2), 300–315.

Cortina, J. M. (1993). What is coefficient alpha?: An examination of theory and applications. Journal of Applied Psychology, 78(1), 98–104.

Coulacoglou, C., & Saklofske, D. H. (2017). Psychometrics and psychological assessment: Principles and applications. Academic Press.

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302.

Csizér, K., & Dörnyei, Z. (2005). The internal structure of language learning motivation and its relationship with language choice and learning effort. Modern Language Journal, 89(1), 19–36.

Dancette, J. (1997). Mapping meaning and comprehension in translation: Theoretical and experimental issues. In J. H. Danks, G. M. Shreve, S. B. Fountain, & M. K. McBeath (Eds.), Cognitive processes in translation and interpreting (pp. 77–103). SAGE.

DeCastellarnau, A. (2018). A classification of response scale characteristics that affect data quality: A literature review. Quality & Quantity, 52(4), 1523–1559.

DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). SAGE.

Diamantopoulos, A. (Ed.). (2008). Formative indicators [Special issue]. Journal of Business Research, 61(12).

Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105(3), 399–412.

Edwards, J. R., & Bagozzi, R. P. (2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 5(2), 155–174.

Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. MIT Press.

Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299.

Flaskerud, J. H. (2012). Cultural bias and Likert-type scales revisited. Mental Health Nursing, 33(2), 130–132.

Goodwin, L. D., & Leech, N. L. (2003). The meaning of validity in the new Standards for Educational and Psychological Testing: Implications for measurement courses. Measurement and Evaluation in Counseling and Development, 36, 181–191.

Gorard, S. (2010). Measuring is more than assigning numbers. In G. Walford, E. Tucker, & M. Viswanathan (Eds.), The SAGE handbook of measurement. SAGE.

Graybill, P., Aggas, J., Dean, R. K., Demers, S., Finigan, E. G., & Pollard Jr., R. Q. (2010). A community-participatory approach to adapting survey items for deaf individuals and American Sign Language. Field Methods, 22(4), 429–448.

Groth-Marnat, G., & Wright, A. J. (2016). Handbook of psychological assessment. Wiley.

Halverson, S. L. (2010). Cognitive translation studies: Developments in theory and method. In G. M. Shreve & E. Angelone (Eds.), Translation and cognition (pp. 349–369). John Benjamins.

Halverson, S. L. (2019). ‘Default’ translation: A construct for cognitive translation and interpreting studies. Translation, Cognition & Behavior, 2(2), 187–210.

Hambleton, R. K., & Patsula, L. (1998). Adapting tests for use in multiple languages and cultures. Social Indicators Research, 45(1–3), 153–171.

Han, C. (2018). Latent trait modeling of rater accuracy in formative peer assessment of English–Chinese consecutive interpreting. Assessment & Evaluation in Higher Education, 43(6), 979–994.

Harkness, J. A., Pennell, B.-E., & Shoua-Glusberg, A. (2004). Survey questionnaire translation and assessment. In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, & E. Singer (Eds.), Methods for testing and evaluating survey questionnaires (pp. 453–473). Wiley.

Harkness, J. A., Villar, A., & Edwards, B. (2010). Translation, adaptation, and design. In J. A. Harkness, M. Braun, B. Edwards, T. P. Johnson, L. Lyberg, P. P. Mohler, B.-E. Pennell, & T. W. Smith (Eds.), Survey methods in multinational, multiregional, and multicultural contexts (pp. 115–140). Wiley.

Harpe, S. E. (2015). How to analyze Likert and other rating scale data. Currents in Pharmacy Teaching and Learning, 7(6), 836–850.

House, J. (2013). Towards a new linguistic-cognitive orientation in translation studies. Target, 25(1), 46–60.

Hubscher-Davidson, S. (2009). Personal diversity and diverse personalities in translation: A study of individual differences. Perspectives, 17(3), 175–192.

Hurtado Albir, A. (Ed.). (2017). Researching translation competence by PACTE Group. John Benjamins.

Hvelplund, K. T. (2014). Eye tracking and the translation process: Reflections on the analysis and interpretation of eye-tracking data. MonTI Special Issue—Minding Translation, Special Issue 1, 201–223.

Hvelplund, K. T. (2017). Eye tracking in translation process research. In J. W. Schwieter & A. Ferreira (Eds.), The handbook of translation and cognition (pp. 248–264). Wiley.

Jääskeläinen, R. (2010). Think-aloud protocol. In Y. Gambier & L. van Doorslaer (Eds.), Handbook of translation studies (Vol 1, pp. 371–373). John Benjamins.

Jääskeläinen, R. (2011). Studying the translation process. In K. Malmkjær & K. Windle (Eds.), The Oxford handbook of translation studies (pp. 123–135). Oxford University Press.

Jakobsen, A. L. (2003). Effects of think aloud on translation speed, revision, and segmentation. In F. Alves (Ed.), Triangulating translation: Perspectives in process oriented research (pp. 69–95). John Benjamins.

Jamieson, S. (2004). Likert scales: How to (ab)use them. Medical Education, 38, 1212–1218.

Jiménez Ivars, A., Pinazo Catalavud, D., & Ruiz i Forés, M. (2014). Self-efficacy and language proficiency in interpreter trainees. The Interpreter and Translator Trainer, 8(2), 167–182.

Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixation to comprehension. Psychological Review, 87(4), 329–354.

Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64(3), 425–461.

Kankaraš, M., & Moors, G. (2010). Researching measurement equivalence in cross-cultural studies. Psihologija, 43(2), 121–136.

King, G., Murray, C. J. L., Salomon, J. A., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98(1), 191–207.

Kruger, J.-L., Soto Sanfiel, M. T., Doherty, S., & Ibrahim, R. (2016). Towards a cognitive audiovisual translatology: Subtitles and embodied cognition. In R. Muñoz Martín (Ed.), Reembedding translation process research (pp. 171–194). John Benjamins.

Lee, J. W., Jones, P. S., Mineyama, Y., & Zhang, X. (2002). Cultural differences in response to a Likert scale. Research in Nursing & Health, 25(4), 295–306.

Lee, S.-B. (2014). An interpreting self-efficacy (ISE) scale for undergraduate students majoring in consecutive interpreting: Construction and preliminary validation. The Interpreter and Translator Trainer, 8(2), 183–203.

Lee, S.-B. (2018). Exploring a relationship between students’ interpreting self-efficacy and performance: Triangulating data on interpreter performance assessment. The Interpreter and Translator Trainer, 12(2), 166–187.

Li, D. (2004). Trustworthiness of think-aloud protocols in the study of translation processes. International Journal of Applied Linguistics, 14(3), 301–313.

Likert, R. A. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 1–55.

Litwin, M. (1995). How to measure survey reliability and validity. SAGE.

Lubke, G. H., Dolan, C. V., Kelderman, H., & Mellenbergh, G. J. (2003). On the relationship between sources of within- and between-group differences and measurement invariance in the common factor model. Intelligence, 31, 543–566.

McGorry, S. Y. (2000). Measurement in a cross-cultural environment: Survey translation issues. Qualitative Market Research, 3(2), 74–81.

Mellinger, C. D. (2015). On the applicability of Internet-mediated research methods to investigate translators’ cognitive behavior. Translation & Interpreting, 7(1), 59–71.

Mellinger, C. D., & Hanson, T. A. (2017). Quantitative research methods in translation and interpreting studies. Routledge.

Mellinger, C. D., & Hanson, T. A. (2018). Interpreter traits and the relationship with technology and visibility. Translation and Interpreting Studies, 13(3), 366–392.

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.

Milfont, T. L., & Fischer, R. (2010). Testing measurement invariance across groups: Applications in cross-cultural research. International Journal of Psychological Research, 3(1), 111–121.

Muñoz Martín, R. (2016). Of minds and men–computers and translators. Poznań Studies in Contemporary Linguistics, 52(2), 351–381.

Muñoz Martín, R. (2017). Looking toward the future of cognitive translation studies. In J. W. Schwieter & A. Ferreira (Eds.), The handbook of translation and cognition (pp. 554–572). Wiley.

Norman, G. (2010). Likert scales, levels of measurement and the ‘laws’ of statistics. Advances in Health Sciences Education, 15(5), 625–632.

Nunnally, J. C. (1978). Psychometric theory. McGraw-Hill.

Oakes, M. P., & Ji, M. (Eds.). (2012). Quantitative methods in corpus-based translation studies. John Benjamins.

O’Brien, S. (2009). Eye tracking in translation process research: Methodological challenges and solutions. In I. M. Mees, F. Alves, & S. Göpferich (Eds.), Methodology, technology, and innovation in translation process research: A tribute to Arnt Lykke Jakobsen (pp. 251–266). Samfundslitteratur.

O’Brien, S. (2013). The borrowers: Researching the cognitive aspects of translation. Target, 25(1), 5–17.

O’Brien, S., & Saldanha, G. (2014). Research methodologies in translation studies. Routledge.

Peters, G.-J. Y. (2014). The alpha and the omega of scale reliability and validity: Why and how to abandon Cronbach’s alpha and the route towards more comprehensive assessment of scale quality. European Health Psychologist, 16(2), 56–69.

Pöchhacker, F. (2005). From operation to action: Process-orientation in interpreting studies. Meta, 50(2), 682–695.

Reid, J. (1990). The dirty laundry of ESL survey research. TESOL Quarterly, 24(2), 323–338.

Risku, H. (2012). Cognitive approaches to translation. In C. A. Chapelle (Ed.), The encyclopedia of applied linguistics (pp. 1–10). Wiley-Blackwell.

Risku, H. (2014). Translation process research as interaction research: From mental to socio-cognitive processes. MonTI Special Issue—Minding Translation, Special Issue 1, 331–353.

Rönkkö, M., & Evermann, J. (2013). A critical examination of common beliefs about partial least squares path modeling. Organizational Research Methods, 16(3), 155–174.

Rubin, R. B., Rubin, A. M., Graham, E. E., Perse, E. M., & Seibold, D. R. (2011). Communication research measures II: A sourcebook. Routledge.

Schaeffer, M., Huepe, D., Hansen-Schirra, S., Hofmann, S., Muñoz, E., Kogan, B., Herrera, E., Ibáñez, A., & García, A. (2020). The Translation and Interpreting Competence Questionnaire: An online tool for research on translators and interpreters. Perspectives, 28(1), 90–108.

Schwarz, N. (2007). Cognitive aspects of survey methodology. Applied Cognitive Psychology, 21(2), 277–287.

Shreve, G. M., & Angelone, E. (2010a). Translation and cognition. John Benjamins.

Shreve, G. M., & Angelone, E. (2010b). Translation and cognition: Recent developments. In G. M. Shreve & E. Angelone (Eds.), Translation and cognition (pp. 1–13). John Benjamins.

Shreve, G. M., Angelone, E., & Lacruz, I. (2018). Are expertise and translation competence really the same?: Psychological reality and the theoretical status of competence. In I. Lacruz & R. Jääskeläinen (Eds.), Innovation and expansion in translation process research (pp. 37–54). John Benjamins.

Smith, T. W. (2010). Survey across nations and cultures. In P. V. Marsden & J. D. Wright (Eds.), Handbook of survey research (pp. 733–763). Emerald.

Streiner, D. L. (2003). Being inconsistent about consistency: When coefficient alpha does and doesn’t matter. Journal of Personality Assessment, 80(3), 217–222.

Van Bork, R., Rhemtulla, M., Waldorp, L. J., Kruis, J., Rezvanifar, S., & Borsboom, D. (2019). Latent variable models and networks: Statistical equivalence and testability. Multivariate Behavioral Research, 1–24.

Vogt, W. P., & Johnson, R. B. (2016). The SAGE dictionary of statistics and methodology (5th ed.). SAGE.

Warrens, M. J. (2015). On Cronbach’s alpha as the mean of all split-half reliabilities. In R. Millsap, D. Bolt, L. van der Ark, & W. C. Wang (Eds.), Quantitative psychology research (pp. 293–300). Springer.

Wicherts, J. M., & Dolan, C. V. (2010). Measurement invariance in confirmatory factor analysis: An illustration using IQ test performance of minorities. Educational Measurement: Issues and Practice, 29(3), 39–47.

Wilcox, J. B., Howell, R. D., & Breivik, E. (2008). Questions about formative measurement. Journal of Business Research, 61(12), 1219–1228.

Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12(1), 58–79.

Zlomke, K. R. (2009). Psychometric properties of internet administered versions of Penn State Worry Questionnaire (PSWQ) and Depression, Anxiety, and Stress Scale (DASS). Computers in Human Behavior, 25, 841–843. 10.1016/j.chb.2008.06.003




How to Cite

Mellinger, C. D., & Hanson, T. A. (2021). Methodological considerations for survey research: Validity, reliability, and quantitative analysis. Linguistica Antverpiensia, New Series – Themes in Translation Studies, 19.