Methodological considerations for survey research: Validity, reliability, and quantitative analysis

Christopher D. Mellinger, Thomas A. Hanson

Abstract


As translation and interpreting studies continue to develop cognitive theories of translator and interpreter behavior and processing, there has been increased emphasis on research methods and data collection methodologies to glean new insights into the translation process. This article presents a critical review of survey research methods in Cognitive Translation Studies and argues for their inclusion as a means of better understanding translator and interpreter attitudes, behaviors, perceptions, and values. The article begins with a reflection on measurement and the need for alignment with theoretical frameworks and constructs; then it reviews important considerations when developing theoretically-grounded, empirically-based survey instruments, namely, validity, reliability, measurement invariance, and quantitative analysis. The article concludes with a call for additional methodological reflection on developing and using survey instruments.


Keywords


Survey; validity; reliability; measurement invariance; Cognitive Translation Studies

Full Text:

PDF HTML

References


Agbo, A. A. (2010). Cronbach’s alpha: Review of limitations and associated recommendations. Journal of Psychology in Africa, 20(2), 233–239. https://doi.org/10.1080/14330237.2010.10820371

Alves, F. (Ed.). (2003). Triangulating translation: Perspectives in process-oriented research. John Benjamins. https://doi.org/10.1075/btl.45

Alves, F. (2015). Translation process research at the interface: Paradigmatic, theoretical, and methodological issues in dialogue with cognitive science, expertise studies, and psycholinguistics. In A. Ferreira & J. W. Schwieter (Eds.), Psycholinguistic and cognitive inquiries into translation and interpreting (pp. 17–40). John Benjamins. https://doi.org/10.1075/btl.115.02alv

Alves, F., & Hurtado Albir, A. (2017). Evolution, challenges, and perspectives for research on cognitive aspects of translation. In J. W. Schwieter & A. Ferreira (Eds.), The handbook of translation and cognition (pp. 537–554). Wiley. https://doi.org/10.1002/9781119241485.ch29

Alvstad, C., Hild, A., & Tiselius, E. (2011). Methods and strategies of process research: Integrative approaches in translation studies. In C. Alvstad, A. Hild, & E. Tiselius (Eds.), Methods and strategies of process research (pp. 1–9). John Benjamins. https://doi.org/10.1075/btl.94

American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1999). Standards for educational and psychological testing. American Educational Research Association.

Angelelli, C. V. (2004). Revisiting the interpreter’s role: A study of conference, court, and medical interpreters in Canada, Mexico, and the United States. John Benjamins. https://doi.org/10.1075/btl.55

Angelelli, C. V., & Baer, B. J. (Eds.). (2016). Researching translation and interpreting. Routledge. https://doi.org/10.4324/9781315707280

Balling, L. W. (2008). A brief introduction to regression designs and mixed-effects modelling by a recent convert. Copenhagen Studies in Language, 36, 175–192.

Bandalos, D. L. (2018). Measurement theory and applications for the social sciences. Guilford Press.

Beaujean, A. A. (2014). Latent variable modeling using R: A step-by-step guide. Routledge. https://doi.org/10.4324/9781315869780

Behr, D., & Shishido, K. (2016). The translation of measurement instruments for cross-cultural surveys. In C. Wolf, D. Joye, T. W. Smith, & Y.-C. Fu (Eds.), The SAGE handbook of survey methodology (pp. 269–287). SAGE. https://doi.org/10.4135/9781473957893.n19

Bernardini, S. (2001). Think-aloud protocols in translation research: Achievements, limits, future prospects. Target, 13(2), 241–263. https://doi.org/10.1075/target.13.2.03ber

Bolaños-Medina, A. (2014). Self-efficacy in translation. Translation and Interpreting Studies, 9(2), 197–218. https://doi.org/10.1075/tis.9.2.03bol

Bollen, K. A. (2002). Latent variables in psychology and the social sciences. Annual Review of Psychology, 53(1), 605–634. https://doi.org/10.1146/annurev.psych.53.100901.135239

Bontempo, K., & Napier, J. (2011). Evaluating emotional stability as a predictor of interpreter competence and aptitude for interpreting. Interpreting, 13(1), 85–105. https://doi.org/10.1075/intp.13.1.06bon

Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press. https://doi.org/10.1017/CBO9780511490026

Boyle, G. J., Saklofske, D. H., & Matthews, G. (Eds.). (2015). Measures of personality and social psychological constructs. Academic Press.

Brown, J. D. (1996). Testing in language programs. Prentice Hall.

Buchanan, E. A., & Hvizdak, E. E. (2009). Online survey tools: Ethical and methodological concerns of human research ethics committees. Journal of Empirical Research on Human Research Ethics, 4(2), 37–48. https://doi.org/10.1525/jer.2009.4.2.37

Carifio, J., & Perla, R. J. (2007). Ten common misunderstandings, misconceptions, persistent myths and urban legends about Likert scales and Likert response formats and their antidotes. Journal of Social Sciences, 3(3), 106–116. https://doi.org/ 10.3844/jssp.2007.106.116

Carifio, J., & Perla, R. J. (2008). Resolving the 50-year debate around using and misusing Likert scales. Medical Education, 42(12), 1150–1152. https://doi.org/10.1111/j.1365-2923.2008.03172.x

Chan, E. K. H. (2014). Standards and guidelines for validation practices: Development and evaluation of measurement instruments. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 9–24). Springer. https://doi.org/10.1007/978-3-319-07794-9_2

Chesterman, A. (2009). The name and nature of translator studies. Hermes, 22(42), 13–22. https://doi.org/10.7146/hjlcb.v22i42.96844

Choi, B. C. K., & Pak, A. W. P. (2005). A catalog of biases in questionnaires. Preventing Chronic Disease, 2(1), A13.

Clark, A. (1996). Being there: Putting brain, body, and world together again. MIT Press. https://doi.org/10.7551/mitpress/1552.001.0001

Cole, D. A., & Preacher, K. J. (2014). Manifest variable path analysis: Potentially serious and misleading consequences due to uncorrected measurement error. Psychological Methods, 19(2), 300–315. https://doi.org/10.1037/a0033805

Cortina, J. M. (1993). What is coefficient alpha?: An examination of theory and applications. Journal of Applied Psychology, 78(1), 98–104. https://doi.org/10.1037/0021-9010.78.1.98

Coulacoglou, C., & Saklofske, D. H. (2017). Psychometrics and psychological assessment: Principles and applications. Academic Press.

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957

Csizér, K., & Dörnyei, Z. (2005). The internal structure of language learning motivation and its relationship with language choice and learning effort. Modern Language Journal, 89(1), 19–36. https://doi.org/10.1111/j.0026-7902.2005.00263.x

Dancette, J. (1997). Mapping meaning and comprehension in translation: Theoretical and experimental issues. In J. H. Danks, G. M. Shreve, S. B. Fountain, & M. K. McBeath (Eds.), Cognitive processes in translation and interpreting (pp. 77–103). SAGE.

DeCastellarnau, A. (2018). A classification of response scale characteristics that affect data quality: A literature review. Quality & Quantity, 52(4), 1523–1559. https://doi.org/10.1007/s11135-017-0533-4

DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). SAGE.

Diamantopoulos, A. (Ed.). (2008). Formative indicators [Special issue]. Journal of Business Research, 61(12).

Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105(3), 399–412. https://doi.org/10.1111/bjop.12046

Edwards, J. R., & Bagozzi, R. P. (2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 5(2), 155–174. https://doi.org/10.1037/1082-989X.5.2.155

Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. MIT Press.

Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299. https://doi.org/10.1037/1082-989X.4.3.272

Flaskerud, J. H. (2012). Cultural bias and Likert-type scales revisited. Mental Health Nursing, 33(2), 130–132. https://doi.org/10.3109/01612840.2011.600510

Goodwin, L. D., & Leech, N. L. (2003). The meaning of validity in the new Standards for Educational and Psychological Testing: Implications for measurement courses. Measurement and Evaluation in Counseling and Development, 36, 181–191. https://doi.org/10.1080/07481756.2003.11909741

Gorard, S. (2010). Measuring is more than assigning numbers. In G. Walford, E. Tucker, & M. Viswanathan (Eds.), The SAGE handbook of measurement. SAGE. https://doi.org/10.4135/9781446268230.n20

Graybill, P., Aggas, J., Dean, R. K., Demers, S., Finigan, E. G., & Pollard Jr., R. Q. (2010). A community-participatory approach to adapting survey items for deaf individuals and American Sign Language. Field Methods, 22(4), 429–448. https://doi.org/10.1177/1525822X10379201

Groth-Marnat, G., & Wright, A. J. (2016). Handbook of psychological assessment. Wiley.

Halverson, S. L. (2010). Cognitive translation studies: Developments in theory and method. In G. M. Shreve & E. Angelone (Eds.), Translation and cognition (pp. 349–369). John Benjamins. https://doi.org/10.1075/ata.xv.18hal

Halverson, S. L. (2019). ‘Default’ translation: A construct for cognitive translation and interpreting studies. Translation, Cognition & Behavior, 2(2), 187–210. https://doi.org/10.1075/tcb.00023.hal

Hambleton, R. K., & Patsula, L. (1998). Adapting tests for use in multiple languages and cultures. Social Indicators Research, 45(1–3), 153–171. https://doi.org/10.1023/A:1006941729637

Han, C. (2018). Latent trait modeling of rater accuracy in formative peer assessment of English–Chinese consecutive interpreting. Assessment & Evaluation in Higher Education, 43(6), 979–994. https://doi.org/10.1080/02602938.2018.1424799

Harkness, J. A., Pennell, B.-E., & Shoua-Glusberg, A. (2004). Survey questionnaire translation and assessment. In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, & E. Singer (Eds.), Methods for testing and evaluating survey questionnaires (pp. 453–473). Wiley. https://doi.org/10.1002/0471654728.ch22

Harkness, J. A., Villar, A., & Edwards, B. (2010). Translation, adaptation, and design. In J. A. Harkness, M. Braun, B. Edwards, T. P. Johnson, L. Lyberg, P. P. Mohler, B.-E. Pennell, & T. W. Smith (Eds.), Survey methods in multinational, multiregional, and multicultural contexts (pp. 115–140). Wiley. https://doi.org/10.1002/9780470609927.ch7

Harpe, S. E. (2015). How to analyze Likert and other rating scale data. Currents in Pharmacy Teaching and Learning, 7(6), 836–850. https://doi.org/10.1016/j.cptl.2015.08.001

House, J. (2013). Towards a new linguistic-cognitive orientation in translation studies. Target, 25(1), 46–60. https://doi.org/10.1075/target.25.1.05hou

Hubscher-Davidson, S. (2009). Personal diversity and diverse personalities in translation: A study of individual differences. Perspectives, 17(3), 175–192. https://doi.org/10.1080/09076760903249380

Hurtado Albir, A. (Ed.). (2017). Researching translation competence by PACTE Group. John Benjamins. https://doi.org/10.1075/btl.127

Hvelplund, K. T. (2014). Eye tracking and the translation process: Reflections on the analysis and interpretation of eye-tracking data. MonTI Special Issue—Minding Translation, Special Issue 1, 201–223. https://doi.org/10.6035/monti.v0i0.292854

Hvelplund, K. T. (2017). Eye tracking in translation process research. In J. W. Schwieter & A. Ferreira (Eds.), The handbook of translation and cognition (pp. 248–264). Wiley. https://doi.org/10.1002/9781119241485.ch14

Jääskeläinen, R. (2010). Think-aloud protocol. In Y. Gambier & L. van Doorslaer (Eds.), Handbook of translation studies (Vol 1, pp. 371–373). John Benjamins. https://doi.org/10.1075/hts.1.thi1

Jääskeläinen, R. (2011). Studying the translation process. In K. Malmkjær & K. Windle (Eds.), The Oxford handbook of translation studies (pp. 123–135). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199239306.013.0010

Jakobsen, A. L. (2003). Effects of think aloud on translation speed, revision, and segmentation. In F. Alves (Ed.), Triangulating translation: Perspectives in process oriented research (pp. 69–95). John Benjamins. https://doi.org/10.1075/btl.45.08jak

Jamieson, S. (2004). Likert scales: How to (ab)use them. Medical Education, 38, 1212–1218. https://doi.org/10.1111/j.1365-2929.2004.02012.x

Jiménez Ivars, A., Pinazo Catalavud, D., & Ruiz i Forés, M. (2014). Self-efficacy and language proficiency in interpreter trainees. The Interpreter and Translator Trainer, 8(2), 167–182. https://doi.org/10.1080/1750399X.2014.908552

Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixation to comprehension. Psychological Review, 87(4), 329–354. https://doi.org/10.1037/0033-295X.87.4.329

Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64(3), 425–461. https://doi.org/10.3102/00346543064003425

Kankaraš, M., & Moors, G. (2010). Researching measurement equivalence in cross-cultural studies. Psihologija, 43(2), 121–136. https://doi.org/10.2298/PSI1002121K

King, G., Murray, C. J. L., Salomon, J. A., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98(1), 191–207. https://doi.org/10.1017/S000305540400108X

Kruger, J.-L., Soto Sanfiel, M. T., Doherty, S., & Ibrahim, R. (2016). Towards a cognitive audiovisual translatology: Subtitles and embodied cognition. In R. Muñoz Martín (Ed.), Reembedding translation process research (pp. 171–194). John Benjamins. https://doi.org/10.1075/btl.128.09kru

Lee, J. W., Jones, P. S., Mineyama, Y., & Zhang, X. (2002). Cultural differences in response to a Likert scale. Research in Nursing & Health, 25(4), 295–306. https://doi.org/10.1002/nur.10041

Lee, S.-B. (2014). An interpreting self-efficacy (ISE) scale for undergraduate students majoring in consecutive interpreting: Construction and preliminary validation. The Interpreter and Translator Trainer, 8(2), 183–203. https://doi.org/10.1080/1750399X.2014.929372

Lee, S.-B. (2018). Exploring a relationship between students’ interpreting self-efficacy and performance: Triangulating data on interpreter performance assessment. The Interpreter and Translator Trainer, 12(2), 166–187. https://doi.org/10.1080/1750399X.2017.1359763

Li, D. (2004). Trustworthiness of think-aloud protocols in the study of translation processes. International Journal of Applied Linguistics, 14(3), 301–313. https://doi.org/10.1111/j.1473-4192.2004.00067.x

Likert, R. A. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 1–55.

Litwin, M. (1995). How to measure survey reliability and validity. SAGE. https://doi.org/10.4135/9781483348957

Lubke, G. H., Dolan, C. V., Kelderman, H., & Mellenbergh, G. J. (2003). On the relationship between sources of within- and between-group differences and measurement invariance in the common factor model. Intelligence, 31, 543–566. https://doi.org/10.1016/S0160-2896(03)00051-5

McGorry, S. Y. (2000). Measurement in a cross-cultural environment: Survey translation issues. Qualitative Market Research, 3(2), 74–81. https://doi.org/10.1108/13522750010322070

Mellinger, C. D. (2015). On the applicability of Internet-mediated research methods to investigate translators’ cognitive behavior. Translation & Interpreting, 7(1), 59–71.

Mellinger, C. D., & Hanson, T. A. (2017). Quantitative research methods in translation and interpreting studies. Routledge. https://doi.org/10.4324/9781315647845

Mellinger, C. D., & Hanson, T. A. (2018). Interpreter traits and the relationship with technology and visibility. Translation and Interpreting Studies, 13(3), 366–392. https://doi.org/10.1075/tis.00021.mel

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741

Milfont, T. L., & Fischer, R. (2010). Testing measurement invariance across groups: Applications in cross-cultural research. International Journal of Psychological Research, 3(1), 111–121. https://doi.org/10.21500/20112084.857

Muñoz Martín, R. (2016). Of minds and men–computers and translators. Poznań Studies in Contemporary Linguistics, 52(2), 351–381. https://doi.org/10.1515/psicl-2016-0013

Muñoz Martín, R. (2017). Looking toward the future of cognitive translation studies. In J. W. Schwieter & A. Ferreira (Eds.), The handbook of translation and cognition (pp. 554–572). Wiley. https://doi.org/10.1002/9781119241485.ch30

Norman, G. (2010). Likert scales, levels of measurement and the ‘laws’ of statistics. Advances in Health Sciences Education, 15(5), 625–632. https://doi.org/10.1007/s10459-010-9222-y

Nunnally, J. C. (1978). Psychometric theory. McGraw-Hill.

Oakes, M. P., & Ji, M. (Eds.). (2012). Quantitative methods in corpus-based translation studies. John Benjamins. https://doi.org/10.1075/scl.51

O’Brien, S. (2009). Eye tracking in translation process research: Methodological challenges and solutions. In I. M. Mees, F. Alves, & S. Göpferich (Eds.), Methodology, technology, and innovation in translation process research: A tribute to Arnt Lykke Jakobsen (pp. 251–266). Samfundslitteratur.

O’Brien, S. (2013). The borrowers: Researching the cognitive aspects of translation. Target, 25(1), 5–17. https://doi.org/10.1075/target.25.1.02obr

O’Brien, S., & Saldanha, G. (2014). Research methodologies in translation studies. Routledge. https://doi.org/10.4324/9781315760100

Peters, G.-J. Y. (2014). The alpha and the omega of scale reliability and validity: Why and how to abandon Cronbach’s alpha and the route towards more comprehensive assessment of scale quality. European Health Psychologist, 16(2), 56–69.

Pöchhacker, F. (2005). From operation to action: Process-orientation in interpreting studies. Meta, 50(2), 682–695. https://doi.org/10.7202/011011ar

Reid, J. (1990). The dirty laundry of ESL survey research. TESOL Quarterly, 24(2), 323–338. https://doi.org/10.2307/3586913

Risku, H. (2012). Cognitive approaches to translation. In C. A. Chapelle (Ed.), The encyclopedia of applied linguistics (pp. 1–10). Wiley-Blackwell. https://doi.org/10.1002/9781405198431.wbeal0145

Risku, H. (2014). Translation process research as interaction research: From mental to socio-cognitive processes. MonTI Special Issue—Minding Translation, Special Issue 1, 331–353. https://doi.org/10.6035/MonTI.2014.ne1.11

Rönkkö, M., & Evermann, J. (2013). A critical examination of common beliefs about partial least squares path modeling. Organizational Research Methods, 16(3), 155–174. https://doi.org/10.1177/1094428112474693

Rubin, R. B., Rubin, A. M., Graham, E. E., Perse, E. M., & Seibold, D. R. (2011). Communication research measures II: A sourcebook. Routledge. https://doi.org/10.4324/9780203871539

Schaeffer, M., Huepe, D., Hansen-Schirra, S., Hofmann, S., Muñoz, E., Kogan, B., Herrera, E., Ibáñez, A., & García, A. (2020). The Translation and Interpreting Competence Questionnaire: An online tool for research on translators and interpreters. Perspectives, 28(1), 90–108. https://doi.org/10.1080/0907676X.2019.1629468

Schwarz, N. (2007). Cognitive aspects of survey methodology. Applied Cognitive Psychology, 21(2), 277–287. https://doi.org/10.1002/acp.1340

Shreve, G. M., & Angelone, E. (2010a). Translation and cognition. John Benjamins. https://doi.org/10.1075/ata.xv

Shreve, G. M., & Angelone, E. (2010b). Translation and cognition: Recent developments. In G. M. Shreve & E. Angelone (Eds.), Translation and cognition (pp. 1–13). John Benjamins. https://doi.org/10.1075/ata.xv.01shr

Shreve, G. M., Angelone, E., & Lacruz, I. (2018). Are expertise and translation competence really the same?: Psychological reality and the theoretical status of competence. In I. Lacruz & R. Jääskeläinen (Eds.), Innovation and expansion in translation process research (pp. 37–54). John Benjamins. https://doi.org/10.1075/ata.18.03shr

Smith, T. W. (2010). Survey across nations and cultures. In P. V. Marsden & J. D. Wright (Eds.), Handbook of survey research (pp. 733–763). Emerald.

Streiner, D. L. (2003). Being inconsistent about consistency: When coefficient alpha does and doesn’t matter. Journal of Personality Assessment, 80(3), 217–222. https://doi.org/10.1207/S15327752JPA8003_01

Van Bork, R., Rhemtulla, M., Waldorp, L. J., Kruis, J., Rezvanifar, S., & Borsboom, D. (2019). Latent variable models and networks: Statistical equivalence and testability. Multivariate Behavioral Research, 1–24. https://doi.org/10.1080/00273171.2019.1672515

Vogt, W. P., & Johnson, R. B. (2016). The SAGE dictionary of statistics and methodology (5th ed.). SAGE.

Warrens, M. J. (2015). On Cronbach’s alpha as the mean of all split-half reliabilities. In R. Millsap, D. Bolt, L. van der Ark, & W. C. Wang (Eds.), Quantitative psychology research (pp. 293–300). Springer. https://doi.org/10.1007/978-3-319-07503-7_18

Wicherts, J. M., & Dolan, C. V. (2010). Measurement invariance in confirmatory factor analysis: An illustration using IQ test performance of minorities. Educational Measurement: Issues and Practice, 29(3), 39–47. https://doi.org/10.1111/j.1745-3992.2010.00182.x

Wilcox, J. B., Howell, R. D., & Breivik, E. (2008). Questions about formative measurement. Journal of Business Research, 61(12), 1219–1228. https://doi.org/10.1016/j.jbusres.2008.01.010

Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12(1), 58–79. https://doi.org/10.1037/1082-989X.12.1.58

Zlomke, K. R. (2009). Psychometric properties of internet administered versions of Penn State Worry Questionnaire (PSWQ) and Depression, Anxiety, and Stress Scale (DASS). Computers in Human Behavior, 25, 841–843. https://doi.org/ 10.1016/j.chb.2008.06.003