Quality assessment of machine-translated post-edited subtitles: an analysis of Brazilian translators’ perceptions
DOI:
https://doi.org/10.52034/lans-tts.v22i.765Keywords:
subtitling, Machine Translation, post-editing, audiovisual translation, acceptabilityAbstract
This study focuses on the translation product in the form of subtitles and, in particular, investigates the perceived quality of machine-translated post-edited interlingual subtitles. Basing this study on data collected from Brazilian professional translators, we analyse whether the use of machine translation has an impact on the perceived quality and acceptability of interlingual subtitles. We also examine those technical parameters and linguistic aspects that the study participants considered troublesome and whether translators would report any evidence that the subtitles were machine-translated. Sixty-eight Brazilian translators volunteered to participate in this study. They completed a questionnaire, watched a movie trailer with post-edited subtitles, and then assessed the quality of the subtitles by rating them using a Likert-type scale. Finally, the participants responded to a written verbal protocol. Our analysis relies on the qualitative and quantitative data provided by the participants. The qualitative and quantitative data from the protocol answers were triangulated with the quantitative results from the Likert-type scale. IBM SPSS Statistics was used for statistical analysis. Our findings show that 28.1% of the translators assessed the post-edited subtitles as very satisfactory whereas 48.4% judged them as satisfactory. These results seem to be an indication of the acceptability of post-editing machine-translated subtitles as they find support in the written verbal protocol answers. Regarding the technical parameters, 43% of the participants reported no issues whereas 22.8% commented on the font colour and the font size. When asked about any linguistic aspects that could have disturbed the viewing experience, 64.1% of them reported that no linguistic issues were evident in the subtitles. A small percentage of the participants reported some linguistic problems, such as a lack of accuracy, the use of literal translation, and unnatural subtitles. Despite this apparent linguistic evidence of machine translation, no participant explicitly affirmed that the subtitles were machine-translated and post-edited.
References
Athanasiadi, R. (2017). Exploring the potential of machine translation and other language assistive tools in subtitling: A new era? In M. Deckert (Ed.), Audiovisual translation: Research and use (pp. 29–49). Peter Lang.
Audiovisual Translators Europe (AVTE). (2021). AVTE Machine translation manifesto. Retrieved from http://avteurope.eu/wp-content/uploads/2021/09/Machine-Translation-Manifesto_ENG.pdf
Baños Piñero, R., & Díaz Cintas, J. (2015). Audiovisual translation in a global context. In R. Baños Piñero & J. Díaz Cintas (Eds.), Audiovisual translation in a global context: Mapping an ever-changing landscape (pp. 1–10). Palgrave Macmillan. https://doi.org/10.1057/9781137552891_1
Birulés-Muntané, J., & Soto-Faraco, S. (2016). Watching subtitled films can help learning foreign languages. PLoS One, 11(6), 1–11. https://doi.org/10.1371/journal.pone.0158409
Brendel, J., & Vela, M. (2022). Quality assessment of subtitles: Challenges and strategies. In P. Sojka, A. Horák, I. Kope?ek, & K. Pala (Eds.), Text, speech, and dialogue (pp. 52–63). Springer. https://doi.org/10.1007/978-3-031-16270-1_5
Burchardt, A., Lommel, A., Bywood, L., Harris, K., & Popovi?, M. (2016). Machine translation quality in an audiovisual context. Target, 28(2), 206–221. https://doi.org/10.1075/target.28.2.03bur
Bywood, L., Etchegoyhen, T., & Georgakopoulou, P. (2017). Embracing the threat: Machine translation as a solution for subtitling. Perspectives: Studies in translation theory and practice, 25(3), 492–508. https://doi.org/10.1080/0907676X.2017.1291695
Bywood, L. (2020). Technology and audiovisual translation. In ?. Bogucki & M. Deckert (Eds.), The Palgrave handbook of audiovisual translation and media accessibility (pp. 503–517). Palgrave Macmillan. https://doi.org/10.1007/978-3-030-42105-2_25
Campos, G. C., & Azevedo, T. A. (2020). Subtitling for streaming platforms: New technologies, old issues. Cadernos de Tradução, 40(3), 222–243. https://doi.org/10.5007/2175-7968.2020v40n3p222
Carl, M., Dragsted, B., Elming, J., Hardt, D., & Jakobsen, A. L. (2011). The process of post-editing: A pilot study. In B. Sharp, M. Zock, M. Carl, & A. L. Jakobsen (Eds.), Proceedings of the 8th International NLPCS Workshop (pp. 131–142). Samfundslitteratur.
Carvalho, S. R. S., & Seoane, A. F. (2019). A relação entre unidades de velocidade de leitura na legendagem para surdos e ensurdecidos. Transversal – Revista em Tradução, 5(9), 137–153.
Castilho, S. (2011). Automatic and semi-automatic translation Of DVD subtitle: Measuring post-editing effort. [Unpublished master’s thesis]. University of Wolverhampton.
Castilho, S., & Resende, N. (2022). Post-editese in literary translations. Information, 13(2), 1–22. https://doi.org/10.3390/info13020066
Daems, J., Macken, L., & Vandepitte, S. (2013). Quality as the sum of its parts: A two-step approach for the identification of translation problems and translation quality assessment for HT and MT+PE. In S. O’Brien, M. Simard, & L. Specia (Eds.), Proceedings of the 2nd Workshop on post-editing technology and practice (WPTP-2) (pp. 63–71). European Association for Machine Translation. https://aclanthology.org/2013.mtsummit-wptp.8/
Díaz Cintas, J. (2013). The technology turn in subtitling. In M. Thelen & B. Lewandowska-Tomaszczyk (Eds.), Translation and meaning (Part 9, pp. 119–132). Maastricht School of Translation and Interpreting.
Díaz Cintas, J., & Massidda, S. (2020). Technological advances in audiovisual translation. In M. O’Hagan (Ed.), The Routledge handbook of translation and technology (pp. 255–270). Routledge. https://doi.org/10.4324/9781315311258-15
Díaz Cintas, J., & Remael, A. (2020). Subtitling: Concepts and practices. Routledge. https://doi.org/10.4324/9781315674278
Doherty, S., & Kruger, J. L. (2018). Assessing quality in human and machine-generated subtitles and captions. In J. Moorkens, S. Castilho, F. Gaspari & S. Doherty (Eds.), Translation quality assessment: From principles to practice (pp. 179–197). Springer. https://doi.org/10.1007/978-3-319-91241-7_9
Etchegoyhen, T., Bywood, L., Fishel, M., Georgakopoulou, P., Jiang, J., & Loenhout, G. (2014). Machine translation for subtitling: A large-scale evaluation. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), (pp. 46–53). https://aclanthology.org/L14-1392/
Freire, R. L. (2015). The introduction of film subtitling in Brazil (Trans. R. I. Pessoa). MATRIZes, 9(1), 187–210. https://doi.org/10.11606/issn.1982-8160.v9i1p187-211
García, I. (2011). Translating by post-editing: Is it the way forward? Machine translation, 25(3), 217–237. https://doi.org/10.1007/s10590-011-9115-8
Georgakopoulou, P. (2019). Technologization of audiovisual translation. In L. Pérez-González (Ed.), The Routledge handbook of audiovisual translation (pp. 516–539). Routledge. https://doi.org/10.4324/9781315717166-32
Georgakopoulou, P., & Bywood, L. (2014). MT in subtitling and the rising profile of the post-editor. Multilingual, 25(1), 24–28.
Guerberof Arenas, A. (2008). Productivity and quality in the post-editing of outputs from translation memories and machine translation. Localization Focus The International Journal of Localization, 7(1), 11–21. https://core.ac.uk/download/pdf/228161879.pdf
International Organization for Standardization. (2017). Translation services—Post-editing of machine translation output—Requirements (ISO Standard No. 18587:2017). https://www.iso.org/standard/62970.html
Karakanta, A. (2022). Experimental research in automatic subtitling: At the crossroads between machine translation and audiovisual translation. Translation Spaces, 11(1), 89–112. https://doi.org/10.1075/ts.21021.kar
Karakanta, A., Bentivogli, L., Cettolo, M., Negri, M., & Turchi, M. (2022). Post-editing in automatic subtitling: A subtitlers’ perspective. Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, 261–270. https://aclanthology.org/2022.eamt-1.29
Koglin, A. (2008). A tradução de metáforas geradoras de humor na série televisiva Friends: Um estudo de legendas [Master’s dissertation, Federal University of Santa Catarina]. UFSC Institutional Repository. https://repositorio.ufsc.br/xmlui/handle/123456789/90889
Koglin, A., Silveira, J. G. P., Matos, M. A., Silva, V. T. C., & Moura, W. H .C. (2022). Quality of post-edited interlingual subtitling: FAR model, translator's assessment and audience reception. Cadernos de Tradução, 42(1), 1–26. https://doi.org/10.5007/2175-7968.2022.e82143
Koponen, M., Sulubacak, U., Vitikainen, K., & Tiedemann, J. (2020). MT for subtitling: Investigating professional translators’ user experience and feedback. Proceedings of the 14th Conference of the Association for Machine Translation in the Americas, 79–92. https://helda.helsinki.fi/handle/10138/324207
Ku?çu-Özbudak, S. (2022) The role of subtitling on Netflix: An audience study. Perspectives: Studies in Translation Theory and Practice, 30(3), 537–551. https://doi.org/10.1080/0907676X.2020.1854794
Läubli, S., Fishel, M., Massey, G., Ehrensberger-Dow, M., & Volk, M. (2013). Assessing post-editing efficiency in a realistic translation environment. Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice, 83–91. https://doi.org/10.5167/uzh-80891
Mangiron, C. (2022). Audiovisual translation and multimedia and game localisation. In F. Zanettin & C. Rundle (Eds.), The Routledge handbook of translation and methodology (pp. 410–424). Routledge. https://doi.org/10.4324/9781315158945-29
Matusov, E., Wilken, P., & Georgakopoulou, P. (2019). Customizing neural machine translation for subtitling. Proceedings of the Fourth Conference on Machine Translation (WMT): Research Papers, 82–93. https://doi.org/10.18653/v1/W19-5209
Netflix (2023). Brazilian Portuguese timed text style guide. https://partnerhelp.netflixstudios.com/hc/en-us/articles/215600497-Brazilian-Portuguese-Timed-Text-Style-Guide
Pedersen, J. (2017). The FAR model: Assessing quality in interlingual subtitling. The Journal of Specialised Translation, 28, 210–229. https://jostrans.org/issue28/art_pedersen.pdf
Plitt, M., & Masselot, F. (2010). A productivity test of statistical machine translation post-editing in a typical localisation context. The Prague Bulletin of Mathematical Linguistics, 93, 7–16. https://ufal.ms.mff.cuni.cz/pbml/93/art-plitt-masselot.pdf
Pym, A. (2012). Democratizing translation technologies: The role of humanistic research. In V. Cannavina & A. Fellet (Eds.), Language and Translation Automation Conference (pp. 14–29). The Big Wave. http://hdl.handle.net/11343/194362
Raff, G. (Director). (2019). The Red Sea Diving Resort [Film]. Bron Studios; EMJAG Productions; Shaken Not Stirred; Creative Wealth Media Finance; Searchlight Pictures.
Robert, I., & Remael, A. (2016). Quality control in the subtitling industry: An exploratory survey study. Meta, 61(3), 578–605. https://doi.org/10.7202/1039220ar
Screen, B. (2017). Productivity and quality when editing machine translation and translation memory outputs: An empirical analysis of English to Welsh translation. Studia Celtica Posnaniensia, 2(1), 119–142. https://doi.org/10.1515/scp-2017-0007
Skadi?š, R., Puri?š, M., Skadi?a, I., & Vasiljevs, A. (2011). Evaluation of SMT in localization to under-resourced inflected language. Proceedings of the 15th International Conference of the European Association for Machine Translation, 35–40. https://aclanthology.org/2011.eamt-1.7.pdf
Szarkowska, A., Díaz Cintas, J., & Gerber-Morón, O. (2021). Quality is in the eye of the stakeholders: What do professional subtitlers and viewers think about subtitling? Universal Access in the Information Society, 20, 661–675. https://doi.org/10.1007/s10209-020-00739-2
Toral, A. (2019). Post-editese: An exacerbated translationese. Proceedings of the Machine Translation Summit, 19–23. https://doi.org/10.48550/arXiv.1907.00900
Vieira, P. A., Assis, Í. A. P., & Araújo, V. L. S. (2020). Tradução audiovisual: Estudos sobre a leitura de legendas para surdos e ensurdecidos. Cadernos de Tradução, 40(spe.2), 97–124. https://doi.org/10.5007/2175-7968.2020v40nesp2p97
Volk, M. (2008). The automatic translation of film subtitles: A machine translation success story? Journal for Language Technology and Computational Linguistics, 24(3), 115–128 https://doi.org/10.21248/jlcl.24.2009.124
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Arlene Koglin, Willian Henrique Cândido Moura, Morgana Aparecida de Matos, João Gabriel Pereira da Silveira
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under the CC BY-NC 4.0 Deed that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal. The material cannot be used for commercial purposes.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
