Key challenges in using automatic dubbing to translate educational YouTube videos
DOI:
https://doi.org/10.52034/lans-tts.v22i.763Keywords:
automatic dubbing, machine translation, speech synthesis, dubbing, educational videosAbstract
The dubbing industry has been particularly concerned with how to cope with the higher demand in dubbing experienced over the past few years. Media companies have invested in Artificial Intelligence, with significant developments on automatic dubbing spearheaded by companies like Google (Kottahachchi & Abeysinghe, 2022), Amazon (Federico et al., 2020; Lakew et al., 2021) or AppTek (Di Gangi et al., 2022). These organisations have clearly seen the potential in automatic dubbing, understood as the combination of automatic speech recognition, machine translation and text to speech technologies to automatically replace the audio track of an original audiovisual text with synthetic speech in a different language, taking into consideration relevant synchronies. Against this backdrop, this article sets out to: 1) provide an overview of automatic dubbing in the current mediascape and to contextualise this practice accordingly; and 2) to identify some of the main challenges of its implementation, especially as regards the integration of MT and speech synthesis in the dubbing workflow. To this end, the performance of the tool Aloud will be evaluated, by analysing the videos currently available in the Spanish version of the YouTube Channel Amoeba Sisters, which have been dubbed using this tool. The evaluation will focus on the issues highlighted by YouTube users in their comments on this YouTube channel, which include naturalness and accuracy. Special attention will be paid to the use of synthetic voices, which are heavily criticised by users. However, they also highlight the usefulness of automatic dubbing for students interested in biology-related topics who are not fluent in English, in line with the original intent of the tool developers of increasing accessibility.
References
abigarcia9588. (2022). Re: Fotosíntesis. [Video]. YouTube. https://www.youtube.com/watch?v=2-lg8yOsZgA
Akhulkova, Y., Hickey, S., & Hynes, R. (2022). The Nimdzi language technology atlas. https://www.nimdzi.com/language-technology-atlas
Ameri, S., & Khoshsaligheh, M. (2018). Exploring the attitudes and expectations of Iranian audiences in terms of professional dubbing into Persian. HERMES – Journal of Language and Communication in Business, 57, 175–193. https://doi.org/10.7146/hjlcb.v0i57.106206
Amoeba Sisters (2016) Biomolecules (Older Video 2016) [Video]. YouTube. https://www.youtube.com/watch?v=YO244P1e9QM
Amoeba Sisters. (2022). Pinky’s ed tech tips: Translations. https://www.amoebasisters.com/pinkys-ed-tech-favorites/community-contributed-subtitles
Amoeba Sisters (n.d.) Home [Video]. YouTube. https://www.youtube.com/@AmoebaSisters
Amoeba Sisters en Español (2021, October 21) Biomoléculas [Video]. YouTube. https://www.youtube.com/watch?v=gk9WmmRWgxA
Amoeba Sisters en Español (n.d.) Home [Video]. YouTube. https://youtube.com/@AmoebaSistersEspanol
Amoeba Sisters em Português (n.d.) Home [Video]. YouTube. https://youtube.com/@AmoebaSistersPortuguese
AmoebaSistersEspanol. (2021). Re: Replicación de ADN. [Video]. YouTube. https://www.youtube.com/watch?v=ddnuI2kYJ6c&
angelguardiaardiaca7364. (2022). Re: Respiración celular. [Video]. YouTube. https://www.youtube.com/watch?v=XZ246DX3zAU
Baños, R. (2014). Orality markers in Spanish native and dubbed sitcoms: Pretended spontaneity and prefabricated orality. Meta, 59(2), 406–435. https://doi.org/10.7202/1027482ar
Baños, R. (2018). Technology and audiovisual translation. In C. Sin-wai (Ed.), An Encyclopedia of practical translation and interpreting (pp. 3–30). Chinese University Press.
BBC Media Centre. (2015). BBC introduces new automatic virtual voiceover translations: Media centre. https://www.bbc.co.uk/mediacentre/latestnews/2015/bbc-virtual-voiceover-translations
Bywood, L. (2020). Technology and audiovisual translation. In ?. Bogucki & M. Deckert (Eds.), The Palgrave handbook of audiovisual translation and media accessibility (pp. 503–517). Palgrave. https://doi.org/10.1007/978-3-030-42105-2_25
carmeboix. (2021) Re: Comparación de mitosis y meiosis. [Video]. YouTube. https://www.youtube.com/watch?v=az1S54aMskI
Chaume, F. (2007). Dubbing practices in Europe: Localisation beats globalisation. Linguistica Antverpiensia, New Series – Themes in Translation Studies, 6, 203–217. https://doi.org/10.52034/lanstts.v6i.188
Chaume, F. (2020). Dubbing. In ?. Bogucki & M. Deckert (Eds.), The Palgrave handbook of audiovisual translation and media accessibility (pp. 103–132). Palgrave. https://doi.org/10.1007/978-3-030-42105-2_6
De Los Reyes Lozano, J. (2022). Traduction automatique et doublage: Impressions d’une expérience d’enseignement. Journal of Data Mining and Digital Humanities. eISSN 2416-5999. https://doi.org/10.46298/jdmdh.9119
Di Gangi, M., Rossenbach, N., Pérez, A., Bahar, P., Beck, E., Wilken, P., & Matusov, E. (2022). Automatic video dubbing at AppTek. Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, 351–352. https://aclanthology.org/2022.eamt-1.65
Di Giovanni, E., & Romero-Fresco, P. (2019). Are we all together across languages? An eye tracking study of original and dubbed films. In I. Ranzato & S. Zanotti (Eds.), Reassessing dubbing: Historical approaches and current trends (pp. 125–144). John Benjamins. https://doi.org/10.1075/btl.148.06di
Federici, F., Declerq, C., Díaz Cintas, J., & Baños Piñero, R. (2023). Ethics, automated processes, machine translation, and crises. In H. Moniz & C. Parra Escartín (Eds.), Towards responsible machine translation: Ethical and legal considerations in machine translation (pp. 135–156). Springer. https://doi.org/10.1007/978-3-031-14689-3_8
Federico, M., Enyedi, R., Barra-Chicote, R., Giri, R., Isik, U., Krishnaswamy, A., & Sawaf, H. (2020). From speech-to-speech translation to automatic dubbing. Proceedings of the 17th International Conference on Spoken Language Translation, 257–264. https://doi.org/10.18653/v1/2020.iwslt-1.31
Flawless (2022). TrueSync. https://flawlessai.com/product.
Franco, E., Matamala, A., & Orero, P. (2013). Voice-over translation: An overview. Peter Lang. https://doi.org/10.3726/978-3-0351-0630-5
Hayes, L. (2021). Netflix disrupting dubbing: English dubs and British accents. Journal of Audiovisual Translation, 4(1), 1–26. https://doi.org/10.47476/JAT.V4I1.2021.148
Karakanta, A., Bhattacharya, S., Nayak, S., Baumann, T., Negri, M., & Turchi, M. (2020). The two shades of dubbing in neural machine translation. Proceedings of the 28th International Conference on Computational Linguistics, 4327–4333. https://doi.org/10.18653/V1/2020.COLING-MAIN.382
Kings and Generals. (2022) Evolution of the Indo-European Languages - Ancient Civilizations DOCUMENTARY [Video]. YouTube. https://www.youtube.com/watch?v=VpXgMdvLUXw
Kings and Generals. (n.d.) Home [Video]. YouTube. https://www.youtube.com/c/KingsandGenerals/videos
Kottahachchi, B. & Abeysinghe, S. (2022). Overcoming the language barrier in videos with Aloud. https://blog.google/technology/area-120/aloud/
Lakew, S. M., Federico, M., Wang, Y., Hoang, C., Virkar, Y., Barra-Chicote, R., & Enyedi, R. (2021). Machine translation verbosity control for automatic dubbing. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing – Proceedings, 2021-June, 7538–7542. https://doi.org/10.48550/arxiv.2110.03847
laSexta (2022, July 2). La 'magia' del deepfake: Un experto en efectos especiales desvela cómo convierten a Dani Mateo y Wyoming en 'Aznarito y Felipón'. https://www.lasexta.com/programas/el-intermedio/experto-efectos-especiales-desvela-como-convierten-dani-mateo-wyoming-aznarito-felipon_2022070262be041203d1fb0001ef38b7.html
lucyhernandez7672. (2021). Re: Síntesis de proteínas. [Video]. YouTube. https://www.youtube.com/watch?v=6VIkfSvB02w
Macrea, C-I., Arias-Badia, B., & Torner Castells, S. (2023). El inglés biosanitario en YouTube: Recepción del uso de series y subtitulado en vídeos de aprendizaje. Panace@, 23(56), 69–83.
Martín-Mor, A., & Sánchez-Gijón, P. (2016). Machine translation and audiovisual products: A case study. The Journal of Specialised Translation, 26, 172–186. https://www.jostrans.org/issue26/art_martin.pdf
Matamala, A. (2015). The ALST project: Technologies for audiovisual translation. Proceedings of the 37th Conference Translation and the Computer-ASLING 2015, 79–89. https://aclanthology.org/2015.tc-1.11.pdf
Mejías-Climent, L. (2021). Enhancing video game localization through dubbing. Springer International. https://doi.org/10.1007/978-3-030-88292-1
montsezaragoza9132. (2021). Re: Síntesis de proteínas. [Video]. YouTube. https://www.youtube.com/watch?v=6VIkfSvB02w
Öktem, A., Farrús, M., & Bonafonte, A. (2019). Prosodic phrase alignment for machine dubbing. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH 2019, 4215–4219. https://doi.org/10.21437/INTERSPEECH.2019-1621
Pavesi, M., & Zamora, P. (2021). The reception of swearing in film dubbing: A cross-cultural case study. Perspectives, 30(3), 382–398. https://doi.org/10.1080/0907676X.2021.1913199
Saboo, A., & Baumann, T. (2019). Integration of dubbing constraints into machine translation. Proceedings of the 4th Conference on Machine Translation, 1, 94–101. https://doi.org/10.18653/V1/W19-5210
Slator. (2021). Slator 2021 video localization report. https://slator.com/slator-2021-video-localization-report/
Spiteri Miggiani, G. (2021). English-language dubbing: Challenges and quality standards of an emerging localisation trend. The Journal of Specialised Translation, 36a, 2–25. https://www.jostrans.org/issue36/art_spiteri.pdf
Spiteri Miggiani, G. (2022). Measuring quality in translation for dubbing: A quality assessment model proposal for trainers and stakeholders. XLinguae, 15(2), 85–102. https://doi.org/10.18355/XL.2022.15.02.07
Terantino, J. (2011). Emerging technologies. YouTube for foreign languages: You have to see this video. Language Learning & Technology, 15(1), 10–16. http://dx.doi.org/10125/44231
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Rocío Baños
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under the CC BY-NC 4.0 Deed that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal. The material cannot be used for commercial purposes.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
