This is a thread on publications that study Machine Translation (mostly #NMT) and #subtitling. I am looking for studies that go beyond the word level and take the image into account.
#NMTsubtitling

"The models learn to distinguish between information from different segments & are surprisingly robust with respect to translation quality. Interesting cross-sentential attention patterns that improve textual coherence in
translation in selected cases". https://arxiv.org/pdf/1708.05943.pdf

Machine translation should not only produce semantically accurate translations, but should also consider pragmatic aspects, such as producing socially appropriate forms of address.>> http://www.aclweb.org/anthology/N16-1005

>>We currently assume that the desired level of politeness is specified by the user. Future work could aim to automatically predict it from the English
source text based on textual features such as titles and names, or meta-textual information about the discourse participants.

(Subtitle corpora are consistently used in MT research) "NMT is better for IT and Subtitles, SMT is better for Law, Medical, and Koran" -- "NMT models do not show robust behavior when confronted with conditions that differ significantly from training https://arxiv.org/pdf/1706.03872.pdf

(Another experiment with subtitles - laughing my heart out with this). "As turn taking is not clearly indicated, we treated consecutive sentences assuming they were uttered by different characters" https://arxiv.org/pdf/1506.05869.pdf

"For MT to tackle discourse phenomena, models must have access to extrasentential linguistic context. Models of context have been principally evaluated with standard automatic metrics, poorly adapted to evaluating discourse phenomena. https://arxiv.org/pdf/1711.00513.pdf

>>"Models are trained and tested on fan-produced parallel subtitles from OpenSubtitles2016" (so many researchers using it, nobody challenging its quality? Of course you'll get "; poor results for coreference and more promising
results for coherence/cohesion"

"Movie scripts span a wide range of topics and contain long dialogues with few participants, making them ideal for researching open domain, long
interaction dialogue systems. They are close to human spoken language" https://www.researchgate.net/profile/Alessandro_Sordoni/publication/280221106_Hierarchical_Neural_Network_Generative_Models_for_Movie_Dialogues/links/55b80c5e08aed621de05bc13/Hierarchical-Neural-Network-Generative-Models-for-Movie-Dialogues.pdf

And here is "A Survey of Available Corpora for Building Data-Driven Dialogue Systems" with lots of movie subtitles and scripts. Very interesting! https://arxiv.org/pdf/1512.05742.pdf

The EU-funded #SUMAT project promised an "An Online Service
for Subtitling by Machine Translation" which it never delivered (and nobody asked for EU money back?) We have research findings though, e.g. https://www.tandfonline.com/doi/abs/10.1080/0907676X.2013.831920

....which of course you have to pay to read! https://www.tandfonline.com/doi/full/10.1080/0907676X.2017.1291695?src=recsys

So far, I haven't found any research taking visual nonverbal context into account. I'll keep looking.

SMT model trained with fansubs: "Assessing the Post-Editing Effort for Automatic Translations of Subtitles". "Noise had to be cleaned such as: i) spelling errors, ii) nonuniform character casing..." (No translation errors cleaned)
http://www.aclweb.org/anthology/R11-1014

Expecting to find answers to my initial question at the presentation titled "Video Analysis Applications and Use Cases in the Media Industry" by Tommaso Cesano @MetaliquidAI http://symposium.omniscien.com/day-1/

Turns out they don't use it for linguistic purposes. Mainly for adding metadata to images and clips, face/brand recognition, highlight focus, segmentation, advertising purposes, etc

However, @omniscientech is planning to use AI and video analysis. By enriching metadata, more context is provided, which in turn helps to improve MT, e.g. by disambiguating terms and phrases

Long before NMT, one of the early attempts of MT in subtitling. Difficult to find MUSA project findings http://www.ilsp.gr/administrator/components/com_jresearch/files/publications/Infrastructure_for_a_multilingual_subtitle_generation_system-final.pdf

No conclusions or any mention on the resulting translation quality. Even the architecture might be useless now http://www.ilsp.gr/administrator/components/com_jresearch/files/publications/LREC2004-Subtitling%20Resources.pdf

Yet another study (MA thesis) using the #OpenSubtitles dataset. When will the giant content owners let researchers use the more reliable professional subtitles? http://Opensubtitles.org is mostly fansubs (lots of prof subs lately too)
#NMT #subtitling https://slator.com/academia/nmt-for-subtitling-could-work-without-specialized-training-study-finds/

A new study on a #NMT and #subtitling, highlighting the importance of subtitling constraints and the need to to preserve utterance duration in subtitling corpora - By
@karakanta et al
http://ceur-ws.org/Vol-2481/paper38.pdf

@karakanta This study shows that the corpus (TED talks) can be used to build models that efficiently segment sentences into #subtitles & proposes a method for annotating existing subtitling corpora with subtitle breaks (with constraint of length)
By @karakanta et al
https://arxiv.org/abs/2002.10829

A paper on Quality estimation for #subtitle translations https://arxiv.org/abs/2004.13828
"A good translation retains all meaning and reads fluently; A bad one bears no resemblance to the meaning of the original and is disconnected from the context in the video; 1/2

"A loose translation is one that uses paraphrasing, colloquialism, idiom, or contains some contextual information not available in [the] source text.”
https://slator.com/machine-translation/good-bad-or-loose-amazons-new-subtitle-quality-estimation-system/ 2/2

"Simplify-then-Translate: Automatic Preprocessing for Black-Box Machine Translation"
A method that could be used in subtitling into low-resource languages https://arxiv.org/abs/2005.11197
And an overview by Slator https://slator.com/machine-translation/how-netflix-researchers-simplify-subtitles-for-translation/

Is there any MT research paper using http://OpenSubtitles.org that addresses its shortcomings? I know it's a conveniently aligned dataset but conclusions based on it need to be set on the right context (e.g. mix of fansubs and professional "stolen" subtitles).
@karakanta?

Lison at al (2016) describe how to use OpenSubtitles to extract large parallel corpora. No mention of how OpenSubtitles is composed, apart from a mention that part of it is OCR from video streams https://www.aclweb.org/anthology/L16-1147.pdf

Koponen et al (2020) mention that OpenSubtitles "comes mostly from volunteers".
Interesting study but very limited scope: 12 professional subtitlers translating six 3min clips from European election debates & semi-scripted lifestyle/cultural programmes
https://eamt2020.inesc-id.pt/proceedings-eamt2020.pdf

Another overview of Netflix's Automated Preprocessing (APP) model with examples in Hindi.
"Human reference translations when back-translated to the original language is a rich source of simplifications (e.g. “jump in” is simplified to “take part”)"
https://analyticsindiamag.com/netflix-ai-subtitles-transalation/

A multilingual audiovisual corpus for #speechrecognition and #speechtotext translation providing 1200 hours of AV speech in 9 languages. Fully transcribed, the first and the largest open
benchmark for multilingual AV speech recognition https://arxiv.org/abs/2303.00628

Problems with automating translation of movie/TV show subtitles - related to:
👉subtitle guidelines (KNPs, spacing, symbols, line breaks)
👉subtitle translation (condensing/paraphrasing, idioms, context, profanity, omissions) etc.
https://arxiv.org/abs/1909.05362