Sony Aims to Improve AI Translation for Indian Language Entertainment Content

In an December 29, 2024 paper by Sony Research India researchers Pratik Rakesh Singh, Mohammadi Zaki, and Pankaj Wasnik comes a framework specifically designed to "improve entertainment content translations" in Indian languages.

They "believe it is the first of its kind," using an amalgamation of context awareness along with style adaptation to produce not only accurate translations but also entertaining for the targeted audience.

The researchers explained that traditional machine translation MT systems usually struggle to handle entertainment content because they mostly translate sentences in isolation. It leads to "disconnected" translations that can't really capture the emotional depth or cultural references behind the original dialogue. This has a particular pronounced effect in entertainment, where all these interconnected conversations and subtle cues in the narrative are so vital.

The challenge, in entertainment translation, lies in preserving the context, mood, and style of the original content while also including creativity and considerations of regional dialects, idioms, and other linguistic nuances," researchers explained.

To tackle this challenge, the researchers developed CASAT: the Context and Style Aware Translation, which combines the two concepts during the translation process.

The CASAT framework starts with segmenting the input text — like dialogues from movies or series — into smaller sections known as "sessions." Sessions are dialogues that are consistent in their genre or mood, such as comedy or drama. This segmentation allows CASAT to focus on the specific emotional and narrative elements of each session.

For every session, CASAT estimates two critical components: context and style. The former is said to be the narrative framework that wraps the dialogue, while the latter denotes the emotional tone and cultural nuances, like seriousness, excitement, or even humor. Understanding these, the framework will be able to make translations that effectively reach the deep recesses of the target audience's psyche.

To facilitate this, CASAT adopts a context retrieval module that gets relevant scenes or dialogues based on the relevant vector database retrieved, so this translation is grounded in appropriate narrative frameworks, and it applies a domain adaptation module to infer insights from sessions and sentences-based dialogues to realize the intended emotion tone and the intent.

Once the context and style are estimated, CASAT generates a customized prompt that is a combination of these elements. The customized prompt is then passed to an LLM that generates translations not only accurate but also carrying the intended emotional tone and cultural nuances of the original content.

Superior Performance

Metrics for CASAT's effectiveness, such as COMET scores and win ratios, have been used to test its performance. CASAT, on the other hand, surpassed baseline LLMs and MT systems like IndicTrans2 and NLLB, providing much better translations in terms of content and context.

"Our method exhibits superior performance by consistently incorporating plot and style information compared to directly prompting creativity in LLMs," the researchers said.

They found that context alone substantially improves translation quality, while including style alone has a minimal improvement. Combining the two improves quality the most.

The researchers noted that CASAT is language and model-agnostic. "Our method is both language and LLM-agnostic, making it a general-purpose tool," they concluded.