Noun Keyword not translated properly

[vipin9457]

I am using the python library dl-translate 0.2.6 version and i am facing the issue while translating the Any Noun.
translation from ENGLISH to HINDI
Scenario 1 :
Input :

text_hi="What is your name ?"
mt.translate(text_hi, source=dlt.lang.ENGLISH, target=dlt.lang.HINDI)
Output
'आपका नाम क्या है?'
Result : Correct

Scenario 2 :
Input :

text_hi="What is your name ?"
mt.translate(text_hi, source=dlt.lang.ENGLISH, target=dlt.lang.HINDI)
Output :
'विपिन कुमर ट्राइपाथी'
Result : Not Correct
Expected Result : 'विपिन कुमार त्रिपाठी'

Scenario 3 :
Input :

text_hi="Piyush Mishra"
mt.translate(text_hi, source=dlt.lang.ENGLISH, target=dlt.lang.HINDI)
Output
'मिश्रा मिश्रा'
Result : Not Correct
Expected Result : 'पीयूष मिश्रा'

Scenario 3 :
Input

text_hi="Banana"
mt.translate(text_hi, source=dlt.lang.ENGLISH, target=dlt.lang.HINDI)
Output
'बादाम'
Result : Not Correct
Expected Result : 'केला'

Scenario 4 :
input :

text_hi="Apple"
mt.translate(text_hi, source=dlt.lang.ENGLISH, target=dlt.lang.HINDI)
Output
'एप्पल'
Result : Not Correct
Expected Result : 'सेब'

@xhluca @bilunsun Please help us to resolve this issue

[xhluca]

@vipin9457 The problem is with the underlying model, which is by default M2M100. Since it's a neural network, although it is good, it still fails in various scenarios. A few solutions:

Alternative models

You can try a larger variant of the model, or use the MBart50 model instead. If you are wondering which model is better for your use case, I recommend taking a look at the papers in the Acknowledgement section of the README, or check out their Huggingface page.

Parameters

You can play with the underlying parameters, see this for more information: https://xhluca.github.io/dl-translate/#interacting-with-underlying-model-and-tokenizer

Manual translations

If you want those translations to be manually stored, you can save them into a json, where the key is the source text and the key is the target text. You can load it and apply a conditional:

import json
import dl_translate as dlt

mt = dl.TranslationModel(...)
custom_translations_en_hi =_{
    "Apple": "सेब"
    ...
}

def custom_translate(text, source, target):
    if text in custom_translations_en_hi:
        return custom_translations_en_hi[text]
    else:
        return mt.translate(...)

Finetuning the model

If you want the model to generally learn to correctly translate a range of terms in various settings, you should finetune the translation model on more translation pair. Note that by doing this, the model might fail in other scenarios, such as a different language pair, or even within the source-target language you are interested in.

You can follow this tutorial for finetuning: https://huggingface.co/docs/transformers/tasks/translation
Just replace their model with M2M100.

Once you are done, you can load that custom model in dl-translate: https://xhluca.github.io/dl-translate/#changing-the-model-you-are-loading

PS: I've moved this to discussion since it's not an issue with dl-translate's internal code, but with the choice of model (default is M2M100).