Single-side deduplication

[ZJaume]

Some experiments that a colleague did during MaCoCu project, found that deduplication taking into account only source side or target side, improved translation quality. IIRC it was not clear what was better, to do it on the source or on the target, but both were better than deduplicating In some cases I think it was about 1 BLEU point for mid-resource languages. This probably reduces the amount of translation inconsistencies.

I couldn't found the table with the results, but I think this is worth exploring.

Maybe you are already doing this, but I was not sure. At least in the old pipeline dedupe is using the whole sentence pair.

[gregtatum]

We are de-duplicating based on source and target.

[eu9ene]

I was investigating another issue and I saw this for en-zh tokenized corpus:

Natural ▁ compound

Natural ▁ products

Natural ▁ product

Natural ▁ Products

correspond to

天然 产物

天然 产物

天然 产物

天然 产物

Google translate translates 天然 产物 as Natural Products.

So probably when we train en-zh it's ok to leave it, but for zh-en it would make sense to do source-side deduplication, otherwise we have 4 different translations for the same Chinese phrase. The question here is which translation is correct... It would make sense to run some model to score each of them and pick the best one instead of naive deduplication.

[ZJaume]

Picking the one with best BCAI score?

天然 产物       Natural compund 0.548
天然 产物       Natural products        0.889
天然 产物       Natural product 0.881
天然 产物       Natural Products        0.901

[gregtatum]

@ZJaume What is the BCAI score?

[ZJaume]

Bicleaner AI, sorry 😅