Can TF-IDF be used as a machine translation loss function?

pasta_paul

TF-IDF cosine similarity is a powerful means of determining the similarity of two text documents . Can it be used as a loss function in a machine translation model training? For example, when measuring the goodness of possible translations, cosine TF-IDF similarity ("distance") could be computed between the pairs, using some intermediate "language of thought" / TF-IDF "space" (e.g., based on Stanza 's morphological analysis or POS tagging ): 🇬🇧...

Originally asked by Geremia on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

2 options: Since the TF‑IDF cosine distance is not differentiable, use an optimization method that does not require a differentiable objective , e.g. see the paper Minimum Risk Training for Neural Machine Translation . Build a differentiable surrogate that resembles TF‑IDF.

Full answer by Franck Dernoncourt: https://ai.stackexchange.com/a/50501 (CC BY-SA 4.0).

Need it done for your business? UphillTech.