That is indeed the 80% solution, but it's really not that simple to get good results. First of all, Jieba as a segmenter has a very high error rate, so I switched to transformer based one which is better, but still quite bad (it's a hard problem). You can mitigate some of this by joining long words that are in CEDICT (but do not cross other word borders!), but this often backfires.
As for CEDICT entries, very often the entry translation does not exist in the sentence translation. Sometimes it exists with a morphological change like "fire -> fired", sometimes more difficult "go -> went". Sometimes it's a synonym which you may be able to find in a thesaurus, maybe not. The thesaurus is also incomplete or has errors. If you use a synonym you often get false positives from completely unrelated hanzi's in the sentence. Sometimes the CEDICT word has 8 different entries and some of them are rare meanings with low frequency words that match many sentences but really shouldn't, how do you detect that? Entries are not ordered by importance. Weight by specificity? Ok, but each entry has an average of 5 words, what if one entry matches 4 high frequency (low information) words but another matches 1 low frequency word? What if multiple words in the sentence has overlapping target words in the translation, which is which?
And now you have words with different pinyin readings, so multiple completely different entries in CEDICT. So you try to first classify the pinyin (or the other way around, use translation correlation to classify pinyin?). But if the initial segmentation is bad, you've got garbage in, garbage out anyway.
Then you have lots of two-character Chinese words where one is regularly omitted, like in my example if you look up 裁 in CEDICT you get "cut", but it's referring to 裁员 which is more like layoff or fired. "Cut" may be acceptable here, but there are many cases where if you do this you get completely nonsensical translations.
Now you need to match names as well. What if names are not capitalized? If they are capitalized, should you match them in order they're mentioned if there are multiple names in the sentence? Ok, so most Chinese names will be in some form of romanization, so you look for something that looks like the Pinyin of the word. Oh, but sometimes it's Wade-Giles isn't it? Oh, and because the translation you have comes from machine translation, now sometimes you get Korean or Japanese names for some reason, try and match those too?
In order to get more variety of word translations in context, I worked out something similar, where I give a machine translator the text "你好吗, 1. 你 2. 好 3. 吗", and often it will give in-context translations that are better, but often fails miserably as well. Now you've got to try to combine these outputs with CEDICT, and perhaps multiple sentence translations (human + machine?) to get the best possible result.
I also tried joint optimization over segmentation, pinyin and word disambiguation, by maximizing overlap and low frequency mappings (high information), but never managed to get it working well enough.
This is just a third of the problems and edge-cases you hit when trying to do this, I hope I've illustrated how much of a mess this becomes. A simple solution might work well enough for personal use as in your case, but I'm running through the scripts of TV-shows and displaying intelligent subtitles, so every error is quite jarring to the end-user. So being able to replace it with a ChatGPT prompt is quite the improvement! Because honestly, doing all this manual algorithm work can be an extremely frustrating process :)
Comments
That is indeed the 80% solution, but it's really not that simple to get good results. First of all, Jieba as a segmenter has a very high error rate, so I switched to transformer based one which is better, but still quite bad (it's a hard problem). You can mitigate some of this by joining long words that are in CEDICT (but do not cross other word borders!), but this often backfires.
As for CEDICT entries, very often the entry translation does not exist in the sentence translation. Sometimes it exists with a morphological change like "fire -> fired", sometimes more difficult "go -> went". Sometimes it's a synonym which you may be able to find in a thesaurus, maybe not. The thesaurus is also incomplete or has errors. If you use a synonym you often get false positives from completely unrelated hanzi's in the sentence. Sometimes the CEDICT word has 8 different entries and some of them are rare meanings with low frequency words that match many sentences but really shouldn't, how do you detect that? Entries are not ordered by importance. Weight by specificity? Ok, but each entry has an average of 5 words, what if one entry matches 4 high frequency (low information) words but another matches 1 low frequency word? What if multiple words in the sentence has overlapping target words in the translation, which is which?
And now you have words with different pinyin readings, so multiple completely different entries in CEDICT. So you try to first classify the pinyin (or the other way around, use translation correlation to classify pinyin?). But if the initial segmentation is bad, you've got garbage in, garbage out anyway.
Then you have lots of two-character Chinese words where one is regularly omitted, like in my example if you look up 裁 in CEDICT you get "cut", but it's referring to 裁员 which is more like layoff or fired. "Cut" may be acceptable here, but there are many cases where if you do this you get completely nonsensical translations.
Now you need to match names as well. What if names are not capitalized? If they are capitalized, should you match them in order they're mentioned if there are multiple names in the sentence? Ok, so most Chinese names will be in some form of romanization, so you look for something that looks like the Pinyin of the word. Oh, but sometimes it's Wade-Giles isn't it? Oh, and because the translation you have comes from machine translation, now sometimes you get Korean or Japanese names for some reason, try and match those too?
In order to get more variety of word translations in context, I worked out something similar, where I give a machine translator the text "你好吗, 1. 你 2. 好 3. 吗", and often it will give in-context translations that are better, but often fails miserably as well. Now you've got to try to combine these outputs with CEDICT, and perhaps multiple sentence translations (human + machine?) to get the best possible result.
I also tried joint optimization over segmentation, pinyin and word disambiguation, by maximizing overlap and low frequency mappings (high information), but never managed to get it working well enough.
This is just a third of the problems and edge-cases you hit when trying to do this, I hope I've illustrated how much of a mess this becomes. A simple solution might work well enough for personal use as in your case, but I'm running through the scripts of TV-shows and displaying intelligent subtitles, so every error is quite jarring to the end-user. So being able to replace it with a ChatGPT prompt is quite the improvement! Because honestly, doing all this manual algorithm work can be an extremely frustrating process :)