---- Proposal for a Tatoeba Rating System ----

I apologize for the wall of text that is to come, but I’ve wanted to write something semi-formal considering the issue of quality control on Tatoeba for some time now, which, in my opinion, is present but not to the extent necessary for a site that now has over 2 million sentences in over 100 languages. The basic idea is a rating system, which I will present by first arguing for why Tatoeba needs one, giving some concrete ideas about how one could be implemented, and then finishing by giving the immediate benefits of such a system as well as potential drawbacks.

So…

1) Does Tatoeba need ratings for its sentences?

I think since the inception of Tatoeba the answer has been “yes”, as I remember the original guide by Trang saying that “owning” sentences was a good way to ensure quality but that ultimately ratings would be needed, although this was never implemented. Ratings have been discussed several times in the past on the Tatoeba wall/sentences, although no definitive conclusion has ever been reached and, again, nothing implementable was proposed. Current practices for ensuring good sentences and translations include:

- OK tags (as originally started by CK for English)
- Other tags (e.g. needs native check, translation check, change) to point out potential mistakes
- Comments and discussions
- Encouraging users to translate into their native language only
- Encouraging users to state what languages they know and how well they know them in their profiles

Although all of these work on a small scale, they are not able to cover the full 2 million sentences and who-knows-how-many translation links. Furthermore, these are subjective – a tag is only as reliable as the user, and comments, though sometimes reaching consensus among several native speakers, sometimes do not. The second-to-last point is a bit of a debate, since many users like to practice translating into languages that they haven’t mastered (myself being one), and some new users simply aren’t aware that translating into their native language is encouraged. The last point is also subjective and not something that is done by all users.

My overall feeling is that Tatoeba remains unreliable. Or rather, it is reliable for *me*, because I’ve been here for long enough to know who the trustworthy users are (and they are not always Advanced Users and Corpus Maintainers), and will treat their sentences/translations as reliable and take the rest with a grain of salt. The new user who just joins Tatoeba has no idea, however. Someone who is not a user and simply wants to use Tatoeba for a quick reference will also have no idea regarding whom to trust. A number on a sentence that would convey some sort of statistical reliability would be worth a lot.

2) How would a good rating system function?

Not sentence-by-sentence. Back in the early days (2-3 years ago), this was the major criticism of implementing a rating system – there were simply too many sentences, and it was unrealistic to expect multiple users to rate each one of them for quality, as well as all of the translations. It’s even less realistic now, since Tatoeba has grown a bit in sentence quantity but not that much in the number of active users.

However, it is this last point that can be exploited, as Tatoeba succumbs in no small way to Pareto’s rule, with 80% of the contributions being the work of 20% of the contributors. Actually, the former might be larger and the latter smaller. Some days it feels like there are just 10-20 users contributing 95-99% of the sentences. This makes rating all sentences and translations possible simply by rating the users, who are a lot fewer in number.

So, what’s a good scheme for rating users? Let’s just say that the ratings will be between 0 (unreliable) and 100 (full confidence). It’s probably not good to have one such rating, since a user could be really good at writing, e.g., sentences in French (100), but be horrible when it comes to Mandarin (0), and lumping those into a score of 50 is misleading. There should be a rating per user per language. Additionally, there needs to be a distinction between writing natural sentences and writing translations. As such, a user should have a rating for every possible language pair as well (at the moment this would be 119*118/2 pairs). As the number of languages grows, these numbers will increase, but storage should be feasible, especially since any rating set will be extremely sparse (i.e. most users only translate between, let’s say, 10 language pairs at most, and not 119*118/2).

For the actual implementation, it would be sufficient to have a thumbs up/down for every sentence that has an owner (to rate how natural it is) and a thumbs up/down for every link (to rate the translation). I would recommend using a weighted rating algorithm that would work like this:

i) New User gets 0 positive points and 100 total points upon joining. Their rating is 100*(positive/total) = 0.

ii) If their sentence/translation gets rated by User A (who has a rating of X for the *same* sentence/translation type) and the rating is positive, then New User’s new rating for that specific type is now 100*(positive+X)/(total+X). If the rating is negative, the new score is 100*(positive)/(total+X).

Here’s an example for User N as rated for the naturalness of his English sentences:

User N joins (positive = 0, total = 100, rating = 0). User N writes an English sentence. User A (whose has an English rating of 50) rates the sentence positively. User N is now: positive = 50, total = 150, rating = 33.3. User B, who has an English rating of 10, rates the sentence negatively. User N is now at: positive = 50, total = 160, rating = 31.25. If User N now adds a French translation and links the two, they have three different ratings (one for the naturalness of their English sentences, one for the naturalness of their French sentences, and one for the quality of their English-French translations).

Starting with 100 total points is a way to prevent a user from jumping up to a rating of 100 right away with a single good rating, and the weighting makes it possible to give more power to users who have already been recognized as trustworthy with respect to particular languages or language pairs. Of course, we couldn’t initialize everyone with a rating of 0 since we wouldn’t go anywhere. So, I would propose picking a single representative for each language / language pair that is trusted based on the community’s experience and giving them a starting rating of 50 for that language / language pair.

3) Benefits and potential drawbacks

i) Every sentence and translation that has an owner would simply inherit the owner’s score, thereby immediately providing a 0-100 number that would tell the naïve visitor if the sentence/translation was trustworthy. This would give some indicator of the reliability of the majority of sentences/translations. The drawback here is that one still needs to interpret the score somehow (i.e. a score of 86 doesn’t have any concrete meaning – it doesn’t mean that there is an 86% chance of the sentence/translation being “right”).

ii) Rogue contributors (who put up bad translations for fun) or new members who didn’t read or don’t care about the rules will be handled very efficiently. Instead of having to correct all of their added sentences or, in some rare cases, to even block such users’ access, it is sufficient for a few reliable users to downrate a few of their bad sentences to make it clear that these sentences are not trustworthy. In fact, not rating up is sufficient, since the rating of any new user would be 0 by default.

iii) Arguments would hopefully lessen in their quantity/intensity as well if there’s a “neutral” rating system in place that’s somehow based on collective opinion. Well, I hope. This may or may not happen.

iv) The debate over whether or not to translate only into your native language would disappear, since bad translations would just reflect badly on the user (either discouraging them from translating or simply accepting the fact that their translations in their non-native languages will not be viewed as reliable).

v) A certain drawback is that such a system would not really do much for languages or language pairs with very few contributors (due to the monopoly on those languages by the few contributors), but I don’t see how one could really solve this problem without getting more contributors…

vi) Storage and more database calls could be a drawback, but I don’t see this as being that bad. In the worst case, you can imagine Tatoeba with 1,000,000 users and 6,000 languages, which would result in 6,000,000,000 (naturalness) + (6,000*5,999/2)*1,000,000 (translation) total ratings to track. That’s a lot, but that’s the worst case and will never be achieved (as many users only work with a handful of languages, not 6,000).

Anyway, that’s all I’ve got. Just wanted to put it out there as an idea, since I think it could solve a number of issues that Tatoeba has with respect to the reliability of its sentences (as well as other things). Might be worth trying, and it would be easy to remove if it doesn’t work. I suggest a big discussion on this regardless, since this appears to be a major problem with Tatoeba at the moment.