Our data is released under various Creative Commons licenses.More information
If you love this content, please consider a donation.
Wall (7 215 threads)
Patarimai
Before asking a question, make sure to read the FAQ.
We aim to maintain a healthy atmosphere for civilized discussions. Please read our rules against bad behavior.
gillux
1 days ago
gillux
1 days ago
LeviHighway
1 days ago
EugeneGS
2 days ago
Ooneykcall
2 days ago
LeviHighway
2 days ago
frpzzd
2 days ago
sharptoothed
2 days ago
EugeneGS
2 days ago
Thanuir
2 days ago
The content of this message goes against our rules and was therefore hidden. It is displayed only to admins and to the author of the message.
The content of this message goes against our rules and was therefore hidden. It is displayed only to admins and to the author of the message.
The content of this message goes against our rules and was therefore hidden. It is displayed only to admins and to the author of the message.
The content of this message goes against our rules and was therefore hidden. It is displayed only to admins and to the author of the message.
I wish the automatically generated traditional/simplified Chinese could be editable, because it sometimes isn't correct. I as an advanced contributor cannot edit them, I'm not sure if corpus maintainers can? it's not written on the wiki.
There are plans to make the traditional/simplified Chinese script editable. You can follow the progress here https://github.com/Tatoeba/tatoeba2/issues/2007
On dirait que le fonctionnement des langues par défaut, pour les phrases insérées, a changé.
J'ai beau sélectionner "détection automatique", toutes les phrases que j'insère en anglais sont immédiatement identifiées comme des phrases en français, ce qui est parfaitement stupide.
Rien n’a changé à ce niveau, si ce n’est que le modèle sur lequel s’appuie la détection des langues est mis à jour chaque semaine sur la base du corpus de Tatoeba (modulo les phrases étiquetées @wrong flag). Le modèle n’est jamais été parfait, notamment sur les phrases courtes.
Can I learn more about the model? When I add Mandarin sentences, the model always detect it to be Cantonese. I know Mandarin and Cantonese are extremely close, so I never use the Detect function at all.
Jos sinulla on isompi ja pienempi kieli jotka ovat hyvin samankaltaisia, ja lisäät lauseen pienempään, saattaa se olla algoritmin mielestä lähempänä isomman kielen lauseita.
Jos lauseessa on pienemmän kielen erityispiirteitä (joita suuremmassa ei ole), näin tapahtuu harvemmin.
Maybe there's also something wrong with the model architecture. I trained a few models myself — one on all Tatoeba data and one only on Mandarin and Cantonese — and both correctly detected about 97% of cases (checked on validation and full datasets).
What's strange is that the Tatoeba model seems to prefer Cantonese, even though it has fewer sentences than Mandarin.
Edit: I have tried another architecture with transformer layers (my first models had LSTM layers). After training on whole Tatoeba database it gave 82% accuracy.
Is your model training/testing code available online anywhere? If so, I would love to take a look for my own edification, since I've been learning about such topics recently.
I've uploaded it on GitHub. The code can be used for pretty much any text classification task.
I honestly didn't expect anyone to be interested, so I'm glad you asked! Some comments in the code might not be super helpful, but if anything's unclear, feel free to reach out via private messages.
https://github.com/kilsense/Tex...2f07/main/LSTM
lol I correct myself, it's not *always* Cantonese, but it's pretty frequent. I noticed that most Cantonese sentences on Tatoeba are very long sentences, I guess that affected the model.
I've noticed there are some weird accounts adding many, usually long, Cantonese sentences often as translations from other languages including Russian (that's why I noticed it), whose quality I suspect is questionable, but unfortunately there are no active native speakers of Cantonese at the moment that could be dealing with that.
Now you mention it, there could be a bias related to traditional/simplified characters. The model only considers the sentence script, not the autogenerated alternative script. As for Mandarin Chinese, 57% of sentences use simplified characters and 43% use traditional, while Cantonese only uses traditional.
Anyway, the language detector on Tatoeba is based on ngrams statistics, which is very old school compared to the technology available nowadays like transformers. Anybody is welcome to improve or even rewrite it https://github.com/Tatoeba/Tatodetect
The content of this message goes against our rules and was therefore hidden. It is displayed only to admins and to the author of the message.
The content of this message goes against our rules and was therefore hidden. It is displayed only to admins and to the author of the message.
✹✹ Stats & Graphs ✹✹
Tatoeba Stats, Graphs & Charts have been updated:
https://tatoeba.j-langtools.com/allstats/
The content of this message goes against our rules and was therefore hidden. It is displayed only to admins and to the author of the message.