ساݙا ڈیٹا مختلف کریٹیو کامنز لائسنساں دے تحت جاری کیتا ویندا ہے۔ٻیاں معلومات
جے تہاکوں ایں مواد نال پیار ہے تاں، سوہݨا، عطیہ ݙیوو۔
وال (ہک تند)
گُر
Before asking a question, make sure to read the FAQ.
We aim to maintain a healthy atmosphere for civilized discussions. Please read our rules against bad behavior.
gillux
ہک منٹ پہلے
gillux
ہک منٹ پہلے
LeviHighway
ہک گھنٹہ پہلے
EugeneGS
کل
Ooneykcall
کل
LeviHighway
کل
frpzzd
کل
sharptoothed
کل
EugeneGS
کل
Thanuir
کل
I wish the automatically generated traditional/simplified Chinese could be editable, because it sometimes isn't correct. I as an advanced contributor cannot edit them, I'm not sure if corpus maintainers can? it's not written on the wiki.
There are plans to make the traditional/simplified Chinese script editable. You can follow the progress here https://github.com/Tatoeba/tatoeba2/issues/2007
On dirait que le fonctionnement des langues par défaut, pour les phrases insérées, a changé.
J'ai beau sélectionner "détection automatique", toutes les phrases que j'insère en anglais sont immédiatement identifiées comme des phrases en français, ce qui est parfaitement stupide.
Rien n’a changé à ce niveau, si ce n’est que le modèle sur lequel s’appuie la détection des langues est mis à jour chaque semaine sur la base du corpus de Tatoeba (modulo les phrases étiquetées @wrong flag). Le modèle n’est jamais été parfait, notamment sur les phrases courtes.
Can I learn more about the model? When I add Mandarin sentences, the model always detect it to be Cantonese. I know Mandarin and Cantonese are extremely close, so I never use the Detect function at all.
Jos sinulla on isompi ja pienempi kieli jotka ovat hyvin samankaltaisia, ja lisäät lauseen pienempään, saattaa se olla algoritmin mielestä lähempänä isomman kielen lauseita.
Jos lauseessa on pienemmän kielen erityispiirteitä (joita suuremmassa ei ole), näin tapahtuu harvemmin.
Maybe there's also something wrong with the model architecture. I trained a few models myself — one on all Tatoeba data and one only on Mandarin and Cantonese — and both correctly detected about 97% of cases (checked on validation and full datasets).
What's strange is that the Tatoeba model seems to prefer Cantonese, even though it has fewer sentences than Mandarin.
Edit: I have tried another architecture with transformer layers (my first models had LSTM layers). After training on whole Tatoeba database it gave 82% accuracy.
Is your model training/testing code available online anywhere? If so, I would love to take a look for my own edification, since I've been learning about such topics recently.
I've uploaded it on GitHub. The code can be used for pretty much any text classification task.
I honestly didn't expect anyone to be interested, so I'm glad you asked! Some comments in the code might not be super helpful, but if anything's unclear, feel free to reach out via private messages.
https://github.com/kilsense/Tex...2f07/main/LSTM
lol I correct myself, it's not *always* Cantonese, but it's pretty frequent. I noticed that most Cantonese sentences on Tatoeba are very long sentences, I guess that affected the model.
I've noticed there are some weird accounts adding many, usually long, Cantonese sentences often as translations from other languages including Russian (that's why I noticed it), whose quality I suspect is questionable, but unfortunately there are no active native speakers of Cantonese at the moment that could be dealing with that.
Now you mention it, there could be a bias related to traditional/simplified characters. The model only considers the sentence script, not the autogenerated alternative script. As for Mandarin Chinese, 57% of sentences use simplified characters and 43% use traditional, while Cantonese only uses traditional.
Anyway, the language detector on Tatoeba is based on ngrams statistics, which is very old school compared to the technology available nowadays like transformers. Anybody is welcome to improve or even rewrite it https://github.com/Tatoeba/Tatodetect
The content of this message goes against our rules and was therefore hidden. It is displayed only to admins and to the author of the message.
The content of this message goes against our rules and was therefore hidden. It is displayed only to admins and to the author of the message.
✹✹ Stats & Graphs ✹✹
Tatoeba Stats, Graphs & Charts have been updated:
https://tatoeba.j-langtools.com/allstats/
The content of this message goes against our rules and was therefore hidden. It is displayed only to admins and to the author of the message.
Please enable Traditional-Simplified convertion to Literary Chinese. Currently, people are contributing in either Traditional or Simplified characters. So I think it needs convertion just like Mandarin Chinese.
Does anyone know any website that is similar to the Tatoeba mechanism but is for vocabularies? I know Glosbe but it seems they doesn't ensure quality at all.
I second this question. I've also seen Glosbe but haven't contributed to it myself because (1) it is not open source, and (2) it does not (as far as I'm aware) allow for bulk data download.
This is probably not what you want because it is mainly between German and other European languages, but I like dict.cc because they allow you to download the dictionary data in its entirety (but it must be requested by email).
https://www.dict.cc
And of course, there is always Wikitionary.
What do you have in mind exactly with "ensuring quality"? Even here on Tatoeba there seems to be quite a bit of debate sometimes when it comes to correcting sentences.
well Glosbe does not have a community/comment function, and I never managed to contact their staff. it's not organized at all, all contributions are kept so it's a total mess. you have no control of anything, your contribution might be hidden at the bottom etc. Tatoeba is much better. I don't like (Chinese) Wiktionary, because it's so complicated, contributing to one entry usually takes a a day.
The content of this message goes against our rules and was therefore hidden. It is displayed only to admins and to the author of the message.
the only Chinese corpse maintainer is inactive. how should we deal with hundreds of Chinese sentences that need to be changed?
https://tatoeba.org/zh-cn/tags/...direction=desc