I think it should prioritise the user's "usual" languages less than it apparently does at the moment. In the end, one could still try to write some sentences in languages he has never added one. The habits should only help to choose in case of ambiguity, rather than be so determinant to the result, in my opinion.

everytime. english isn't the only language using the latin alphabet.

i will try to add a bulgarian sentence (for testing) if it detects the sentence is russian, i will report the bug.

Bedankt, Sysko! Het werkt heel goed.
Dankon, Sysko! Ĝi bonege funkcias.
Merci, Sysko! Cela marche tres bien.

great, actually it works thanks to all the contributions so we have all contribute to make it in an a way or an other :)

The goal is to make all these "non-vital" and "not hardly coupled to the core feature of adding/removing/editing sentences/links" features handled by standalone services for several reason:
1 - they can be reused by other projects (for the example the autocompletion you have on tags is a "standalone" server too, written in C by a friend of mine that use it for autocompletion of username etc.)
2 - in case of data corruption crash or whatsoever, that will not perturb the main service
3 - I can stop some services in case of nasty bugs (security hole, possible data corruption) or in case of high load on the server in order to focus on the main features , in the case i also "standalone" the wall , that will permit to still have the wall when i get some problems on the main features
4 - that can permit a simple "scalability" (if in the future we have several servers, we can put one with the core, one with the wall and language detection, an other with the autocompletion features etc.
5 - that can maybe easier to split the work on several people

after of course that maybe duplicate the code, so if i start to really generalize this process, maybe i will need a little more highlevel layer library upon cppcms ?

(I know my last points are not related to your questions, but as I'm not a "web architect" expert or whatsoever, I wanted for a long time to start a discussion on this with people with more experiences about my previous points.

actually the language detection is made by a separate service that run on a different server, so Tatoeba simply "connect" to it by doing a simple curl call.

no for two reasons
1 - I don't know have a "dictionnary" for every languages (especially for language like berbere , shanghainese, etc.)
2 - and I can not admit the space separator to "split" words as you know that will not work for Japanese, Chinese etc.

Moreover that will require using other data sources, which are not that easy to found under a "open" licence for a lot of languages. And that would have require maybe doing some "one language"-specific code

Instead I use a quite naive and simple method which is to cut the sentences in group of 2,3,4 and 5 characters (regardless if they are punctuations or so)

for example "Bonjour[narrow non break space]!" for the "5 by 5" table will be

Bonjo
onjou
njour
jour[non breakable space]
our[non-breakable]!

for each couple (language,5-gram) I keep two numbers the absolute number of times it appears in Tatoeba and the relative frequency in that language
(same for 4-grams,3-grams,2-grams)
I delete the couples that appear too fews times as they may come from "tricky sentence" (for example like a sentence that say
"你好" is how we say hello in Chinese" )

after this when you enter a sentence

this sentence is first splitted in group of 5 letters the same way as decribe above
I search all the languages that contain the first group

in a "absolute" table i keep the absolute score of this 5-gram in all the language it appears
in a relative table i keep the relative score
if this 5-gram appears in only one language (the reason why i delete those who appears too few), the absolute and relative score for this given 5-gram is multiplied by a bonus as a "unique" 5-gram is more relevant

we continue for each 5-gram, summings the score

after we get the two first languages in both score , and we keep the "1st language" that have more "distance" with the 2nd of its table

so that for example at the end if we have something like that


absolute table

1st: Mandarin: 10000
2st: Cantonese: 9800
[...]
nth: Teochew 500 (because we have really few sentence in this language)

relative score

1st Teochew : 0,50
2nd Mandarin: 0,01
[...]

We will detect it as Teochew

This way we both keep in account that some language are more likely to appears than other (an unknown user is more likely to contribute in German than in Low-saxon for example) but still give chance to "rare" languages

moreover if it is contributed by a known-user, we don't keep in the table the languages in which this user does not contribute (so someone who contribute only in Chinese and Teochew will have no chance to get a sentence detected as Cantonese) (except if he start contribute a lot in Cantonese, then at the next refresh of the database, he will be considered as also contributing in Cantonese) So that the system is not "fixed" and will adapt itself with times (though not in a clever way as Jakov described)

for the moment the algorithm is really naive and no "learning" yet (away from using the new dump every week), my first goal was to get quickly something to replace the google API

I'm have some notions of machine learning, but I'm not familiar enough to quickly integrate it, but I think yes in the future we can imagine that a bad detection decrease the score of the n-gram of the "bad language" contained in the sentence, and increase the score for the language set by the user

I will try to put in the wiki page of the project the raw algorithm so that after everyone with computer science knowledge would be able to propose improvement as I don't pretend to be of any expertise on this field.

The purpose is to automatically detect the language of the sentence you enter in Tatoeba, rather than to select which language it is on your own. It is very helpful to Tatoeba because many people forget to change the flag of their sentences to the right one, when they contribute in multiple languages.
To solve this, you select "automatic detection" in the languages list for your sentence, once (afterwards, that choice is automatically retained), and here you go, spoiled child...
Furthermore, sysko designed it as an API that can be used from anywhere to detect the language of any sentence, based on Tatoeba sentences statistics...
That means Tatoeba's corpus might be used as a base for language detection for other services.

http://tatoeba.org/fre/sentences/show/1517450