To continue the discussion from these threads:

https://tatoeba.org/en/wall/sho...#message_39333
https://tatoeba.org/en/wall/sho...#message_39361
https://tatoeba.org/en/wall/sho...#message_39380
https://tatoeba.org/en/wall/sho...#message_39395

Humans need variety and context in order to understand and learn language. A single word may have several meanings. It may be used with some words but not others. It may have a level of formality that makes it only suitable for certain situations. If these nuances are to be induced from a collection of sentences, the collection must be sufficiently diverse. In addition, the human mind needs constant stimulation. Duplicate or near-duplicate sentences create boredom, and a bored mind refuses to learn.

The set of sentences "They learned some Greek while they were on vacation", "Rima and Skura learned valuable life lessons the hard way", and "Our mother, we learned later, had been saving money in a separate account" demonstrate different kinds of things that can be learned (in addition to teaching additional vocabulary). By contrast, the set of sentences "<X> learned Berber", "<Y> learned Berber", and "<Z> learned Berber" tells you nothing about Berber other than that it can be learned, teaches you nothing about learning other than that it is a process that can be applied to Berber, and gives you no vivid picture that might help your mind retain the words or even simply brighten your day. If X, Y, and Z have different gender, number, and person, and the sentences are translated into a language where this affects the form of the equivalent of the word "learned", this set of sentences can show you how to conjugate the verb "learn" in the past tense, but it won't tell you anything else. It won't even tell you as much about verb conjugation as a verb chart will. A set of sentences in random order can't give you visual clues, such as placement on the page, that bring out the inherent patterns. And with no starting point, there's no suitable place to explain important points such as where the patterns hold and where they change or break down.

Sixteen years ago, there was no shortage of dictionaries, or verb charts, or descriptions of other aspects of grammar, either printed or in electronic form. But it was hard to find to find searchable collections of sample sentences that would demonstrate the usage of words and phrases. Trang founded what became Tatoeba in order to address her own frustration with that shortage. She opened her new project up to accept contributions from virtually anyone, with no barrier to entry other than the ability to work on an electronic device over the Internet using a free and anonymous account, very few restrictions other than a set of gradually evolving guidelines that were minimally enforced, and an interface that enabled correction of errors after they had been introduced instead of a moderated pipeline that would require sentences to be approved before going live. The website also put itself in the world of social media by providing the ability to leave comments on sentences and on a general "wall", and provided statistics on the number of contributions by language and by individual contributor.

This structure was both a blessing and a curse. It contained the potential for a wide variety of people around the world to collaborate in harmony to contribute an almost infinitely diverse set of utterances. It also contained the potential for individuals to antagonize each other while mass-producing colorless sentences of minimal value that were less valuable at Tatoeba than they would have been elsewhere.

Tatoeba is somewhere between these extremes of paradise and hell. There are many worthwhile corners of variety in the corpus for people who know how to look for it and are willing to expend the effort to extricate the jewels from the trash. The problem is that large-scale contribution of near-duplicate sentences makes this harder and harder to do. Since those sentences have similar content and are added to Tatoeba at the same time, they'll all show up in the same place in your results, and will crowd out everything else. I'm not saying that adding ten "<> learned some Berber" sentences is a problem, or that someone needs to panic about adding a sentence that might be similar to one that someone else has already added. But adding a hundred "<> learned some Berber" and a hundred "<> will learn some Berber" sentences, or going on a rampage of adding a "Ziri" copy of every "Tom" sentence you can find, is another matter.

Furthermore, the elimination of variety tends to propagate itself via translation, or via copycat or retaliatory behavior. In an unintentionally ironic post below (https://tatoeba.org/en/wall/sho...essage_39426), CK filled a screen with a near-duplicate series of near-duplicate sentence pairs (35 of them!) in an apparent attempt to bolster his assertion that eliminating variety in the use of proper nouns is a way to avoid lack of variety in other aspects of language. I think the preposterousness of this idea is self-evident: it's like claiming that preventing someone from learning history is going to make them a better math student. But it's especially worth pointing out that many of those duplicate sentences were added in reaction to the ubiquity of the name "Tom" throughout the corpus. So CK, or more broadly speaking, the general elimination of one aspect of variety, had a large hand in creating the problem in the first place.

It would be nice to introduce some mechanized way of mitigating the problems introduced by mechanized creation of sentences (including using one's mind like a machine, whether or not a computer is involved). But as I have already described, I don't think this can be done easily, quickly, or in a way that would approach any meaningful consensus. Instead, I want to explain to the people who are tempted to add near-duplicate sentences to Tatoeba on an industrial scale that they are probably better off doing the same thing elsewhere, and that there are easy ways to do it that they might not have thought of.

Let's say you want to collect sentences and audio containing Kabyle placenames for a GPS, or that you want to cover all the conjugated forms of Berber verbs that you can think of. It's not as though you have thousands of contributors in unison acting on a top-security project that requires millisecond response time. Is it not simply possible to write the sentences in a spreadsheet on your computer and put the audio files you record into a folder? Or in Google Docs or Google Drive? Or set up your own simple website? (A simple search for "easy free ways to set up a database on the web" brings up 924 million results.) They'll be easier to retrieve from such a place anyway. Why involve Tatoeba at all?

Choosing the best site to help you prepare a special project takes a little bit of imagination and a little bit of effort. So does contributing sentences with enough variety to be worth something to others at Tatoeba. Don't let that stop you.

חברי דוברי העברית, אנחנו במרחק 4 משפטים מתרגום מלא של מנשק המשתמש לעברית, מנסיבות של חוסר הבנה מעמיקה בבלשנות יפנית ודרישות המערכת אני לא יודע איך לתרגם את 4 ההודעות הנותרות.
אשמח אם יודעי דבר יוכלו לתת שם כתף ונסגור את נושא התרגום לעת עתה.

תודה!

🍎 Japanese N-grams

日本語 N-gram Searches of the Tatoeba Corpus - Top 4,000
http://tatoeba.ueuo.com/jpn_ngrams-1.html


The following is another set of longer N-grams that are "sentence endings."

日本語 ・ N-grams ending with a sentence boundary
http://tatoeba.ueuo.com/jpn_7-g...e_endings.html

These are pages I put together in 2016 based on research done by Susumu Yata.
The "advanced search" links will return whatever is currently in the tatoeba.org database.

🍎 We now have over 200 Swedish sentences with audio.

The list, showing only English translations
https://tatoeba.org/en/sentence...how/171175/eng

The list, showing all translations
https://tatoeba.org/en/sentence...how/171175/und

ull joined our project on February 8, 2023.

> 11,111,107 sentences

16:20 UTC
February 12, 2023

Soon, we'll hit 11,111,111.

An English-speaking Japanese dog might say, the following. 😊

ワン、 ワン、
ワン、 ワン、 ワン、
ワン、 ワン、 ワン 。

Is there a list of sentences by difficulty? Let's say I wanted to order 500 easiest sentences in Korean, how would do I this?

🍎 A list with over 2,000 Japanese sentences that don't have kanji

https://tatoeba.org/en/sentences_lists/show/170911

All sentences on this list are owned by native Japanese speakers.

Is it only with my Tatoeba being so often out of order in the last two weeks?