Work in progress

There are currently lots of index records that need altering because words have been added / removed / changed in Edict.

For example
* headwords that were unique may no longer be unique (so readings need to be added).
* Entries that were once two separate dictionary records may have been merged into one (so indexing needs to be changed to one keyword only as well)
* Adding, removing and merging words may leave the |1, |2, ... notations needing an update.
* There is also a great deal of general checking to do.

Unique identification of a JMDICT entry.

This is technical stuff, only really of interest for people who want to deal with the 'index' field of Japanese sentences.

The obvious way to identify a dictionary entry as used in WWWJDIC + Edict is by the entry number (duh!). However that's not how it's done in the index field. Why? Because 2147630 is not 'human friendly' for whoever is creating and editing the index fields. (i.e. you can't look at 2147630 and know what word it refers to)

You could identify by the headword of the dictionary entry - あっという間に will only match one record. However there are around 3,000 dictionary entries where that will not be enough. 前(まえ) is not the same entry as 前(ぜん). So, for ambiguous kanji headwords, you include the reading of the word as well.

You've now reached the basic method used by Jim when he links WWWJDIC to example records. However that is not enough to identify 100% of the entries in JMdict uniquely. There are kana headwords like 'は' that are present in more than one entry.

はあ; は (int) (1) yes; indeed; well; (2) ha!; (3) what?; huh?; (4) sigh

は (prt) (1) (pronounced わ in modern Japanese) topic marker particle; (2) indicates contrast with another option (stated or unstated); (3) adds emphasis; (P)

Note that the second [EX] link in the はあ; は entry is actually for the particle は. There are a few like this that can only be fixed by re-doing the indexing system that Jim uses (in a way that would be more complicated and slower than he wants to deal with).

Now we come to the notation used in Tatoeba (and that I used 'at home' when maintaining the Tanaka example collection. Every headword that is not unique has the notation |1, |2, |3, added.

So instead of 前(さき) 前(ぜん) 前(まえ)
you might have 前|1(さき) 前|2(ぜん) 前|3(まえ)

Some points to note:
* The numbers are assigned in order by JMDict entry ID. So 前|1(さき) (Entry 1387210) comes before 前|2(ぜん) (Entry 1392570) and 前|3(まえ) (Entry 1392580)
* Because I used Access (and Excel) some characters are treated as 'equivalent' by Microsoft that are not actually identical. So ヽ|1 ヾ|2 ゝ|3 ゞ|4 and 々|5 all need to be distinguished by the numerical notation.
* The order of headwords / readings is significant in JMDict - most common headwords are supposed to come first, least common last. When indexing the preference is to use the first headword / reading when possible.

NOTE that the point of the index data is to uniquely identify a dictionary entry, NOT to reflect 100% accurately the dictionary form and reading of the word being indexed.

e.g. If you used 挙がります in a sentence the index would include
上がる{挙がります}
it would not include
挙がる{挙がります}

Apart from anything else this ensures that there is only one [EX] link from dictionary entries.

Finally the square brackets note the sense of the word
上がる[02] = Second sense of the word 上がる
and the curly brackets show the /exact match/ for the indexed word in the sentence.

僕は学校の成績が上がった。
僕|2(ぼく)[01] は|1 学校 乃{の} 成績 が 上がる{上がった}

*Psst* Trang - maybe this should be on a page somewhere?

Busy signal

Just to note that I'm going to be really busy with fixing lots and lots of stuff with the Japanese Index field for the next few days.

If you left a comment on a sentence you'd like me to read, please PM me because I won't have time to deal with it now.

[removed by CK, since this no longer applies]

[not needed anymore- removed by CK]

Bump.

Examples.gz, how often is it updated?

On the Monash FTP Archive page it says:

[...] It is updated daily from the server site.
* examples.gz (8371412 bytes) the file.

But I think that is no longer accurate. The one I've just downloaded hasn't been updated from a week ago. Also could the ID numbers given in examples.gz be from the Japanese sentences, not the English sentences? The English ones aren't unique so the IDs are pretty much useless.

Example export system.

Japanese sentences with multiple English translations are (sometimes?) being exported with both versions.

For example:

4924 73899 「これが探していたものだ」と彼は叫んだ。 "This is what I was looking for," he exclaimed. 此れ[01]{これ} が 探す{探していた} 物|1(もの)[01]{もの} だ と|1 彼|2(かれ)[01] は|1[01] 叫ぶ{叫んだ}
4924 1513 「これが探していたものだ」と彼は叫んだ。 "This is what I was looking for!" he exclaimed. 此れ[01]{これ} が 探す{探していた} 物|1(もの)[01]{もの} だ と|1 彼|2(かれ)[01] は|1[01] 叫ぶ{叫んだ}

On the sentence annotations page only 73899 is given as the 'meaning' field with the Index field. So either a) The meaning field in the sentence annotations page isn't used, or b) There can be two or more 'meaning' field values, but only one is shown on the sentence annotations page.

.csv format in downloads.

Just a note for those using the downloads. You use \ as the escape character.

This is a line from your csv file:

"4923";"1512";"「信用して」と彼は言った。";"\"Trust me,\" he said.";"信用 為る|1(する){して} と|1 彼|2(かれ)[01] は|1 言う{言った}"

This is how it appears when loaded into Excel.

4923 1512 「信用して」と彼は言った。 \Trust me,\" he said." 信用 為る|1(する){して} と|1 彼|2(かれ)[01] は|1 言う{言った}

Excel uses double " marks when escaping quotes. The same line in csv for Excel would be...

"4923";"1512";"「信用して」と彼は言った。";"""Trust me,"" he said.";"信用 為る|1(する){して} と|1 彼|2(かれ)[01] は|1 言う{言った}"

Which imports to Excel as follows:

4923 1512 「信用して」と彼は言った。 "Trust me," he said. 信用 為る|1(する){して} と|1 彼|2(かれ)[01] は|1 言う{言った}

I think the 'escaping with extra quote mark' may be the more standard version ...

Right. So. I'm not 13 years old. It was an honest mistake. Here's the problem:

I was wondering if Tatoeba had any sort of resistance to profanity. I thought something like "damnit" would be a common enough thing. So I MEANT to SEARCH for "damn". Turns out I added it as a sentence instead. Same for "fuck" because it took me two tries to realise I was using the wrong text box.

So those can be deleted outright. I didn't see any way to do it so I abandoned the sentences instead in case there is such a way and someone else wants to adopt them to get them deleted. 380292 and 380290.

Apologies.

stemming should be working again for most languages when using the search engine
i.e search "think" should also return "thinking" "thought" etc. same for French / Spanish / Italian / Russian etc.

by the way it will not work with Ukrainian but I was wondering if using the russian stemmer will produce "better than nothing" result ? Demetrius, Dorenda ?
still looking for Arabic and georgian stemmers