Wall (6,213 threads)
Before asking a question, make sure to read the FAQ.
We aim to maintain a healthy atmosphere for civilized discussions. Please read our rules against bad behavior.
an hour ago
4 hours ago
7 hours ago
7 hours ago
16 hours ago
22 hours ago
23 hours ago
23 hours ago
Are the tool and data used available online anywhere?
yep it's from the kakasi project
as said before, the project seems more than dead :(
Oooh yes. I remember this now.
The bad news is that kakasi probably isn't really fixable. I think you'd need to re-writing the code in a major way, not just add a few lines to the dictionary, to fix it.
The good news? Removing the line
from the file 'kakasidict' may correct one romaji error in generated romaji.
I think we can also try to find if there's people motivated to start a project for automatic romanization of japanese, or looking if there's not an embryon of such project and see how we can help
For now, if there was the possibility to enter the romaji explicitly, and if manually entered and automatically generated romaji could be separated, that should make for a good test set for evaluating different methods for automatic generation.
I think that ideally one would start with a mature project, and automatically add corrections to the training set.
> I think that ideally one would start with a mature
> project, and automatically add corrections to the
> training set.
There are six main approaches that could be taken.
1. Drop romaji support.
2. Allow manual correction of romaji.
3. Develop romaji generation code that uses the WWWJDIC index line.
4. Further develop kakasi
5. Look for alternative romaji conversion software.
6. Develop romaji conversion software from scratch.
I would recommend 1, 2, or 3.
4. Could be done, but I think you would soon reach limits on what is achievable.
(I don't speak japanese at all, so excuse me if i speak non sense)
Nemo talk about JUMAN to replace kakasi, which can output in kana,
is kana not better as that way we're sure people who can't write japanese will not "accidently" mess up the "romanization", by restricting the "reading" part to kana characters , and we're also sure people use the same convention as there's only one kana per "sound"
(Trang always take about different way to write the romaji)
what do you think ? Trang ?
I should give a little more information than I have in the past posts I have, I think, because there seems to have been little progress. I don't really want to come off as being harsh, but the reality is that Kakashi is a lost cause. Whoever coded the program did so in a very naive way, and to use sed to correct its errors would take an inordinate amount of both human and CPU time, and in the best case scenario, it would cause such undue load on the server so as to make tatoeba unusable. I've gotten the impression that Kakashi was chosen with little to no consideration of other options (c.f. below), despite the fact that there exist ways to accurately dissect Japanese text into parseable units, which could be further changed into romaji. The reality is, Kakashi is nowhere near mature enough to produce accurate results, and as an abandoned project there is little hope of it reaching that maturity -- its output will never get any better than it is. In contrast, Juman seems to be near-perfect, though I will admit that I have not tried the other romanizers suggested in the blog post, nor have I done extensive testing of Juman. Regardless, Juman seems to be acceptable, even optimal. Kakashi falls so far short of the mark that I'm not sure why it is even in use. I would even go so far as to state that if Kakashi remained the method of conversion, that by the time tatoeba becomes popular, greasemonkey scripts will be produced which correct romaji via some other means, if that's even feasible. (Here's the blog post I referenced: http://blog.tatoeba.org/2009/02...anization.html )
Yes, to be honest, KAKASI was chosen with no consideration of other options. It was the first one I found that when I searched for a romaji converter, so I picked it.
And only later I wrote this blog post where actually searched and I listed other solutions. Solutions that I should have explored but never had the time to =/
I completely agree with you that KAKASI is not the long term solution.
Anyway, considering you have been taking the time to write all these posts, I will take a look at Juman ;). But if you can just tell me quickly what command to use to get a Japanese sentence parsed and converted into kana, that can save me some time from going through the documentation. Ah but, does JUMAN supports UTF-8...?
From a quick look, it looks like you have to convert to and from EUC-JP. Piping a sentence through "juman -b -c" gives one line per word, with readings in the second of the space-separated columns.
There's a powerpoint tutorial, I'll look at it when I have time and translate it. The translated user guide focuses on the whole idea behind the system, and why it was/how it was developed, and then when it comes to the syntax, it's just a bunch of "I don't know this word" and "If you break this down, it would mean something like..."
If Juman's kana/categorization output is accurate, it can produce 100% perfect romaji output. Kana give a representation of how something is said, along with its syntactical representation. There are ambiguities in kana, but JUMAN gives enough information that the pronunciation and syntax can be reconciled to provide a perfect, phonetic, romanization.
My ideal approach would be using WWWJDIC indices, combined with a better software for conversion into romaji or kana.
As for making romaji editable, if we were to make anything editable, I'd rather it be kana, like what sysko suggested.
If the purpose is to provide something useful for learning, then it's obviously better to have a sentence in kana, with spaces so that the learner knows how the sentence is composed. And of course we can use the sentence in kana to generate correct romaji.
That took me forever to write but hopefully it will prevent us from explaining certain things over and over again: http://blog.tatoeba.org/2010/02...n-tatoeba.html
WOooOW i haven't read it entirely, and you've did a really damn good job, many thanks to Trang :)
Thanks Trang, this clears up a lot of problems. Very helpful. =) And kudos for writing all of this.
Autolinking broken (See latest comment in http://tatoeba.org/eng/sentence...81800#comments )
ends up pointing to
yep the issue is known and already fixed, it will reported in next release (which will come soon) :)
Here's another suggestion.
There's some space on the right hand side of the 'home' page. I suggest you use it to show the most recent posts in the 'Wall'. Probably best if you just show the first line or two and make it a link to the #-anchor of the message in question.
In a break from the difficult and / or controversial suggestions I have one simple one to offer.
I suggest that the sentence list pages (e.g.
) should use their description as their page title (e.g.
Sentence lists: jpn->eng translations needed
instead of just
Seriously - romaji editing now. ;-)
I don't think there's any point in waiting for "a serious Japanese contributor". Most of the romaji errors are very obvious and either I, or half a dozen or so regulars here, would be well able to correct them if they had the chance.
I would go as far as to say that it would be better not to have romaji AT ALL rather than leave them in the current state.
Then it may be no romaji at all... But I want opinion from more users first. Is it better to have no romaji at all, or is it better to have something even if it's not 100% correct?
I know Nemo is against romaji as well, but if I have added it, it was because more than two people had requested it in the past.
Regarding editable romaji, I'd rather avoid having people to waste time on correcting romaji which is why I don't want to make it editable.
Most of the time it's a systematic error that can be found in more than 100 other sentences. If I made romaji editable, you'd have to edit them one by one.
You'd also have to make sure everyone agrees on the romanization rules and follows them, which is again more work.
I think it's better to improve the software (not necessarily KAKASI) to the point where it can't get any better. It would save time for so many other people in the world...
Perhaps there is someone out there who is actively developing an open source Japanese parser and furgina-romaji converter. I haven't had time to search, but if you do find one (and by "you" I mean anyone who is reading this), by all means, let me know.
> I'd rather avoid having people to waste time on
> correcting romaji which is why I don't want to make
> it editable.
That's basically another way of saying that the romaji isn't important. If the romaji isn't important I'd rather it wasn't there than be there and often incorrect. ;-)
Having a kana version or furigana would be a nice alternative. kana would get rid of the
o / wo
e / he
wa / ha
confusion. Note that a combination of Edict and the Index information could be used to generate pretty-much-correct furigana or kana. (Not that easy, but doable)
I agree with Paul that furigana might me preferable to broken rōmaji. Learning the basics of kana shouldn't take you more than a month or two, after that, kanji readings become the hard part. Furigana should, in my opinion, eliminate the need for rōmaji for learners of Japanese.
Rōmaji is mostly useful for transcribing Japanese for a public that cannot read any Japanese at all. Also, the rōmaji generated by Kakasi is wāpuro-rōmaji.
I'm for having all the romaji on the site be accurate. If the best way to do that is deleting all of the romaji, then I'd say do that. If you really want an accurate romaji representation, it will probably need to be written ad hoc. I don't think this should be too hard though, so long as it is written for this project specifically, and it is done soon. This site is currently comprised mostly of the Tanaka Corpus, so far as I am aware, so almost every word in the Japanese examples should be also present in EDICT, which has the reading of every word in it in kana. If there are multiple readings, I would just make the output something like:
*** boku wa (shijyou | ichiba) e itta
So that the edge cases could be fixed. It's still a lot of work, but it's doable. (In this case, the difference is irrelevant, but in many it could be relevant). You could then dump the database into a text file of all beginning with ***. I believe EDICT even has the readings listed in order of frequency, so if you wanted to you could have it just guess the first one every time, and fixing the few that got put in incorrectly would not be a huge ordeal. I would recommend keeping some automatic conversion in place, and storing things in the database as:
and having the conversion take place from the kana to romaji on-the-fly. Also, force those editing the romaji to use kana. Basically introduce a learning curve that will discourage those who don't know better from thinking they do. Also, changes in romanization could be implemented very easily. I personally use wapuro romaji whenever I do, which is rare still, but I know this is less than ideal for learning.
My whole post is a waste of time, lol. The software you are using has an output to kana mode, which would not be subject to the pitfalls that romaji is. I suggest we use that. Kana is not that difficult to learn, and there's no sense in learning grammar/sentences before kana anyway.
We need post editing, haha. JUMAN does exactly what you need. It converts from kanji to hiragana, and labels each word with what it is. So, if it says は is a 助詞 (particle), you can output wa, and the same for all of the others. I'm not sure that it outputs romaji (The sample set-up does not), but with kana and part of speech, romaji is just a lookup table away.
May I disturb the silence once again. Consider the situation where somebody translates sentence A into B, then somebody later comes along to translate B into C. It then turns out that B is wrong and is consecutively changed. This invalidates C. Any ideas/plans for that?
Part of the answer is in the comment I wrote here:
And in my todo list for the weekend: write some guideline so that users know how contribute correctly.
Thx to both of you. I see the whole system is well thought-out.
this have already been discussed ^^
soon, we will add an unlink feature, so you will be able to say "these sentence are no longer translations to each other"
so what to do in your case
in fact you're not supposed to change the B sentence, as long as the sentence is by itself correct, because as you've said, it will make translation of B erroneous too
so you just add a B2 sentence and add a note that the B sentence will need to be unlink to A sentence
Finding contributors (to the code): Why don't you guys (& girls) share (i.e. open source) your site's code on say github? First that would make this a truly "open source" project, and secondly people could help add features. Think about it.
Some guys here seem pretty eager to get their features implemented ;)
Like sysko said, we are actually open source. The reason why it's not promoted anywhere is because:
1) The code hasn't met my standards of elegance yet... Still too many parts that make me cringe when I look at them.
2) We still don't have a sound methodoly and organization in our way of working and I really don't have time to manage more people ^^'
(we're in PHP though, sorry :P)
Good enough. Desolé for the PHP part, but then it's self-inflicted :)
Something this thread brings up, the user interface should be stored in a cookie, not as part of the URL. French is easy enough, but what if I click http://tatoeba.org/chi/sentences/show/366507/ and I'm new and/or don't read Chinese? I'm stuck and have to go back, close the window, or start over at the main page. Quick and Dirty fix would be change all of the languages to something like "English - English; Francais - French; zhongwen - Chinese; nihongo - Japanese" etc. (Not suggesting they be romanized, I just don't feel like dealing with my IME.) Not trying to be Anglocentric, but most people can decipher language names in English.
it is stored in the session,
moreover at the top of each pages you have a menu to change the language interface ;-) with the name of each language in it's own way (français,english, 中文 ...) and when changing it reload the page with the new language, and change the language of your session
and all new pages you want to show will be displayed in this language
My point is, not everyone knows Chinese/Japanese. In fact the vast majority cannot read a single character. So they arrive at a page, seeing "中文" at the top is of little or no help, nor is "日本語". It's not a problem for you, not a problem for me, but sit some random people down at the page at tell them to navigate it.
ok, agree, will see to make it clearer :)
In fact the code is already available on a svn public repository (with read only access and a AGPL licence (I'm a bit FOSS fanatic), but I don't think we're against code contributors, so write access can be granted for motivated code-contributors)
after why it's not explicity written somewhere on the website, hmm I think (Trang has maybe much relevant reason than me) it's because the project lacks documentation and is in a rewriting / cleaning phase, so we prefer to show a pretty reviewed code ^^
but if you want to take a look, I can give you the repo in private message
Wrt link to repo: depends on the language. I ain't touching PHP :p
it's PHP :p
Hehe, to be honnest the use of PHP is more about an historical choice ( i would like python too :p)
For the 'wish list' I would like to suggest a couple of features.
1. People who don't own a sentence can click an icon (check a checkbox) when posting a comment to make it an official request to change a sentence.
2. Add an extra line to the "Your Links" section of the profile.
* View all my sentences
* View sentences with change requests
* View my favorite sentences
* Sentences with undetected language
It is too easy for sentence owners to miss comments. For example I suspect fcbond hasn't noticed the comment I made in the following link.
Maybe that's a bit too complicated. At least there are several other feature requests in the pipe, that might have more impact.
As you indicated in the 2nd note already though, adding a "news feed" for requests on one's sentences is surely needed.
I would propose to change your request into something with a broadend scope: "disown request" to disown sb by taking over their sentence. This could also be used on inactive users. Some time without reaction *zing* you go ahead.
I also was considering recommending a way of taking over neglected sentences, but I considered the 'change request' to be a less controversial idea. After all not everybody who's away for a month or two has actually given up.
maybe in a more general way, we can make something "a la twitter", I mean, when you want a comment to be notify to someone, you just @userNickName, and maybe add a "feed" section in the profile
anyway in future release we plan to review a bit the architecture, add some category in the profile, and make your profile a more central page
what do you think ?
PS: for disowning, I will see with Trang how she planned to handle it (we used to talk about that, but i've a weak memory :( )
The way I envision it is not really "disowning", it's more about finding a better owner if the current owner doesn't do his job (kind of like, if you're a bad parent, your kids are taken away).
But I still don't know what would be the best solution because we don't really have a real need for that yet. I mean, it's not very frequent that you'd want to disown someone from his/her sentences. Which means it's not an urgent feature either.
Normally he fcbond should have received a notfication when you commented his sentence, but the notification system was broken :'( ...about 100 notifications that should have been sent but weren't, *sigh*.
Anyway, having a link to a page that lists the comments posted on your sentences is something I should have done a long time ago... When I decided to integrate an "ownership/adoption" system in replacement of the moderation system we used to have, it was obvious that users should be able to quickly access comments on their sentences.
But I think that having a checkbox would be add unnecessary complexity. People will usually read all the comments on their sentences, so there's no need to filter out specifically those that require a correction.
+1 feature request:
Tag sentences for a special form. Say your language has several possible translations for a given sentence, tag these sentences for the given feature. Example: http://tatoeba.org/eng/sentences/show/366466 I would add one for informal you (french 'tu') and one for formal you (french 'vous').
A wiki would be nice to list those feature request, I think this "Wall" might get a bit to muddled.
+1 for the wiki, in fact it's already in the todo list (in fact as for the code, we've also a ticket system for developper)
and you're true it can be great as someone has already talked about tagging
I was kinda thinking a PHPbb install or a Google Group would be effective. I'd limit access though.