52nlp's Learning Notes » Dictionary

Moses Support Digest:Dictonary use during training

52nlp — Wed, 24 Feb 2010 12:37:43 +0000

[Moses-support] Dictonary use during training

Hi all, I’m wondering if you would know where I can find an english to spanish parallel, word to word dictionary to complement my training corpus.

Also, from what I have searched I understand you can either add the dictionary words at the end of the corpus or use the giza option. I would like to try both, but for the giza option -d I see that the file format uses the word’s ids, then where will the real words (from the parallel dictionary) go? in the corpus as well? or in a separate file?

Any other suggestions for using a dictionary are welcome.

Thank you.

Re:[Moses-support] Dictonary use during training

re: adding dictionary entries, this is certainly a hack. but the standard trick is to pretend that the dictionary actually consists of tiny parallel sentences. you therefore just append each word-entry as a new sentence pair. don’t bother with that -d option.

Miles

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Moses Support Digest:dictionary problem solved

52nlp — Wed, 23 Dec 2009 15:50:50 +0000

[Moses-support] dictionary problem solved

Hi all,

This dictionary problem is finally solved. “-d” option works well. I made a silly mistake here and caused the problem. I converted the dictionary file to UTF8, but the coding of other files is:7bit ASCII characters. So sorry to bother you for such a long time…
I really appreciate your kind help, especially Mark Fishel and Chris Dyer. You have helped this green hand a lot

As I google this dictionary problem, all I found is my own question. So, to those who may use dictionary and don’t know how, here’s the advice:
1. well…make sure your texts of the same coding
2. check your giza++ source code, and find variable “useDict”, make sure it’s set to ture
3. add a “-d” option to your command, followed by your dictionary the dictionary should be in this format:
target-word-id source-word-id
it must be sorted by the target-word-id.

here’s my command line:
(you may have to know those options which are set to 0 or 1, or a lot of files would be generated )

./GIZA++
5 -CoocurrenceFile korean-chinese.cooc
6 -c korean-chinese-int-train.snt
7 -m1 5 -m2 0 -mh 5 -m3 3 -m4 3
8 -model1dumpfrequency 1
9 -model2dumpfrequency 1
10 -model345dumpfrequency 1
11 -hmmdumpfrequency 1
12 -model4smoothfactor 0.4
13 -nbestalignments 1
14 -onlyaldumps 0
15 -nodumps 0
16 -nsmooth 4
17 -d ck.txt
18 -o korean-chinese
19 -onlyaldumps 1
20 -p0 0.999
21 -s chinese.vcb
22 -t korean.vcb

2009-12-23

Best regards,

Lee Xianhua

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Moses Support Digest:How to run giza++ with a dictionary

52nlp — Mon, 21 Dec 2009 12:27:03 +0000

[Moses-support] How to run giza++ with a dictionary?

hi all,
How to run giza++ with a dictionary?

I’ve looked through both moses manual and giza++ readme,but there seems no answer to this question.All I got to know is the format of dictionary, and so on.Could somebody please help me with this?

My command line is like this:

./GIZA++-m3 -CoocurrenceFile en-ch.cooc -c en-ch-int-train.snt -m1 5 -m2 0 -mh 5 -m3 5 -m4 0 -model1dumpfrequency 1 -model4smoothfactor 0.4 -nodumps 0 -nsmooth 4 -o en-ch -onlyaldumps 1 -p0 0.999 -s ch.vcb -t en.vcb > logec 2> errec &

Thanks in advance.

2009-12-21

Best regards,

Lee Xianhua

Re:[Moses-support] How to run giza++ with a dictionary?

I’m not sure what the command line options are off the top of my head, but I seem to recall that Giza just boosts the counts of pairs from the dictionary by some fixed amount (5 or something). You can get same effect by adding the entries from your dictionary to the end of your corpus.

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

A Cool Dictionary for Natural Language Processing

52nlp — Mon, 30 Nov 2009 15:53:53 +0000

I found Professor Bill Wilson’s “The Natural Language Processing Dictionary” accidentally tonight, and thought it very cool for nlpers. Except from the NLP Dictionary, you also can find the Prolog, Artificial Intelligence and Machine learning Dictionary in this web page. Below is from this Dictionary:

“You should use The NLP Dictionary to clarify or revise concepts that you have already met. The NLP Dictionary is not a suitable way to begin to learn about NLP. Further information on NLP can be found in the class web page lecture notes section.

Other places to find out about artificial intelligence include the AAAI (American Association for Artificial Intelligence) AI Overview page or AI Reference Shelf

If you wish to suggest an item or items that should be included, or if you found an item that you felt was unclear, please let me know (E-mail: billw at cse.unsw.edu.au). ”

If you are interested in NLP Dictionary and others, visit them:
The Natural Language Processing Dictionary – URL:http://www.cse.unsw.edu.au/~billw/nlpdict.html
The Prolog Dictionary – URL: http://www.cse.unsw.edu.au/~billw/prologdict.html
The Artificial Intelligence Dictionary – URL: http://www.cse.unsw.edu.au/~billw/aidict.html
The Machine Learning Dictionary – URL: http://www.cse.unsw.edu.au/~billw/mldict.html