I Love Natural Language Processing

[Moses-support] POS LM

Hi

There is pos.lm of the target language in factored model training. I want to know the steps involved in preparing the POS.lm and the kind of input parameters altogether.

-Doren
Read the rest of this entry »

[Moses-support] tuning tree-based models

Hello All,

I want to tune a tree-based trained system. I pass the moses_chart decoder to the mert-moses-new.perl script. The script seems to make 1 decoding run producing features.dat and scores.dat files, but it writes an empty extract.out file. Then the script stops. I also noticed that the train-moses.perl script only writes out a “X” under [non-terminals] in the moses.ini file. I have several non-terminals on the input side.
Thanks,

–
Regards,
John J Morgan
Read the rest of this entry »

[Moses-support] Tuning failure with “Language model type unknown.”

Hello,
I am currently trying to train a Moses system, which goes fine until the tuning step. I get the following error :

Creating lexical reordering…
weights: 0.111111 0.111111 0.111111 0.111111 0.111111 0.111111
binary file loaded, default OFF_T: -1
Created lexical orientation reordering
ERROR:Language model type unknown. Probably not compiled
into library
ERROR:no LM created. We probably don’t have it compiled
Exit code: 1
The decoder died. CONFIG WAS -w 0.000000 -lm 0.111111 -d
0.111111 0.111111 0.111111 0.111111 0.111111 0.111111 0.111111 -tm
0.033333 0.022222 0.033333 0.022222 0.000000

I have checked the archives of the mailing list, and couldn’t find anything to resolve the problem.
The language model line in the run1.moses.ini is the following
# language models: type(srilm/irstlm), factors, order, file
[lmodel-file]
1 0 5 /home/s0678231/disk1/mt-system/lm/europarl.binlm.1

Is anyone able to help with this ?

Thanks in advance,
Ronan
Read the rest of this entry »

[Moses-support] running giza in parts

Dear list,

can anyone direct me to a description of the exact algorithm of running giza++ in parts? I know the co-occurrence file is used for more memory efficient storage of the translation table and probably basically defines which word pairs are to be included into the t-table. However I’m not sure how the combination of several co-occurrence files is performed if the training data is processed in several parts (–parts N). I tried reading the training script (the “run_single_giza_on_parts” sub) and the algorithm is still a mystery to me.

Thank You in advance,
Mark Fishel
Read the rest of this entry »

[Moses-support] The results of your email commands

Hi,
I would really appreciate it if you could help me with the following question I have:
I was wondering if a LM created with RANDLM can be converted into a binary format?

Or is there maybe another way of loading the model faster?

I know it is possible with IRSTLM and SRILM but I couldn’t find anything about RANDLM.
Thank you in advance for your support

Best regards,
Arda Tezcan
Read the rest of this entry »

[Moses-support] dictionary problem solved

Hi all,

This dictionary problem is finally solved. “-d” option works well. I made a silly mistake here and caused the problem. I converted the dictionary file to UTF8, but the coding of other files is:7bit ASCII characters. So sorry to bother you for such a long time…
I really appreciate your kind help, especially Mark Fishel and Chris Dyer. You have helped this green hand a lot

As I google this dictionary problem, all I found is my own question. So, to those who may use dictionary and don’t know how, here’s the advice:
1. well…make sure your texts of the same coding
2. check your giza++ source code, and find variable “useDict”, make sure it’s set to ture
3. add a “-d” option to your command, followed by your dictionary the dictionary should be in this format:
target-word-id source-word-id
it must be sorted by the target-word-id.

here’s my command line:
(you may have to know those options which are set to 0 or 1, or a lot of files would be generated )

./GIZA++
5 -CoocurrenceFile korean-chinese.cooc
6 -c korean-chinese-int-train.snt
7 -m1 5 -m2 0 -mh 5 -m3 3 -m4 3
8 -model1dumpfrequency 1
9 -model2dumpfrequency 1
10 -model345dumpfrequency 1
11 -hmmdumpfrequency 1
12 -model4smoothfactor 0.4
13 -nbestalignments 1
14 -onlyaldumps 0
15 -nodumps 0
16 -nsmooth 4
17 -d ck.txt
18 -o korean-chinese
19 -onlyaldumps 1
20 -p0 0.999
21 -s chinese.vcb
22 -t korean.vcb

2009-12-23

Best regards,

Lee Xianhua
Read the rest of this entry »

[Moses-support] How to run giza++ with a dictionary?

hi all,
How to run giza++ with a dictionary?

I’ve looked through both moses manual and giza++ readme,but there seems no answer to this question.All I got to know is the format of dictionary, and so on.Could somebody please help me with this?

My command line is like this:

./GIZA++-m3 -CoocurrenceFile en-ch.cooc -c en-ch-int-train.snt -m1 5 -m2 0 -mh 5 -m3 5 -m4 0 -model1dumpfrequency 1 -model4smoothfactor 0.4 -nodumps 0 -nsmooth 4 -o en-ch -onlyaldumps 1 -p0 0.999 -s ch.vcb -t en.vcb > logec 2> errec &

Thanks in advance.

2009-12-21

Best regards,

Lee Xianhua
Read the rest of this entry »

[Moses-support] -lmodel-dub parameter

Hi List,

Could anyone please explain what is the usage of -lmodel-dub parameter? Does it related to language model filtering? In the help message it says “-lmodel-dub parameter” but no furthur information is given e.g. format of dictionary, how the boundary works etc.

Thanks a lot!
–Q
Read the rest of this entry »

[Moses-support] moses threads compilation problem (with RandLM)

Hi Barry and other folks,

I’m also having trouble compiling Moses with threads and RandLM, there seems to be a bug in MainMT.cpp ?

Here is what I am doing:

Get fresh copy of Moses (I did this on Monday night).

./regenerate-makefiles.sh
./configure –enable-threads
–with-srilm=/home/users6/fraser/statmt/srilm-1.5.7
–with-randlm=/home/users6/fraser/statmt/randlm-v0.11
–with-boost=/home/users6/fraser –with-boost-thread=boost_thread
make

(The last argument –with-boost-thread is necessary to stop it from picking up the globally installed boost thread library).

I attach config.log, which makes it through fine (though I think there is some weird use of -DWITH_THREADS in there which might be interesting).

I also attach make.log (which only contains the compilation error, I typed make twice).

Let me know if I can provide any more info.

Thanks a lot for your help!

Cheers, Alex
————– next part ————–
A non-text attachment was scrubbed…
Name: config.log
Type: application/octet-stream
Size: 17359 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20091216/b1812336/config-0001.obj
————– next part ————–
A non-text attachment was scrubbed…
Name: make.log
Type: application/octet-stream
Size: 5391 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20091216/b1812336/make-0001.obj
Read the rest of this entry »

[Moses-support] How do you solve this moses problem? Thanks

Hi friend,

I have encountered the same problem as somebody, like in this link:

http://www.mail-archive.com/moses-support@mit.edu/msg01646.html

Before three months, I did the step-by-step guide for moses in

http://www.statmt.org/moses_steps.html

Then, everything is right.

I do not know how to solve this problem. I want to know whether you have solved it. If so, could you please tell me how to manage it?

Best wishes;
Jun Lang
Read the rest of this entry »