I Love Natural Language Processing

Moses Support Digest: About moses-chart

52nlp — Mon, 08 Mar 2010 10:44:46 +0000

[Moses-support] About moses-chart

Dir sir,
When I train rule table for moses chart, at the end of rule I saw two parameter (see rule example)

! ” . [X] ||| ! ” [X] ||| ||| 0.333333 0.004318 1 0.0854249 2.718 ||| 1.5 0.5

Could you please explain the meaning of tow parameter.
Thank you!

–
Thu.

Re:[Moses-support] About moses-chart

hi thuy

the last 2 numbers, 1.5 0.5, are the (fractional) counts used to calculate the probabilities. Its not used for anything at the moment, just for debugging

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Moses Support Digest:mt3_chart compilation error

52nlp — Wed, 03 Mar 2010 16:26:58 +0000

[Moses-support] mt3_chart compilation error

Hi,

When trying to compile mt3_chart i get a compilation error for PhraseDictionaryDynSuffixArray.o, what could be the cause/solution for this, could it be caused by adding -std=c++0x to the AM_CPPFLAGS (ref: http://www.mail-archive.com/moses-support@mit.edu/msg01483.html)? Im compiling with gcc4.4.

g++ -DHAVE_CONFIG_H -I. -I../.. -W -Wall -ffor-scope -D_FILE_OFFSET_BITS=64
-D_LARGE_FILES -std=c++0x -I/home/marce/srilm64/include
-I/home/marce/irstlm/include -g -O2 -MT PhraseDictionaryDynSuffixArray.o
-MD -MP -MF .deps/PhraseDictionaryDynSuffixArray.Tpo -c -o
PhraseDictionaryDynSuffixArray.o PhraseDictionaryDynSuffixArray.cpp
In file included from PhraseDictionaryDynSuffixArray.h:5,
from PhraseDictionaryDynSuffixArray.cpp:1:
DynSuffixArray.h: In member function âvoid
Moses::DynSuffixArray::printAuxArrays()â:
DynSuffixArray.h:43: warning: comparison between signed and unsigned integer
expressions
In file included from PhraseDictionaryDynSuffixArray.cpp:1:
PhraseDictionaryDynSuffixArray.h: In member function âbool
Moses::ScoresComp::operator()(const Moses::Scores&, const Moses::Scores&)
constâ:
PhraseDictionaryDynSuffixArray.h:61: error: âtypeofâ was not declared in
this scope
PhraseDictionaryDynSuffixArray.h:61: error: expected â;â before âitrâ
PhraseDictionaryDynSuffixArray.h:61: error: âitrâ was not declared in this
scope
PhraseDictionaryDynSuffixArray.h:62: error: expected â;â before âitrâ
PhraseDictionaryDynSuffixArray.h:62: error: âitrâ was not declared in this
scope

Thanks,
Marce

Re:[Moses-support] mt3_chart compilation error

maybe. the typeof function is not standard c++ may be causing problems with different compiler versions.

it was used by Abby who is working on suffix arrays. You can try & fix
it, or comment out the code that uses it.

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Moses Support Digest:SRILM installation problem

52nlp — Mon, 01 Mar 2010 11:08:01 +0000

[Moses-support] SRILM installation problem

Hi All,

I had a problem to install Moses and posted it here but no response is coming out yet. Can any one help me where to post this. I appreciate you guys for any help.

Mulu G.

Hi All,

I was trying to install srilm on Debian linux 5 lenny, MACHINE_TYPE= i686, with gcc version v 4.3.2. I have changed the Makefile according the manuals and without success. So when I try to enter #make World, it produces the following message

/bin/sh: /root/srilm: is a directory
mkdir include lib bin
mkdir: cannot create directory `include’: File exists
mkdir: cannot create directory `lib’: File exists
mkdir: cannot create directory `bin’: File exists
make: [dirs] Error 1 (ignored)
make init
/bin/sh: /root/srilm: is a directory
make[1]: Entering directory `/root/srilm’
for subdir in misc dstruct lm flm lattice utils; do
(cd $subdir/src; make SRILM=/root/srilm [SRILM] MACHINE_TYPE=
OPTION= MAKE_PIC= init) || exit 1;
done
make[2]: Entering directory `/root/srilm/misc/src’
/root/srilm/common/Makefile.common.variables:96:
/root/srilm/common/Makefile.machine.: No such file or directory
make[2]: *** No rule to make target `/root/srilm/common/Makefile.machine.’.
Stop.
make[2]: Leaving directory `/root/srilm/misc/src’
make[1]: *** [init] Error 1
make[1]: Leaving directory `/root/srilm’
make: *** [World] Error 2

Could you please help me in sorting out the problem and how to fix it.

Thank you all,

Mulu G.

Re:[Moses-support] SRILM installation problem

hi mulu

it seems you have an srilm compilation problem rather than a moses problem. from my experience, srilm is easy to compile but you have to read & follow the instruction very carefully. If you dont succeed, I suggest you download SRILM and start again.

you will know when it succeeds when you have these files are created:
liboolm.a
libdstruct.a
libflm.a
liblattice.a
libmisc.a

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Moses Support Digest:GIZA++ error

52nlp — Sat, 27 Feb 2010 12:59:43 +0000

[Moses-support] GIZA++ error

sir,

while running the Giza++ by the command

./GIZA++ -S corp.en.vcb -T corp.ta.vcb -C corp.en_corp.ta.snt

we are getting as below :

what is the coocurrence file ?

how to rectify this problem and run Giza++ ?

reading vocabulary files
Source vocabulary list has 35497 unique tokens
Target vocabulary list has 71683 unique tokens
Calculating vocabulary frequencies from corpus corp.en_corp.ta.snt
Reading more sentence pairs into memory …
Corpus fits in memory, corpus has: 14035 sentence pairs.
Train total # sentence pairs (weighted): 14035
Size of source portion of the training corpus: 330148 tokens
Size of the target portion of the training corpus: 262033 tokens
In source portion of the training corpus, only 35496 unique tokens appeared
In target portion of the training corpus, only 71681 unique tokens appeared
lambda for PP calculation in IBM-1,IBM-2,HMM:= 262033/(344183-14035)==
0.793683
ERROR: NO COOCURRENCE FILE GIVEN
Aborted

thank you

Re:[Moses-support] GIZA++ error

Hi Sreeja,

the error below is due to the fact that you are using a GIZA++ version compiled for use with coocurrence files without using such parameter. Typically, when you compile GIZA++ as part of the Moses toolkit, GIZA++ is compiled so as to use coocurrence files. This allows for better memory usage (and reduces computational time).

Hence, there are two ways to solve this problem:

1.- you compile GIZA++ without the coocurrence file option
2.- you provide a coocurrence file to the GIZA++ you have already compiled.

In order to produce a coocurrence file, you will need to use the
following command:

snt2cooc.out

In your case, this would look like this:

snt2cooc.out corp.en.vcb corp.ta.vcb corp.en_corp.ta.snt > corp.cooc

This will generate a coocurrence file, i.e. corp.cooc, will you will need to pass to GIZA++ as:

./GIZA++ -S corp.en.vcb -T corp.ta.vcb -C corp.en_corp.ta.snt -CoocurrenceFile corp.cooc

the snt2cooc.out binary is to be found in the GIZA++ compilation directory.

Cheers,

Germán Sanchis-Trilles

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Moses Support Digest:Is reordering model a must-be-used component to use?

52nlp — Fri, 26 Feb 2010 16:10:14 +0000

[Moses-support] Is reordering model a “must-be-used” component to use?

I wonder if I may exclude the reordering model during search.Since I came up with my morpho-syntactical preprocessor to transform source languagein both training-time and run-time, and reordering model deals with the local reordering or wordsduring translation, there doesn’t seem to be the need for using it if the preprocessorcompletely got rid of local/global distortion, even for language model?Is my hypothesis justified?Actually, from my subjective evaluation, using phrase-table and lm alone shows better result thanwhen I use them with reordering model. But I am not sure of my theory.

Re:[Moses-support] Is reordering model a “must-be-used” component to use?

The reordering model tries to deal with global reordering rather than local reordering because the latter is generally solved with phrase generation.

So it depends on the language pair used. Cases like Catalan-Spanish may omit the use of reordering models because they are very much alike. English-Spanish or Chinese-English, on the other hand, consider the reordering model as a “must-be-used” component. Phrase generation will not be able to deal with reordering alone there.

You may read more on the Lexicalized Reordering Model here

http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc1

or you may got to the Tillmann paper here

http://portal.acm.org/citation.cfm?id=1613984.1614010

Which languages are you working on?

Re:[Moses-support] Is reordering model a “must-be-used” component to use?

Do you mean “reordering model” in general? If so, it must be used unless the preprocessor “completely got rid of local/global distortion”.

On the other hand, some specific reordering models such as Lexicalized Rerordering Model in Moses may not be useful. Distance-based reordering doesn’t deal with global reordering well, rather it relocate some phrases within a certain boundary.

Despite many researches have been suggested for global reordering, it is a still unsolved problem..

Re:[Moses-support] Is reordering model a “must-be-used” component to use?

I mean the model for the local distortion event when only surface(token) factor is considered, with maximum length 3. I’ve been experimenting with quasi-syntax-based approach in which I transform source corpus ( English ) in a way that (hopefully) minimizes both local and global distortion at training/run-time, using transfer rules I’ve empirically but manually induced.

Yes, the preprocessor is not perfect in getting rid of global distortion, but it works quite well.

Currently I am working on better way to train the “preprocessing” model, not dependent on a rule-based preprocessing step.

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Moses Support Digest:about the moses-chart reordering

52nlp — Thu, 25 Feb 2010 13:58:17 +0000

[Moses-support] about the moses-chart reordering

Dear all:

Could you tell me where can I find the details of moses-chart reordering? It seems that the reordering table is quite different from that of moses. Is there any papers or documents for it? Thank you very much!

Best regards!

Jie Jiang
CNGL, School of Computing,
Dublin City University,
Glasnevin, Dublin 9.
Tel: +353 (0)1 700 6724

Re:[Moses-support] about the moses-chart reordering

Hi,

the best reference would be David Chiang’s Computational Linguistics article on hierarchical phrase-based models.

-phi

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Moses Support Digest:Dictonary use during training

52nlp — Wed, 24 Feb 2010 12:37:43 +0000

[Moses-support] Dictonary use during training

Hi all, I’m wondering if you would know where I can find an english to spanish parallel, word to word dictionary to complement my training corpus.

Also, from what I have searched I understand you can either add the dictionary words at the end of the corpus or use the giza option. I would like to try both, but for the giza option -d I see that the file format uses the word’s ids, then where will the real words (from the parallel dictionary) go? in the corpus as well? or in a separate file?

Any other suggestions for using a dictionary are welcome.

Thank you.

Re:[Moses-support] Dictonary use during training

re: adding dictionary entries, this is certainly a hack. but the standard trick is to pretend that the dictionary actually consists of tiny parallel sentences. you therefore just append each word-entry as a new sentence pair. don’t bother with that -d option.

Miles

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Moses Support Digest:2.718 in the phrase-table

52nlp — Tue, 23 Feb 2010 11:54:44 +0000

[Moses-support] 2.718 in the phrase-table

The exact purpose of the value 2.718 is what I am not sure of.As far as I know, the value is used for preferring the hypothesis with less phrases tothe one that use more phrases(words) for the same coverage of the source sentence duringthe prefix cost comparison inside priority queue( for the hyps that covers the same source range ).for example ) phrase_1 + phrase_2 + phrase_3 )to prefer hyp1 to hyp2 by multiplying hyp1 by 2.718.This is how I understand the use of the value, since longer phrase has empirically better translationthan the one made up of word-based translation.Is there any one who can confirm my belief, or to correct my conclusion?p.s. Is 2.718 the Euler’s number? If it is, why is the weight determined as such?

Re:[Moses-support] 2.718 in the phrase-table

The last weight from the phrase-table corresponds to the phrase penalty as explained in

http://www.statmt.org/moses/?n=FactoredTraining.ScorePhrases

This features benefits translations that use less units to decode.

It is the Euler’s number and it is determined as such because the decoder uses natural logarithms to add the weights, instead of multiplying probabilities. Hence, 2.718 will become 1 during decoding as a way of saying “I’m using yet another phrase unit to perform a translation”.

So you were quite right after all )

Re:[Moses-support] 2.718 in the phrase-table

Anecdotally, this feature also isn’t especially important, see e.g.:

http://www.mt-archive.info/AMTA-2006-Lopez.pdf

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Moses Support Digest:mert extractor

52nlp — Mon, 22 Feb 2010 13:19:14 +0000

[Moses-support] mert extractor

Another basic question: When trying to run the mert-moses-new.pl script it seems to require a mert script called extractor, but my training/cmert-0.5/ directory does not contain such a script.

the mert-moses-new.pl contains these two lines:

my $mert_extract_cmd = “$mertdir/extractor”;
my $mert_mert_cmd = “$mertdir/mert”

Where can I find the extractor or what could be the reason it didnt get included, when executing ”make release”.

Thanks!
Marce

Re:[Moses-support] mert extractor

Hi Marce

mert-moses-new.pl is designed to work with the new implementation of mert, which is found in the mert directory, beneath trunk. It doesn’t work with cmert,

regards
Barry

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Moses Support Digest:Translation from English to Foreign Language

52nlp — Mon, 08 Feb 2010 11:31:30 +0000

[Moses-support] Translation from English to Foreign Language

Is it possible to translate with moses from English to any Foreign language?

because I tried using moses, and at “Sanity Check Trained Model” step, the result of best translation always UNK (English-Foreign), but if I translate from Foreign-English, it’s work fine

I modified Train Phrase Model into:
nohup nice
tools/moses-scripts/scripts-YYYYMMDD-HHMM/training/train-factored-phrase-model.perl -scripts-root-dir tools/moses-scripts/scripts-YYYYMMDD-HHMM/ -root-dir
work -corpus work/corpus/news-commentary.lowercased -e en -f id -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm
0:3:/home/jschroe1/demo/work/lm/news-commentary.lm >&
work/training.out &

the original was:
nohup nice tools/moses-scripts/scripts-YYYYMMDD-HHMM/training/train-factored-phrase-model.perl -scripts-root-dir tools/moses-scripts/scripts-YYYYMMDD-HHMM/ -root-dir work -corpus work/corpus/news-commentary.lowercased -f fr -e en -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm 0:3:/home/jschroe1/demo/work/lm/news-commentary.lm >& work/training.out &

I’ve build language model for my foreign language, not English-language-model. Is there anything I missed here?

Laurent

Re:[Moses-support] Translation from English to Foreign Language

Dear Laurentia,

ss you can read somewhere in the moses manual
-the parameter “-f” tells you which is the source language
-the parameter “-e” tells you which is the target language

The names “-f” and “-e” do not relate with French and English specifically, but to generic languages

assuming that your parallel corpus have the suffixes “.fr” and “.en” for French and English respectively
the right command line to train an English-to-French system is

nohup nice tools/moses-scripts/scripts-YYYYMMDD-HHMM/training/train-factored-phrase-model.perl -scripts-root-dir tools/moses-scripts/scripts-YYYYMMDD-HHMM/ -root-dir work -corpus work/corpus/news-commentary.lowercased -f en -e fr -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm 0:3:/home/jschroe1/demo/work/lm/news-commentary.lm >& work/training.out &

Your modified command does not modify actually anything, but the source language which you set to “id”. Is “id” the suffix of your source language?

best regards,

Re:[Moses-support] Translation from English to Foreign Language

Thx nicola

“id” is the suffix for my target language

my bad, I always think that -f and -e refers to “French” and “English” and I never noted that when I read the manual before

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.