I Love Natural Language Processing

[Moses-support] Moses release

Hiya Mosers and Mosettes,

It’s been a year since the last release& there’s been lots of changes, by lots of people, that we thought you should know about.

A new release tar ball and zip file are on sourceforge, or svn update as usual

https://sourceforge.net/projects/mosesdecoder/

Also, there is likely to be big changes in the next month as we merge the hierarchical/syntax branch into trunk. Please avoid svn up after today, and double check with someone else before committing large chunks of code to the trunk.

Changes since the last time:
1. minor bug fixes& tweaks, especially to the decoder, MERT scripts
(thanks to too many people to mention)
2. fixes to make decoder compile with most versions of gcc, Visual
studio and other compilers (thanks to Tom Hoar, Jean-Bapist Fouet).
3. multi-threaded decoder (thanks to Barry Haddow)
4. update for IRSTLM (thanks to nicola bertoldi& Marcello Federico)
5. run mert on a subset of features (thanks to nicola bertoldi)
6. Training using different alignment models (thanks to Mark Fishel)
7. “a handy script to get many translations from Google” (thanks to
Ondrej Bojar)
8. Lattice MBR (thanks to Abhishek Arun& Barry Haddow)
9 . Option to compile moses as a dynamic library (thanks to
Jean-Bapist Fouet).
10. hierarchical re-ordering model (thanks to Christian Harmeier,
Sara Styme, Nadi, Marcello, Ankit Srivastava, Gabriele Antonio Musillo,
Philip Williams, Barry Haddow).
11. Global Lexical re-ordering model (thanks to Philipp Koehn)
12. Experiment.perl scripts for automating the whole MT pipeline (thanks to Philipp Koehn)

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

[Moses-support] different bleu scores from nist and moses scripts

Dear list,

I am getting different BLEU scores from the NIST mteval script (version) and the multi-bleu.perl script within Moses’s distribution for the same reference and hypothesis translations — even the individual n-gram precisions are different:

BLEU = 16.80, 53.0/26.2/13.4/6.4 (BP=0.905, ratio=0.909, hyp_len=281, ref_len=309)

and

BLEU score = 0.1681 for system “x”

Individual N-gram scoring
1-gram 2-gram 3-gram 4-gram 5-gram 6-gram 7-gram
8-gram 9-gram
—— —— —— —— —— —— ——
—— ——
BLEU: 0.5246 0.2591 0.1326 0.0630 0.0328 0.0213 0.0133
0.0046 0.0000 “x”

The files that produced the scores are here: mtj.ut.ee/diffbleu.tgz .

Does everyone else get different scores? Can anyone suggest a reason for that? It’s not the smoothing of the NIST script, both support UTF8 i/o, etc; so I’m out of ideas, and before comparing the implementations I wanted to ask for opinions.

Thanks in advance,
Mark
Read the rest of this entry »

[Moses-support] Word Alignment – Moses?

Hi..
Couldy you please tell me if I can use Moses as a tool like Giza++ for the purpose word alignment. If yes could you give me the link of where can I get further refernce wrt this. Could you tell me any other word alignment tool for parallel corpora which could come to my aid apart from Giza++, NATool.

Thank you for your time

Mohnish
Read the rest of this entry »

[Moses-support] About mert-moses in mose-chart

Dear Sir,

When I use mert-moses in Mose-chart with the command:

nohop nice ./mert-moses.pl working-dir/tunin /nc-dev2007.lowercased.fr
working-dir/tuning/lowercased.en
/home/i8001/s0810207/Mose-chartN/mose-chart/moses-cmd/src/moses working-dir/model/moses.ini
–working-dir working-dir/tuning –rootdir /home/i8001/s0810207/Mose-chartN/bin/moses-scripts/scripts-20100302-1240/
–decoder-flags “-v 0″ >& working-dir/tuning/mert.out

I met problem I can’t use hierarchical rules and lexicalised re-ordering at the same time.

I send moses.ini and problem in the attached files

Can you solve this problem for me

A lot of thanks and best regards

Bui Hung Thanh
Read the rest of this entry »

[Moses-support] BLEU_error

Dear all,

I tried BLEU for evaluation.It is giving an error.Can anyone help me to rectify this error.I am copy pasting that error below. After training and testing i directly tried evaluation .i skipped the steps recaser, detokenizier, tuning etc..Are these steps necessary?.

The error is………….FATAL INPUT ERROR: no ‘src_set’ tag in src_file
‘/root/Desktop/tools/data/src.
……….

Thanking You

with regards

rashli

Re:[Moses-support] BLEU_error

Hi,

You may confirm your xml wrapping of “src_file” is correct. It’s like

…

Ragards,
–
Hwidong Na
KLE lab, POSTECH, KOREA

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

[Moses-support] About moses-chart

Dir sir,
When I train rule table for moses chart, at the end of rule I saw two parameter (see rule example)

! ” . [X] ||| ! ” [X] ||| ||| 0.333333 0.004318 1 0.0854249 2.718 ||| 1.5 0.5

Could you please explain the meaning of tow parameter.
Thank you!

–
Thu.
Read the rest of this entry »

[Moses-support] mt3_chart compilation error

Hi,

When trying to compile mt3_chart i get a compilation error for PhraseDictionaryDynSuffixArray.o, what could be the cause/solution for this, could it be caused by adding -std=c++0x to the AM_CPPFLAGS (ref: http://www.mail-archive.com/moses-support@mit.edu/msg01483.html)? Im compiling with gcc4.4.

g++ -DHAVE_CONFIG_H -I. -I../.. -W -Wall -ffor-scope -D_FILE_OFFSET_BITS=64
-D_LARGE_FILES -std=c++0x -I/home/marce/srilm64/include
-I/home/marce/irstlm/include -g -O2 -MT PhraseDictionaryDynSuffixArray.o
-MD -MP -MF .deps/PhraseDictionaryDynSuffixArray.Tpo -c -o
PhraseDictionaryDynSuffixArray.o PhraseDictionaryDynSuffixArray.cpp
In file included from PhraseDictionaryDynSuffixArray.h:5,
from PhraseDictionaryDynSuffixArray.cpp:1:
DynSuffixArray.h: In member function âvoid
Moses::DynSuffixArray::printAuxArrays()â:
DynSuffixArray.h:43: warning: comparison between signed and unsigned integer
expressions
In file included from PhraseDictionaryDynSuffixArray.cpp:1:
PhraseDictionaryDynSuffixArray.h: In member function âbool
Moses::ScoresComp::operator()(const Moses::Scores&, const Moses::Scores&)
constâ:
PhraseDictionaryDynSuffixArray.h:61: error: âtypeofâ was not declared in
this scope
PhraseDictionaryDynSuffixArray.h:61: error: expected â;â before âitrâ
PhraseDictionaryDynSuffixArray.h:61: error: âitrâ was not declared in this
scope
PhraseDictionaryDynSuffixArray.h:62: error: expected â;â before âitrâ
PhraseDictionaryDynSuffixArray.h:62: error: âitrâ was not declared in this
scope

Thanks,
Marce
Read the rest of this entry »

[Moses-support] SRILM installation problem

Hi All,

I had a problem to install Moses and posted it here but no response is coming out yet. Can any one help me where to post this. I appreciate you guys for any help.

Mulu G.

Hi All,

I was trying to install srilm on Debian linux 5 lenny, MACHINE_TYPE= i686, with gcc version v 4.3.2. I have changed the Makefile according the manuals and without success. So when I try to enter #make World, it produces the following message

/bin/sh: /root/srilm: is a directory
mkdir include lib bin
mkdir: cannot create directory `include’: File exists
mkdir: cannot create directory `lib’: File exists
mkdir: cannot create directory `bin’: File exists
make: [dirs] Error 1 (ignored)
make init
/bin/sh: /root/srilm: is a directory
make[1]: Entering directory `/root/srilm’
for subdir in misc dstruct lm flm lattice utils; do
(cd $subdir/src; make SRILM=/root/srilm [SRILM] MACHINE_TYPE=
OPTION= MAKE_PIC= init) || exit 1;
done
make[2]: Entering directory `/root/srilm/misc/src’
/root/srilm/common/Makefile.common.variables:96:
/root/srilm/common/Makefile.machine.: No such file or directory
make[2]: *** No rule to make target `/root/srilm/common/Makefile.machine.’.
Stop.
make[2]: Leaving directory `/root/srilm/misc/src’
make[1]: *** [init] Error 1
make[1]: Leaving directory `/root/srilm’
make: *** [World] Error 2

Could you please help me in sorting out the problem and how to fix it.

Thank you all,

Mulu G.
Read the rest of this entry »