Moses Support Digest: TER evaluation tool

Posted on April 20, 2010 by 52nlp

[Moses-support] TER evaluation tool

Hello All,

I want to know is there any TER evaluation tool which can be used for scoring moses output ?

Best Regards,
S.Bakhshaei
Continue reading →

Posted in Moses, SMT | Tagged , , , | Leave a comment

Moses Support Digest: Any documentation about the Multiple Decoding Path functionality

Posted on April 15, 2010 by 52nlp

[Moses-support] Any documentation about the Multiple Decoding Path functionality..

Hi All,

I have been using Moses for a couple of months now. I work in Domain Adaptation and I was using the Multiple Decoding Path functionality to combine multiple domain specific translation models.
While am comfortable using this functionality in my experiments, I was trying to know exactly how the mechanism runs in the back-end.
Is it possible to obtain any documentation on how exactly the Alternate
Multiple Decoding Path mechanism works.
Any paper or any pointers would be of great help.

Thanks and regards,

Pratyush Banerjee
Phd. Student
CNGL, School of Computing
Dublin City University
Ireland
Continue reading →

Posted in Moses, SMT | Tagged , , , | Leave a comment

Moses Support Digest: cannot find -loolm

Posted on April 10, 2010 by 52nlp

[Moses-support] cannot find -loolm

libtool: link: g++ -g -O2 -o lmbrgrid LatticeMBRGrid.o LatticeMBR.o IOWrapper.o -L/home/sripirakas/SMT/tools/srilm/lib/i686 ../../moses/src/.libs/libmoses.a -L/home/sripirakas/SMT/tools/moses/OnDiskPt/src -lOnDiskPt -loolm -ldstruct -lmisc -lz
/usr/bin/ld: cannot find -loolm
collect2: ld returned 1 exit status
make[2]: *** [moses] Error 1
make[2]: *** Waiting for unfinished jobs….
/usr/bin/ld: cannot find -loolm
collect2: ld returned 1 exit status
make[2]: *** [lmbrgrid] Error 1
make[2]: Leaving directory `/home/sripirakas/SMT/tools/moses/moses-cmd/src’
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/sripirakas/SMT/tools/moses’
make: *** [all] Error 2

Re: [Moses-support] cannot find -loolm

First check in /home/sripirakas/SMT/tools/srilm/lib/i686 to see if the
library oolm is compiled successfully.

If your machine’s architecture is x86-64, SRILM actually compiles its
library for x86 instead. You need to explicitly tell SRILM to target x86-64
– I usually just edit the srilm/sbin/machine-type script and change

else if (`uname -m` == x86_64) then
#set MACHINE_TYPE = i686-m64
set MACHINE_TYPE = i686

to

else if (`uname -m` == x86_64) then
set MACHINE_TYPE = i686-m64
#set MACHINE_TYPE = i686

And then do a clean build.

Liu Chang
National University of Singapore

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Posted in Moses, SMT | Tagged , , , , | Leave a comment

Moses Support Digest: Experiment.perl publications & documentation

Posted on April 7, 2010 by 52nlp

[Moses-support] Experiment.perl publications & documentation

Hi,

At the MT Marathon, Jon and the other LoonyBin folks presented a paper describing the LoonyBin workflow manager. I was wondering if there’s any similar publication describing the experiment.perl workflow manager, and if there’s any tutorial or documentation on using it.

Thanks,
Lane

Re: [Moses-support] Experiment.perl publications & documentation

Hi,

this is still work in progress – the documentation at the time of
the MT Marathon is here:

http://www.statmt.org/mtm4/?n=Main.EMSDocumentation

-phi

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Posted in Moses, SMT | Tagged , , , | Leave a comment

Moses Support Digest: Moses decoder on windows freezes after 5 sentences

Posted on April 6, 2010 by 52nlp

[Moses-support] Moses decoder on windows freezes after 5 sentences

Dear All

I have this little python script to run quick tests on moses models (see below). The same script works on unix and windows, given the right paths. I’m also using the same basic idea as a basis for the moses thread in a translation GUI.

Essentially, the MosesCMD class starts up moses in a subprocess, and subsequent calls to translate() send data in on the subprocess’ stdin and receive translations on its stdout.

Anyway, on unix (actually MacOSX) this will run and run (haven’t tested it on 1000s of sentences, but plenty); on windows, it packs in after five sentences (sometimes fewer) and completely freezes. Automatically killing the subprocess and starting a new one after every three sentences works sometimes, but sometimes (eg when it’s powering the moses thread in a GUI) actually makes things worse.

The moses decoder I’m using on windows I compiled under cygwin. Both decoders are using the same irstlm language models.

I’m not very familiar with windows. Does anyone have any ideas what might be causing the windows moses to keel over? Is it something to do with the way windows manages threads/processes/memory? Anything I can do? Unfortunately has to run on windows.

With thanks and best wishes

Ivan

Here’s the script:
Continue reading →

Posted in Moses, SMT | Tagged , , , , | Leave a comment

Moses Support Digest: Moses release

Posted on March 29, 2010 by 52nlp

[Moses-support] Moses release

Hiya Mosers and Mosettes,

It’s been a year since the last release& there’s been lots of changes, by lots of people, that we thought you should know about.

A new release tar ball and zip file are on sourceforge, or svn update as usual

https://sourceforge.net/projects/mosesdecoder/

Also, there is likely to be big changes in the next month as we merge the hierarchical/syntax branch into trunk. Please avoid svn up after today, and double check with someone else before committing large chunks of code to the trunk.

Changes since the last time:
1. minor bug fixes& tweaks, especially to the decoder, MERT scripts
(thanks to too many people to mention)
2. fixes to make decoder compile with most versions of gcc, Visual
studio and other compilers (thanks to Tom Hoar, Jean-Bapist Fouet).
3. multi-threaded decoder (thanks to Barry Haddow)
4. update for IRSTLM (thanks to nicola bertoldi& Marcello Federico)
5. run mert on a subset of features (thanks to nicola bertoldi)
6. Training using different alignment models (thanks to Mark Fishel)
7. “a handy script to get many translations from Google” (thanks to
Ondrej Bojar)
8. Lattice MBR (thanks to Abhishek Arun& Barry Haddow)
9 . Option to compile moses as a dynamic library (thanks to
Jean-Bapist Fouet).
10. hierarchical re-ordering model (thanks to Christian Harmeier,
Sara Styme, Nadi, Marcello, Ankit Srivastava, Gabriele Antonio Musillo,
Philip Williams, Barry Haddow).
11. Global Lexical re-ordering model (thanks to Philipp Koehn)
12. Experiment.perl scripts for automating the whole MT pipeline (thanks to Philipp Koehn)

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Posted in Moses, SMT | Tagged , , , | Leave a comment

Moses Support Digest: different bleu scores from nist and moses scripts

Posted on March 26, 2010 by 52nlp

[Moses-support] different bleu scores from nist and moses scripts

Dear list,

I am getting different BLEU scores from the NIST mteval script (version) and the multi-bleu.perl script within Moses’s distribution for the same reference and hypothesis translations — even the individual n-gram precisions are different:

BLEU = 16.80, 53.0/26.2/13.4/6.4 (BP=0.905, ratio=0.909, hyp_len=281, ref_len=309)

and

BLEU score = 0.1681 for system “x”

Individual N-gram scoring
1-gram 2-gram 3-gram 4-gram 5-gram 6-gram 7-gram
8-gram 9-gram
—— —— —— —— —— —— ——
—— ——
BLEU: 0.5246 0.2591 0.1326 0.0630 0.0328 0.0213 0.0133
0.0046 0.0000 “x”

The files that produced the scores are here: mtj.ut.ee/diffbleu.tgz .

Does everyone else get different scores? Can anyone suggest a reason for that? It’s not the smoothing of the NIST script, both support UTF8 i/o, etc; so I’m out of ideas, and before comparing the implementations I wanted to ask for opinions.

Thanks in advance,
Mark
Continue reading →

Posted in Moses, SMT | Tagged , , , | Leave a comment

Moses Support Digest: Word Alignment – Moses

Posted on March 19, 2010 by 52nlp

[Moses-support] Word Alignment – Moses?

Hi..
Couldy you please tell me if I can use Moses as a tool like Giza++ for the purpose word alignment. If yes could you give me the link of where can I get further refernce wrt this. Could you tell me any other word alignment tool for parallel corpora which could come to my aid apart from Giza++, NATool.

Thank you for your time

Mohnish
Continue reading →

Posted in Moses, SMT | Tagged , , , | Leave a comment

Moses Support Digest: About mert-moses in mose-chart

Posted on March 16, 2010 by 52nlp

[Moses-support] About mert-moses in mose-chart

Dear Sir,

When I use mert-moses in Mose-chart with the command:

nohop nice ./mert-moses.pl working-dir/tunin /nc-dev2007.lowercased.fr
working-dir/tuning/lowercased.en
/home/i8001/s0810207/Mose-chartN/mose-chart/moses-cmd/src/moses working-dir/model/moses.ini
–working-dir working-dir/tuning –rootdir /home/i8001/s0810207/Mose-chartN/bin/moses-scripts/scripts-20100302-1240/
–decoder-flags “-v 0″ >& working-dir/tuning/mert.out

I met problem I can’t use hierarchical rules and lexicalised re-ordering at the same time.

I send moses.ini and problem in the attached files

Can you solve this problem for me

A lot of thanks and best regards

Bui Hung Thanh
Continue reading →

Posted in Moses, SMT | Tagged , , , | Leave a comment

Moses Support Digest: BLEU_error

Posted on March 12, 2010 by 52nlp

[Moses-support] BLEU_error

Dear all,

I tried BLEU for evaluation.It is giving an error.Can anyone help me to rectify this error.I am copy pasting that error below. After training and testing i directly tried evaluation .i skipped the steps recaser, detokenizier, tuning etc..Are these steps necessary?.

The error is………….FATAL INPUT ERROR: no ‘src_set’ tag in src_file
‘/root/Desktop/tools/data/src.
……….

Thanking You

with regards

rashli

Re:[Moses-support] BLEU_error

Hi,

You may confirm your xml wrapping of “src_file” is correct. It’s like



…

Ragards,
–
Hwidong Na
KLE lab, POSTECH, KOREA

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Posted in Moses, SMT | Tagged , , , | Leave a comment