I Love Natural Language Processing

[Moses-support] Issues with Score data

Hello

I am trying to tune my system ( baseline ) and I have been having the following error ( copied from my cmert.log file):

Seeding random numbers with system clock
Scorer config string:
Using scorer regularisation strategy: none
Using scorer regularisation window: 0
Using case preservation: 1
Using reference length strategy: closest
Data::score_type BLEU
Data::Scorer type from Scorer: BLEU
BleuScorer: 9
ScoreData: number_of_scores: 9
Loading Data from: statscore.data and features.data
loading feature data from features.data
loading score data from statscore.data
Data loaded : [0] seconds
terminate called after throwing an instance of ‘std::runtime_error’
what(): Score data is empty
/home/user/mtsw/trunk/scripts/training/cmert-0.5/enhanced-mert: line 47:
19558 Aborted $SCRIPTS_ROOTDIR/training/cmert-0.5/mert -d
$size

I don’t understand why it says that the score data is empty.
I have been trying to run my tunning.sh script for the last 3 days, everytime I ended up with errors after (max) 12h running.

Thanks for your help
Read the rest of this entry »

[Moses-support] Note that regr. test for ptable-filtering fails

Hi,

I just wanted to warn recent committers that the regression test for moses
ptable filtering fails (the built-in filtering, not the script
filter-model-given-input.perl).

Cheers, O.
Read the rest of this entry »

[Moses-support] TER evaluation tool

Hello All,

I want to know is there any TER evaluation tool which can be used for scoring moses output ?

Best Regards,
S.Bakhshaei
Read the rest of this entry »

[Moses-support] Any documentation about the Multiple Decoding Path functionality..

Hi All,

I have been using Moses for a couple of months now. I work in Domain Adaptation and I was using the Multiple Decoding Path functionality to combine multiple domain specific translation models.
While am comfortable using this functionality in my experiments, I was trying to know exactly how the mechanism runs in the back-end.
Is it possible to obtain any documentation on how exactly the Alternate
Multiple Decoding Path mechanism works.
Any paper or any pointers would be of great help.

Thanks and regards,

Pratyush Banerjee
Phd. Student
CNGL, School of Computing
Dublin City University
Ireland
Read the rest of this entry »

[Moses-support] cannot find -loolm

libtool: link: g++ -g -O2 -o lmbrgrid LatticeMBRGrid.o LatticeMBR.o IOWrapper.o -L/home/sripirakas/SMT/tools/srilm/lib/i686 ../../moses/src/.libs/libmoses.a -L/home/sripirakas/SMT/tools/moses/OnDiskPt/src -lOnDiskPt -loolm -ldstruct -lmisc -lz
/usr/bin/ld: cannot find -loolm
collect2: ld returned 1 exit status
make[2]: *** [moses] Error 1
make[2]: *** Waiting for unfinished jobs….
/usr/bin/ld: cannot find -loolm
collect2: ld returned 1 exit status
make[2]: *** [lmbrgrid] Error 1
make[2]: Leaving directory `/home/sripirakas/SMT/tools/moses/moses-cmd/src’
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/sripirakas/SMT/tools/moses’
make: *** [all] Error 2

Re: [Moses-support] cannot find -loolm

First check in /home/sripirakas/SMT/tools/srilm/lib/i686 to see if the
library oolm is compiled successfully.

If your machine’s architecture is x86-64, SRILM actually compiles its
library for x86 instead. You need to explicitly tell SRILM to target x86-64
– I usually just edit the srilm/sbin/machine-type script and change

else if (`uname -m` == x86_64) then
#set MACHINE_TYPE = i686-m64
set MACHINE_TYPE = i686

to

else if (`uname -m` == x86_64) then
set MACHINE_TYPE = i686-m64
#set MACHINE_TYPE = i686

And then do a clean build.

Liu Chang
National University of Singapore

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

[Moses-support] Experiment.perl publications & documentation

Hi,

At the MT Marathon, Jon and the other LoonyBin folks presented a paper describing the LoonyBin workflow manager. I was wondering if there’s any similar publication describing the experiment.perl workflow manager, and if there’s any tutorial or documentation on using it.

Thanks,
Lane

Re: [Moses-support] Experiment.perl publications & documentation

Hi,

this is still work in progress – the documentation at the time of
the MT Marathon is here:

http://www.statmt.org/mtm4/?n=Main.EMSDocumentation

-phi

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

[Moses-support] Moses decoder on windows freezes after 5 sentences

Dear All

I have this little python script to run quick tests on moses models (see below). The same script works on unix and windows, given the right paths. I’m also using the same basic idea as a basis for the moses thread in a translation GUI.

Essentially, the MosesCMD class starts up moses in a subprocess, and subsequent calls to translate() send data in on the subprocess’ stdin and receive translations on its stdout.

Anyway, on unix (actually MacOSX) this will run and run (haven’t tested it on 1000s of sentences, but plenty); on windows, it packs in after five sentences (sometimes fewer) and completely freezes. Automatically killing the subprocess and starting a new one after every three sentences works sometimes, but sometimes (eg when it’s powering the moses thread in a GUI) actually makes things worse.

The moses decoder I’m using on windows I compiled under cygwin. Both decoders are using the same irstlm language models.

I’m not very familiar with windows. Does anyone have any ideas what might be causing the windows moses to keel over? Is it something to do with the way windows manages threads/processes/memory? Anything I can do? Unfortunately has to run on windows.

With thanks and best wishes

Ivan

Here’s the script:
Read the rest of this entry »

[Moses-support] Moses release

Hiya Mosers and Mosettes,

It’s been a year since the last release& there’s been lots of changes, by lots of people, that we thought you should know about.

A new release tar ball and zip file are on sourceforge, or svn update as usual

https://sourceforge.net/projects/mosesdecoder/

Also, there is likely to be big changes in the next month as we merge the hierarchical/syntax branch into trunk. Please avoid svn up after today, and double check with someone else before committing large chunks of code to the trunk.

Changes since the last time:
1. minor bug fixes& tweaks, especially to the decoder, MERT scripts
(thanks to too many people to mention)
2. fixes to make decoder compile with most versions of gcc, Visual
studio and other compilers (thanks to Tom Hoar, Jean-Bapist Fouet).
3. multi-threaded decoder (thanks to Barry Haddow)
4. update for IRSTLM (thanks to nicola bertoldi& Marcello Federico)
5. run mert on a subset of features (thanks to nicola bertoldi)
6. Training using different alignment models (thanks to Mark Fishel)
7. “a handy script to get many translations from Google” (thanks to
Ondrej Bojar)
8. Lattice MBR (thanks to Abhishek Arun& Barry Haddow)
9 . Option to compile moses as a dynamic library (thanks to
Jean-Bapist Fouet).
10. hierarchical re-ordering model (thanks to Christian Harmeier,
Sara Styme, Nadi, Marcello, Ankit Srivastava, Gabriele Antonio Musillo,
Philip Williams, Barry Haddow).
11. Global Lexical re-ordering model (thanks to Philipp Koehn)
12. Experiment.perl scripts for automating the whole MT pipeline (thanks to Philipp Koehn)

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

[Moses-support] different bleu scores from nist and moses scripts

Dear list,

I am getting different BLEU scores from the NIST mteval script (version) and the multi-bleu.perl script within Moses’s distribution for the same reference and hypothesis translations — even the individual n-gram precisions are different:

BLEU = 16.80, 53.0/26.2/13.4/6.4 (BP=0.905, ratio=0.909, hyp_len=281, ref_len=309)

and

BLEU score = 0.1681 for system “x”

Individual N-gram scoring
1-gram 2-gram 3-gram 4-gram 5-gram 6-gram 7-gram
8-gram 9-gram
—— —— —— —— —— —— ——
—— ——
BLEU: 0.5246 0.2591 0.1326 0.0630 0.0328 0.0213 0.0133
0.0046 0.0000 “x”

The files that produced the scores are here: mtj.ut.ee/diffbleu.tgz .

Does everyone else get different scores? Can anyone suggest a reason for that? It’s not the smoothing of the NIST script, both support UTF8 i/o, etc; so I’m out of ideas, and before comparing the implementations I wanted to ask for opinions.

Thanks in advance,
Mark
Read the rest of this entry »

[Moses-support] Word Alignment – Moses?

Hi..
Couldy you please tell me if I can use Moses as a tool like Giza++ for the purpose word alignment. If yes could you give me the link of where can I get further refernce wrt this. Could you tell me any other word alignment tool for parallel corpora which could come to my aid apart from Giza++, NATool.

Thank you for your time

Mohnish
Read the rest of this entry »