52nlp's Learning Notes » moses

Moses Support Digest: CALL FOR PAPERS – PBML

52nlp — Mon, 05 Jul 2010 14:16:57 +0000

[Moses-support] CALL FOR PAPERS – PBML

CALL FOR PAPERS:
OPEN SOURCE TOOLS FOR MACHINE TRANSLATION
The Fifth Machine Translation Marathon, which will take place September
13-18 in Le Mans, France, is hosting an Open Source Convention to
advance the state of the art in machine translation. The MT Marathon
is organised by the computer science laboratory of Le Mans (LIUM) and the
University of Le Mans on behalf of the EuroMatrixPlus Consortium.
We invite developers of open source tools to present their work and
submit a paper of up to 10 pages that (a) describes the underlying
methodology and (b) includes instructions how to use the tools.
We are looking for stand-alone tools and extensions of existing tools,
such as the Moses open source systems. Accepted papers will be
presented during the MT Marathon and published as a special issue of
the Prague Bulletin of Mathematical Linguistics (

http://ufal.mff.cuni.cz/pbml).

Possible topics:
* training of machine translation models
* machine translation decoders
* tuning of machine translation systems
* evaluation of machine translation
* visualization, annotation or debugging tools
* tools for human translators
* interfaces for web-based services or APIs
* extensions of existing tools
* other tools for machine translation
This is the third time that the MT Marathon will host the Open Source
Convention. The papers from previous years are available online:

http://ufal.mff.cuni.cz/pbml-91-100.html

Papers will be reviewed by two reviewers appointed by the program
committee
e.g. papers which will be marked as the best by the reviewers (with no
substantial
modifications required) will be printed as usual in the PBML journal in time
for the MT Marathon.
Others will be printed after requested modifications in the next PBML
issue.
Important dates:
Deadline for paper submission: August 1 2010
Notification of acceptance: August 7 2010
Camera-ready paper due: August 14 2010
Presentations: September 13-17 2010 (at the MT Marathon in Le Mans)
Camera-ready (next issue) : 15 December 2010
Please send full non-anonymous submissions in PDF to Philipp Koehn <
pkoehn at inf.ed.ac.uk>
and the full Xe(La)TeX source for technical pre-review to Ondřej
Bojar .
Please use the PBML style files from

http://ufal.mff.cuni.cz/pbml-instructions.html

(follow the “short paper” track instructions).
Program Committee
Philipp Koehn
Ondrej Bojar
Holger Schwenk
Loïc Barrault

Moses Support Digest: moses-irstlm memory racing with 5-gram lm

52nlp — Mon, 07 Jun 2010 12:39:39 +0000

[Moses-support] moses-irstlm memory racing with 5-gram lm

I’m troubleshooting a new moses system with these components:
1) GIZA++ (SVN rev 8, v 1.0.3)
2) IRSTLM (SVN rev 38, v 5.40.01)
3) Moses (SVN rev 3210, dated 4-26-2010)
4) Ubuntu-server 10.04 LTS 64-bit.
5) 3.4 Ghz Pentium-D with 4gb ram.

Using a 3-gram lm, the system works as expected. Training, tuning and
evaluation a small (135K pairs) en-nl subset of europarl.v5 work fine. BLEU
score was 23.

I then built a 5-gram model, edited the moses.ini config and started
mert-moses-new. It creates a filtered model, and then launches moses. The
memory usage grows and within 10 minutes, the system kills moses.

In both cases, the lm is only the target half of the bitext corpus, about
135K lines.

The moses.ini files:

[lmodel-file]
1 0 3 /media/models/irstlm/europarl.v5.mini/3-gram.nl.blm

[lmodel-file]
1 0 5 /media/models/irstlm/europarl.v5.mini/5-gram.nl.blm

I know of one other who has anyone the same problem with the 4-1-2010
moses build and irstlm from March/April last year.

Any suggestions? Could it be the new Ubuntu or the g++-4.4.1 compiler?

Thanks,
Tom

Re: [Moses-support] moses-irstlm memory racing with 5-gram lm

i’m not an expert on irstlm but i think you have to rename or softlink
the file so that the extension ends in .mm to minimise memory usage.

http://www.statmt.org/moses/?n=FactoredTraining.BuildingLanguageModel#ntoc7

otherwise it’ll just allocate as much memory as needed to load the LM file

Re: [Moses-support] moses-irstlm memory racing with 5-gram lm

Problem solved.

To review the symptoms, I ran the following two mert-moses-new.pl command
lines:

CASE 1:
nice mert-moses-new.pl
/media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/mert.en

/media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/mert.nl

/usr/local/lib/moses-irstlm/moses-cmd/src/moses

/media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/moses0.ini

–working-dir
/media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl
–rootdir /usr/local/lib/moses-irstlm/scripts
–mertdir=/usr/local/lib/moses-irstlm/mert
–nbest=50
–decoder-flags -v 0

CASE 2:
nice mert-moses-new.pl
/media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/mert.en

/media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/mert.nl

/usr/local/lib/moses-irstlm/moses-cmd/src/moses

/media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/moses0.ini

–working-dir
/media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl
–rootdir /usr/local/lib/moses-irstlm/scripts
–mertdir=/usr/local/lib/moses-irstlm/mert
–nbest=50
–decoder-flags -v 0

Only one line (the lmodel-file) was different in the respective starting
config files:

CASE 1:
/media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.3.en-nl/moses0.ini:
[ttable-file]
0 0 0 5
/media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase-table.gz
[lmodel-file]
1 0 3 /media/models/irstlm/europarl.v5.mini/3-gram.nl.blm.mm
[distortion-file]
0-0 wbe-msd-bidirectional-fe-allff 6
/media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering-table.wbe-msd-bidirectional-fe.gz

CASE 2:
/media/models/tables/europarl.v5.mini/en-nl/mert.irstlm.5.en-nl/moses0.ini:
[ttable-file]
0 0 0 5
/media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase-table.gz
[lmodel-file]
1 0 5 /media/models/irstlm/europarl.v5.mini/5-gram.nl.blm.mm
[distortion-file]
0-0 wbe-msd-bidirectional-fe-allff 6
/media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering-table.wbe-msd-bidirectional-fe.gz

In both cases, mert-moses-new.pl filtered the phrase table successfully.
In CASE 1, the tuning process continued and concluded with a final
moses.ini file with new weights. In CASE 2, however, mert-moses-new.pl
created run1.moses.ini. The moses process rapidly (less than 5 minutes)
consumed all RAM and virtual memory leaving nothing for other processes. It
never sent output to the run1.out file. The system killed moses and
mert-moses-new.pl. This occurred from the mert-moses-new.pl script or from
the command line using the run1.moses.ini file.

Furthermore, I changed run1.moses.ini to use the binarized phrase and
reordering tables:
0 0 0 5
/media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase-table.gz
changed to:
1 0 0 5
/media/models/tables/europarl.v5.mini/en-nl/model.en-nl/phrase-table

0-0 wbe-msd-bidirectional-fe-allff 6
/media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering-table.wbe-msd-bidirectional-fe.gz
changed to
0-0 wbe-msd-bidirectional-fe-allff 6
/media/models/tables/europarl.v5.mini/en-nl/model.en-nl/reordering-table.wbe-msd-bidirectional-fe

With this modified config from the command line (not mert-moses-new.pl),
moses loaded in seconds and translated stdin/stdout just fine. Only
configurations with the full .gz model and filtered model exhibited the
problems. The filtered model, by the way, is only 20 MB for phrase AND
reordering tables.

SOLUTION:
When I first built IRSTLM with MACHTYPE=x86_64, it created
$IRSTLM/bin/x86_64. Then, building moses using –with-irstlm=$IRSTLM
finished without fatal errors. I recently read the moses-support threads
about using a $SRILM/bin/i686 folder. So, I applied the same solution to
IRSTLM. I rebuilt IRSTLM and I created two symlinks:
ln -s $IRSTLM/bin/x86_64 $IRSTLM/bin/i686
ln -s $IRSTLM/lib/x86_64 $IRSTLM/lib/i686

Then, I rebuild moses –with-irstlm=$IRSTLM.

RESULTS: the mert-moses-new.pl script runs flawlessly with 3-gram and
5-gram IRSTLM language models and the exact same config files in CASE 1 and
CASE 2 above.

Go figure!

Hope this helps others.

Tom

Moses Support Digest: EMS table binarization

52nlp — Sun, 30 May 2010 15:21:25 +0000

[Moses-support] EMS: table binarization

Hi all,

I’m trying to run a system with a binary phrase table, and it doesn’t appear to be doing the binarization. Looking in experiment.meta the relevant section appears to be

binarize-config
in: config
out: bin-config
pass-unless: binarize-all
rerun-on-change: config
default-name: model/moses.bin.ini
template: $binarize-all $ttable-binarizer $rtable-binarizer OUT IN

I can specify ttable-binarizer and rtable-binarizer, but I can’t work out what the value of binarize-all should be. Any suggestions?

Thanks,
Suzy

Re: [Moses-support] EMS: table binarization

Hi,

this seems to be a rudimentary feature from ancient times that ceased to function properly. It should not be to hard to fix it. Let me know, if you need help with it.

-phi

Moses Support Digest: processPhraseTable in windows

52nlp — Mon, 24 May 2010 15:03:47 +0000

[Moses-support] processPhraseTable in windows

Dear All,

I would like to know how to run processPhraseTable in windows to make Binary Phrase Tables.

Best Regards,
Kyaw Kyaw Zin

Re: [Moses-support] processPhraseTable in windows

the same as on any other platform, but please read

http://www.statmt.org/moses/?n=Moses.FAQ#ntoc9

Moses Support Digest: compiling on Mac

52nlp — Fri, 21 May 2010 15:50:30 +0000

[Moses-support] compiling on Mac

I have a doubt about moses.

when I try to compile with make on Mac (Snow Leopard or Leopard), it gives me strange libtool errors (see below). It’s only recently been doing this so might be caused by software update. Its a clean install of OSX so there shouldn’t be resource problems. And it compile fine with xcode.

error:
———————————-
make
make all-recursive
Making all in moses/src
make all-am
/bin/sh ../../libtool –tag=CXX –mode=compile g++ -DHAVE_CONFIG_H -I.
-I../.. -W -Wall -ffor-scope -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES
-DTRACE_ENABLE=1 -g -O2 -MT AlignmentInfo.lo -MD -MP -MF
.deps/AlignmentInfo.Tpo -c -o AlignmentInfo.lo AlignmentInfo.cpp
../../libtool: fork: Resource temporarily unavailable
../../libtool: fork: Resource temporarily unavailable
../../libtool: fork: Resource temporarily unavailable
../../libtool: fork: Resource temporarily unavailable
../../libtool: fork: Resource temporarily unavailable
../../libtool: line 712: *** Warning: inferring the mode of operation is
deprecated.: command not found
../../libtool: line 713: *** Future versions of Libtool will require
-mode=MODE be specified.: command not found
../../libtool: line 745: : warning: cannot infer operation mode without
MODE-ARGS: command not found

Re: [Moses-support] compiling on Mac

Dear Hieu

The only trouble I had compiling on a Mac was I needed to install a recent version of automake. I note the doc [1] says

For OS X versions 10.4 and lower, you need to upgrade aclocal and
automake to at least version 1.9 (1.6 is the default in 10.4) and
set the variables ACLOCAL and AUTOMAKE in ./regenerate-makefiles.sh.

I was running 10.5 or 6 and I had to upgrade [2].

Best wishes

Ivan

[1] Moses Installation and Training Run-Through

http://www.statmt.org/moses_steps.html

[2] Installing Moses and irstlm on Mac OS X and Cygwin

https://www.bangor.ac.uk/~cbs007/mt/getting_started.html

Re: [Moses-support] compiling on Mac

ah, i found the problem & solution. (libtool on mac) != (libtool on linux)

have to install gnu libtool, then rename
libtoolize
in
*regenerate-makefiles.sh*
to
glibtoolize

command line is sooo much fun

Moses Support Digest: about CreateOnDiskPt parameter

52nlp — Thu, 20 May 2010 16:21:44 +0000

[Moses-support] about CreateOnDiskPt parameter

Hi all:

Could please anyone explain a bit about these two parameter of
CreateOnDiskPt:
[#source factors] [#target factors]
Because when I’m trying to binarize the New format of moses [version
2010/4/26], I have to use
[#source factors] = 1
[#target factors] =1
to make it run successfully.

However, the specification in moses.ini has a factor 0 for both source and
target.

Does it mean in the source code, now moses suppose the source/target words
as the first factor?

Best regards!

Re: [Moses-support] about CreateOnDiskPt parameter

hi jie

not sure what you mean. In CreateOnDiskPt, you specify the number of factors
(1). In the ini file, you specify which factor (0).

Moses Support Digest: experiment management system and Moses scripts

52nlp — Wed, 19 May 2010 15:07:31 +0000

[Moses-support] experiment management system and Moses scripts

Hi all,

I’ve just downloaded and installed the latest subversion revision (3261) and I’m trying out the new experiment management system, but the Moses scripts are causing me problems. When I point the config variable ‘moses-script-dir’ to ~/mosesdecoder/scripts (the scripts directory inside the repository), executing the ems crashes because BINDIR is not set in train-model.perl. However if I instead run ‘make release’ from ~/mosesdecoder/scripts and set moses-scripts-dir to point to the directory thus created, the ems crashes when checking the config because that scripts directory doesn’t include generic/mteval-v12.pl or the tokenizer and ems directories. Is the solution to update ~/mosesdecoder/scripts/released_files, or to change BINDIR in train-model.perl without running ‘make release’, or something else?

Thanks,
Suzy

[Moses-support]Re:experiment management system and Moses scripts

Hi,

thanks for trying out experiment.perl, please let me know of any other problems you encounter.

The two solutions you indicate are both correct. I just updated in the main branch ~/mosesdecoder/scripts/released_files so that should be up to date now.

-phi

Moses Support Digest: Regarding moses.weight-reused.ini

52nlp — Mon, 17 May 2010 15:36:04 +0000

[Moses-support] Regarding moses.weight-reused.ini

Hello

I have a question to which I have been wanting to know the answer, I
googled, I read but I am still not clear about the following: I replicated
the baseline, following these steps:
1. Preprocessing of training data
2. Actual training ( LM, RM and TM ) =>moses.ini
3. Translating
4. Evaluating
5. Tuning => moses.weight-reused.ini

So far all is ok, but now I need to improve my baseline by trying to find
the suitable parameter values. My question is: when I modify certain
parameters ( like distortion limit, stack size, translation-table etc) do I
use during the translation the moses.ini file or the moses.weigh-reused.
file. I know that the latter has the values for weight-t, weight-l,weight-d,
weight-w different, but I want to “play” with the other parameters as well.
With which one do I work for improvement on the system?

Thank you

Ligia

Moses Support Digest: Moses seems to hang

52nlp — Sat, 15 May 2010 16:29:25 +0000

[Moses-support] Moses seems to hang

I just built a new moses system with the newest releases of all
components:
1) GIZA++ (SVN rev 8, v 1.0.3)
2) IRSTLM (SVN rev 38, v 5.40.01)
3) Moses (SVN rev 3210, dated 4-26-2010)

My host system is the new Ubuntu-server 10.04 LTS 64-bit. I’ll be
contributing several patches as soon as I confirm they’re working.

The all components now builds successfully. The LM training goes smoothly,
as does the train-factored-phrase-model.perl training. Binarizing LM,
phrase and reordering tables goes as expected.

When I try to run my first test translations before tuning, Moses seems to
load the reordering table, phrase table and language models. Then, the
system does nothing. There is no output. It stays suspended until I hit
Ctrl-C.

Here’s the input:

dit is niet gebeurd . | moses -f ./model/moses.ini

Here’s the output:

Defined parameters (per moses.ini or switch):
config: ./model/moses.ini
distortion-file: 0-0 wbe-msd-bidirectional-fe-allff 6
./model/reordering-table
distortion-limit: 6
input-factors: 0
lmodel-file: 0 0 3 ./lm/irstlm-3.en.blm.mm
mapping: 0 T 0
ttable-file: 0 0 0 5 ./model/phrase-table
ttable-limit: 20
weight-d: 0.3 0.3 0.3 0.3 0.3 0.3 0.3
weight-l: 0.5000
weight-t: 0.2 0.2 0.2 0.2 0.2
weight-w: -1
Loading lexical distortion models…have 1 models
Creating lexical reordering…
weights: 0.300 0.300 0.300 0.300 0.300 0.300
binary file loaded, default OFF_T: -1
Start loading LanguageModel ./lm/irstlm-3.en.blm.mm : [0.000] seconds
Loading Internal LM: ./lm/irstlm-3.en.blm.mm
./lm/europarl.v5.mini-irstlm-3.en.blm.mm
Finished loading LanguageModels : [1.000] seconds
Start loading PhraseTable ./model/phrase-table : [1.000] seconds
filePath: ./model/phrase-table

((( this is where it stops until I hit Ctrl-C )))

I’ve never seen this before. Any suggestions?

Thanks,
Tom

Re: [Moses-support] Moses seems to hang

It looks like it is trying to use the internal language model instead of
the IRSTLM one. Check the line for your language model in model.ini.
Here is what I have for an IRSTLM 5-gram unfactored language model:

# language models: type(srilm/irstlm), factors, order, file
[lmodel-file]
1 0 5 /tmp/work/sq.blm

(This might not be related to your problem, but it struck me as odd at
any rate)

Regards,

Fran

Moses Support Digest: Copyright application issue for a Snippet Translation System using MOSES

52nlp — Wed, 12 May 2010 16:21:04 +0000

[Moses-support] Copyright application issue for a Snippet Translation System using MOSES

Dear Sir,

We have developed a reordered Phrase based SMT System with reordering of input snippets obtained from a search engine. The system translates English / Hindi snippets to the following Indian languages: Hindi/Bengali/Marathi/Tamil/Telugu/Punjabi. For developing this Snippet Translation system we have used MOSES Training Toolkit and Decoder and also the SRILM toolkit for generating the Language model in English .

The above system was developed by us as Member of the Project Team for R&D project entitled: Development of Cross Lingual Information Access System (CLIA) and the machine translation system is being copyrighted with Department of Information Technology (DIT), Government of India, New Delhi & Jadavpur University, Kolkata as the owners.

Please tell us whether we need to include the reference to MOSES in the application for registration of copyright and how we can do that.

Regards,

Sivaji

Re: [Moses-support] Copyright application issue for a Snippet Translation System using MOSES

Hi Sivaji

Moses is open-source software, licensed under the Gnu Lesser General Public License (LGPL). This means that you are free to use it as you wish in your own system, however the copyright to moses remains with its authors. The LGPL only really matters if you wish to distribute modified versions of moses, outside your organisation.

SRILM is different though. You should check the exact conditions on the SRILM website, but afaik it is only free for research use – there is a license fee for commercial use. If you want an open source language modelling toolkit, then use IRSTLM.

As regards the translations that are output by moses, I’m not sure what their exact status is, but I’m certain that the moses authors do not retain any copyright over these,

Hope that helps,

regards
Barry