I Love Natural Language Processing

[Moses-support] moses-irstlm memory racing with 5-gram lm

I’m troubleshooting a new moses system with these components:
1) GIZA++ (SVN rev 8, v 1.0.3)
2) IRSTLM (SVN rev 38, v 5.40.01)
3) Moses (SVN rev 3210, dated 4-26-2010)
4) Ubuntu-server 10.04 LTS 64-bit.
5) 3.4 Ghz Pentium-D with 4gb ram.

Using a 3-gram lm, the system works as expected. Training, tuning and
evaluation a small (135K pairs) en-nl subset of europarl.v5 work fine. BLEU
score was 23.

I then built a 5-gram model, edited the moses.ini config and started
mert-moses-new. It creates a filtered model, and then launches moses. The
memory usage grows and within 10 minutes, the system kills moses.

In both cases, the lm is only the target half of the bitext corpus, about
135K lines.

The moses.ini files:

[lmodel-file]
1 0 3 /media/models/irstlm/europarl.v5.mini/3-gram.nl.blm

[lmodel-file]
1 0 5 /media/models/irstlm/europarl.v5.mini/5-gram.nl.blm

I know of one other who has anyone the same problem with the 4-1-2010
moses build and irstlm from March/April last year.

Any suggestions? Could it be the new Ubuntu or the g++-4.4.1 compiler?

Thanks,
Tom
Read the rest of this entry »

[Moses-support] Moses Support Digest:A translation chain prototype with Moses + IRSTLM

http://code.google.com/p/moses-for-mere-mortals/

This site offers a set of 3 scripts that, together, create a basic translation chain prototype (with Moses + IRSTLM) able of processing very large corpora. The idea is to help build a translation chain for the real world, but it should also enable a quick evaluation of Moses for actual translation work and guide users in their first steps of using Moses. Read the rest of this entry »