I Love Natural Language Processing

[Moses-support] prune-lm in endless loop

Hi,

I created a 3-gram LM with the irstlm toolkit (5.0.22). The LM has about 25M entries:

ngram 1= 300209
ngram 2= 4864097
ngram 3= 20336549

I tried to prune it with prune-lm on a Linux machine.

prune-lm –threshold=1e-6,1e-6 sun.irstlm.gz sun.pruned.irlstlm &> x.out

In the out x.out I get repeated error messages

ng: qu 0 ts=1.00059 tbs=0.0196106 k=0 ns=20

probably more than 100M identical ones. After running the pruning over night the stderr output reached 100GB size and I stopped the process.

Just looking at the source code I assume that lmtable::wdprune() loops endless over the “prune:” goto statement. Are there any problems with the pscale() routine?

Any hints where to look at are highly appreciated.

best regards
Christof


Re:[Moses-support] prune-lm in endless loop

Hi,

just for the records, recompiling prune-lm with additionally setting the compiler option

-fno-strict-aliasing

solved the problem. It seems gcc 4.1.2 didn’t like the magic casting that’s used in some of the source files.

Is there a place where these kind of issues are documented?

best regards
Christof

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Related posts:

1. Moses Support Digest: moses_chart and recaser phrase-table
2. Moses Support Digest:moses threads compilation problem with RandLM
3. Moses Support Digest: moses-irstlm memory racing with 5-gram lm
4. Moses Support Digest: Moses on the iPhone
5. Moses Support Digest:ConfusionNet GetSubString error when using lattice with UTF8 input
6. Moses Support Digest:mt3_chart compilation error
7. Moses Support Digest:Moses compilation problem on fedora 11
8. Moses Support Digest: Moses seems to hang
9. Moses Support Digest:CreateBerkeleyPt and On-Disk Rule Table
10. Moses Support Digest: about CreateOnDiskPt parameter