[Moses-support] prune-lm in endless loop
Hi,
I created a 3-gram LM with the irstlm toolkit (5.0.22). The LM has about 25M entries:
ngram 1= 300209
ngram 2= 4864097
ngram 3= 20336549
I tried to prune it with prune-lm on a Linux machine.
prune-lm –threshold=1e-6,1e-6 sun.irstlm.gz sun.pruned.irlstlm &> x.out
In the out x.out I get repeated error messages
ng: qu 0 ts=1.00059 tbs=0.0196106 k=0 ns=20
probably more than 100M identical ones. After running the pruning over night the stderr output reached 100GB size and I stopped the process.
Just looking at the source code I assume that lmtable::wdprune() loops endless over the “prune:” goto statement. Are there any problems with the pscale() routine?
Any hints where to look at are highly appreciated.
best regards
Christof
Re:[Moses-support] prune-lm in endless loop
Hi,
just for the records, recompiling prune-lm with additionally setting the compiler option
-fno-strict-aliasing
solved the problem. It seems gcc 4.1.2 didn’t like the magic casting that’s used in some of the source files.
Is there a place where these kind of issues are documented?
best regards
Christof
NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.
Related posts:
- Moses Support Digest: moses_chart and recaser phrase-table
- Moses Support Digest:moses threads compilation problem with RandLM
- Moses Support Digest: moses-irstlm memory racing with 5-gram lm
- Moses Support Digest: Moses on the iPhone
- Moses Support Digest:ConfusionNet GetSubString error when using lattice with UTF8 input
- Moses Support Digest: Moses seems to hang
- Moses Support Digest:CreateBerkeleyPt and On-Disk Rule Table
- Moses Support Digest:RDBMS for the decoder
- Moses Support Digest:Aligned phrase counts
- Moses Support Digest:Moses compilation problem on fedora 11