Hi all,

I’m trying to run a system with a binary phrase table, and it doesn’t appear to be doing the binarization. Looking in experiment.meta the relevant section appears to be

binarize-config
in: config
out: bin-config
pass-unless: binarize-all
rerun-on-change: config
default-name: model/moses.bin.ini
template: $binarize-all $ttable-binarizer $rtable-binarizer OUT IN

I can specify ttable-binarizer and rtable-binarizer, but I can’t work out what the value of binarize-all should be. Any suggestions?

Thanks,
Suzy

Re: [Moses-support] EMS: table binarization

Hi,

this seems to be a rudimentary feature from ancient times that ceased to function properly. It should not be to hard to fix it. Let me know, if you need help with it.

-phi

Related posts:

1. Moses Support Digest:Alignment information from binary phrase table
2. Moses Support Digest:CreateBerkeleyPt and On-Disk Rule Table
3. Moses Support Digest:2.718 in the phrase-table
4. Moses Support Digest: Moses seems to hang
5. Moses Support Digest: moses-irstlm memory racing with 5-gram lm
6. Moses Support Digest:Reading binary phrase table from the disk
7. Moses Support Digest: moses_chart and recaser phrase-table
8. Moses Support Digest:ConfusionNet GetSubString error when using lattice with UTF8 input
9. Moses Support Digest:Tuning failure with Language model type unknown
10. Moses Support Digest:word lattice and multiple translation tables optimization problem

The exact purpose of the value 2.718 is what I am not sure of.As far as I know, the value is used for preferring the hypothesis with less phrases tothe one that use more phrases(words) for the same coverage of the source sentence duringthe prefix cost comparison inside priority queue( for the hyps that covers the same source range ).for example ) phrase_1 + phrase_2 + phrase_3 )to prefer hyp1 to hyp2 by multiplying hyp1 by 2.718.This is how I understand the use of the value, since longer phrase has empirically better translationthan the one made up of word-based translation.Is there any one who can confirm my belief, or to correct my conclusion?p.s. Is 2.718 the Euler’s number? If it is, why is the weight determined as such?

Re:[Moses-support] 2.718 in the phrase-table

The last weight from the phrase-table corresponds to the phrase penalty as explained in

http://www.statmt.org/moses/?n=FactoredTraining.ScorePhrases

This features benefits translations that use less units to decode.

It is the Euler’s number and it is determined as such because the decoder uses natural logarithms to add the weights, instead of multiplying probabilities. Hence, 2.718 will become 1 during decoding as a way of saying “I’m using yet another phrase unit to perform a translation”.

So you were quite right after all )

Re:[Moses-support] 2.718 in the phrase-table

Anecdotally, this feature also isn’t especially important, see e.g.:

http://www.mt-archive.info/AMTA-2006-Lopez.pdf

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Related posts:

1. Moses Support Digest:Alignment information from binary phrase table
2. Moses Support Digest:Reading binary phrase table from the disk
3. Moses Support Digest: moses_chart and recaser phrase-table
4. Moses Support Digest:Aligned phrase counts
5. Moses Support Digest:CreateBerkeleyPt and On-Disk Rule Table
6. Moses Support Digest: EMS table binarization
7. Moses Support Digest: Regarding moses.weight-reused.ini
8. Moses Support Digest: Moses seems to hang
9. Moses Support Digest:ConfusionNet GetSubString error when using lattice with UTF8 input
10. Moses Support Digest:POS translation