Moses Support Digest:CreateBerkeleyPt and On-Disk Rule Table

[Moses-support] CreateBerkeleyPt and On-Disk Rule Table

Hi folks!

I am trying to use on disk rule table

cd CreateBerkeleyPt/src/
[svp at mt src]$ make

g++ -g -O2 -o CreateBerkeleyPt Main.o -L../../BerkeleyPt/src
-L../../moses/src -lBerkeleyPt -lmoses -lz -ldb_cxx -lpthread
../../moses/src/libmoses.a(LanguageModelSRI.o): In function
`Moses::LanguageModelSRI::CreateFactors()’:
/home/svp/mosesdecoder/moses-chart/moses/src/LanguageModelSRI.cpp:85:
undefined reference to `VocabIter::VocabIter(Vocab const&, bool)’
../../moses/src/libmoses.a(LanguageModelSRI.o): In function
`VocabIter::next()’:
/home/svp/transl/srilm/srilm2009/include/Vocab.h:275: undefined reference to
`VocabIter::next(unsigned int&)’
../../moses/src/libmoses.a(LanguageModelSRI.o): In function `~VocabIter’:
/home/svp/transl/srilm/srilm2009/include/Vocab.h:271: undefined reference to
`LHashIter::~LHashIter()’
/home/svp/transl/srilm/srilm2009/include/Vocab.h:271: undefined reference to
`LHashIter::~LHashIter()’
../../moses/src/libmoses.a(LanguageModelSRI.o): In function
`Moses::LanguageModelSRI::Load(std::basic_string std::char_traits, std::allocator > const&, unsigned int, float,
unsigned int)’:
/home/svp/mosesdecoder/moses-chart/moses/src/LanguageModelSRI.cpp:58:
undefined reference to `Vocab::Vocab(unsigned int, unsigned int)’
/home/svp/mosesdecoder/moses-chart/moses/src/LanguageModelSRI.cpp:59:
undefined reference to `Ngram::Ngram(Vocab&, unsigned int)’
/home/svp/mosesdecoder/moses-chart/moses/src/LanguageModelSRI.cpp:67:
undefined reference to `File::File(char const*, char const*, int)’
/home/svp/mosesdecoder/moses-chart/moses/src/LanguageModelSRI.cpp:74:
undefined reference to `File::~File()’
/home/svp/mosesdecoder/moses-chart/moses/src/LanguageModelSRI.cpp:74:
undefined reference to `File::~File()’
collect2: ld returned 1 exit status
make: *** [CreateBerkeleyPt] Error 1

Do you have any ideas what’s wrong?

Thanx in advance.
Sergey


Re:[Moses-support] CreateBerkeleyPt and On-Disk Rule Table

hey sergey

it looks like it can’t find srilm. I’m not sure why it needs the language model, but best to make sure the configure paths are correct

Re:[Moses-support] CreateBerkeleyPt and On-Disk Rule Table

Ok, thank you, I fixed it with ./configure –with-srilm=

now, I have correct rule table (moses_chart works fine),
I convert it with command
./CreateBerkeleyPt/src/CreateBerkeleyPt 1 1 5 20
working-dir/model/rule-table.gz working-dir/model/rule-table
it create in working-dir/model/rule-table/ some db files:
Misc.db Source.db Target.db TargetInd.db TargetPhraseColl.db Vocab.db

Also I change working-dir/model/moses.ini
6 0 0 5 working-dir/model/rule-table.gz
=>
6 0 0 5 working-dir/model/rule-table

Now, moses starts good, but translation the same as source for any input:
test => test

Created input-output object : [6.000] seconds
test
Translating: test ||| [0,0]=X (1) [0,1]=X (1) [0,2]=X (1) [1,1]=X
(1) [1,2]=X (1) [2,2]=X (1)

[0..0]=1 [1..1]=1 [2..2]=1 [0..1]=1 [1..2]=0 [0..2]=1Num of hypo = 6 —
cells:
0 1 2
1 1 1
1 0
1
test

but old moses.ini moses work fine:
test
Translating: test ||| [0,0]=X (1) [0,1]=X (1) [0,2]=X (1) [1,1]=X
(1) [1,2]=X (1) [2,2]=X (1)

[0..0]=1 [1..1]=3 [2..2]=1 [0..1]=3 [1..2]=2 [0..2]=1Num of hypo = 23 —
cells:
0 1 2
1 3 1
3 2
1
испытание
(test in russian)

Also I download db-4.7.25.NC, install it in /usr/local/BerkeleyDB.4.7
I think that problem in berkeleyDB version, but result is the same

Re:[Moses-support] CreateBerkeleyPt and On-Disk Rule Table

Also I try to convert sample model

http://www.statmt.org/moses/download/sample-models.tgz

../CreateBerkeleyPt/src/CreateBerkeleyPt 1 1 1 20
syntax-model/rule-table syntax-model/rule-table-out
Starting : [0] seconds
Finished : [0] seconds

/usr/local/BerkeleyDB.4.8/bin/db_dump
syntax-model/rule-table-out/Source.db |wc -l
90

echo ‘das ist ein haus’ | ./moses_chart -f syntax-model/moses.ini 2> /dev/null
this is a house

syntax-model/moses.ini:
[ttable-file]
6 0 0 1 /home/svp/w/moses-chart/sample-models/syntax-model/rule-table
=>
[ttable-file]
6 0 0 1 /home/svp/w/moses-chart/sample-models/syntax-model/rule-table-out

echo ‘das ist ein haus’ | ./moses_chart -f syntax-model/moses.ini
Defined parameters (per moses.ini or switch):
config: syntax-model/moses.ini
glue-rule-type: 0
input-factors: 0
inputtype: 3
lmodel-file: 0 0 3
/home/svp/w/moses-chart/sample-models/lm/europarl.srilm.gz
mapping: 0 T 0
max-chart-span: 20 1000
non-terminals: X S
pop-limit: 1000
search-algorithm: 3
ttable-file: 6 0 0 1
/home/svp/w/moses-chart/sample-models/syntax-model/rule-table-out
ttable-limit: 20
weight-l: 0.5
weight-t: 0.5
weight-w: 0.0
Added 0 !UnknownWordPenalty 0-0
Added 1 WordPenalty 1-1
Loading lexical distortion models…
have 0 models
Start loading LanguageModel
/home/svp/w/moses-chart/sample-models/lm/europarl.srilm.gz : [0.000]
seconds
Added 2 LanguageModel 2-2
Finished loading LanguageModels : [1.000] seconds
Start loading PhraseTable
/home/svp/w/moses-chart/sample-models/syntax-model/rule-table-out :
[1.000] seconds
filePath: /home/svp/w/moses-chart/sample-models/syntax-model/rule-table-out
Added 3 PhraseModel 3-3
Start loading new format pt model : [2.000] seconds
Finished loading phrase tables : [2.000] seconds
IO from STDOUT/STDIN
Created input-output object : [2.000] seconds
Translating: das ist ein haus ||| [0,0]=X (1) [0,1]=X (1)
[0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X
(1) [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1)
[2,4]=X (1) [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X
(1) [4,5]=X (1) [5,5]=X (1)

[0..0]=1 [1..1]=1 [2..2]=1 [3..3]=1 [4..4]=1 [5..5]=1 [0..1]=0
[1..2]=0 [2..3]=0 [3..4]=0 [4..5]=0 [0..2]=0 [1..3]=0 [2..4]=0
[3..5]=0 [0..3]=0 [1..4]=0 [2..5]=0 [0..4]=0 [1..5]=0 [0..5]=0Num of
hypo = 6 — cells:
0 1 2 3 4 5
1 1 1 1 1 1
0 0 0 0 0
0 0 0 0
0 0 0
0 0
0
NO BEST TRANSLATION

End. : [2.000] seconds

Re:[Moses-support] CreateBerkeleyPt and On-Disk Rule Table

Hi sergey

change
6 0 0 1 ….
to
7 0 0 1 ….
the 7 tells the decoder to use the berkeley db implementation

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Related posts:

1. Moses Support Digest:Hierarchical rule extraction
2. Moses Support Digest:Reading binary phrase table from the disk
3. Moses Support Digest: Moses seems to hang
4. Moses Support Digest: About moses-chart
5. Moses Support Digest: moses_chart and recaser phrase-table
6. Moses Support Digest:about the moses-chart reordering
7. Moses Support Digest: moses-chart error while compiling training scripts
8. Moses Support Digest:Alignment information from binary phrase table
9. Moses Support Digest:compiling moses 3 chart
10. Moses Support Digest: About mert-moses in mose-chart