52nlp's Learning Notes » binary phrase table

Moses Support Digest: processPhraseTable in windows

52nlp — Mon, 24 May 2010 15:03:47 +0000

[Moses-support] processPhraseTable in windows

Dear All,

I would like to know how to run processPhraseTable in windows to make Binary Phrase Tables.

Best Regards,
Kyaw Kyaw Zin

Re: [Moses-support] processPhraseTable in windows

the same as on any other platform, but please read

http://www.statmt.org/moses/?n=Moses.FAQ#ntoc9

Moses Support Digest:Alignment information from binary phrase table

52nlp — Thu, 07 Jan 2010 16:26:19 +0000

[Moses-support] Alignment information from binary phrase table

Hello all,

I am trying to get the alignment information from a binary phrase table that was generated from a plain phrase table containing alignment information.

I load the phrase table through the following piece of code:
——————
numScoreComponent=5;
numInputScores=0;
tableLimit=0;
weightWP=0;

input.push_back(0);
output.push_back(0);

weight.push_back(0); weight.push_back(0);
weight.push_back(0); weight.push_back(0); weight.push_back(0);

PhraseDictionaryTreeAdaptor *pd=new
PhraseDictionaryTreeAdaptor(numScoreComponent, numInputScores);

pd->DisableCache();

if (!pd->Load(input, output, “/path/to/phrase/table”, weight, tableLimit, lmList, weightWP)) {
delete pd;
cerr<<"Error: Cannot load Moses binary phrase table, path='"< exit(EXIT_FAILURE);
}
------------------

and I get the following error message:

reading bin ttable
size of OFF_T 8
ERROR:You are asking binary phrase table without word alignments but the
file do not exist. Please check if you had generated the correct phrase
table without word alignment
(phrase-table.binphr.srctree,phrase-table.binphr.tgtdata)

ERROR:bin ttable was read in a wrong way

When the binary phrase table does not contain any alignment information
it works without problems.

How should a binary phrase table with alignment information be loaded?
Should I change some parameters? Which? Any help will be very valuable.

Thank you very much in advance, and happy new year.

Re:[Moses-support] Alignment information from binary phrase table

hi felipe

The decoder no longer read the word alignment information and the binary phrase table no longer carries that information.

there were problems with the implementation which made the decoder really slow and ate memory.

i have a new binary phrase table which does carry the information which you’re welcome to use but you’ll need to integrate it with the existing decoder.

Alternatively, roll back to before the alignment information was taken out

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Moses Support Digest:Reading binary phrase table from the disk

52nlp — Tue, 05 Jan 2010 12:56:16 +0000

[Moses-support] Reading binary phrase table from the disk

Hello,

As I understand, when given a binary phrase table, moses does not load it into memory, but rather seeks for phrases in file directly from disk. Could you please tell me, what are the functions in C++ that are used for this purpose.

Thank you,

Michael.

Re:[Moses-support] Reading binary phrase table from the disk

There are several layers to the implementation, with the program-level API defined in the PhraseDictionaryTreeAdaptor class.
PhraseDictionaryTreeAdaptor forwards its calls to PDAImp, which is a wrapper on PhraseDictionaryTree, which uses a PrefixTreeF template class. PrefixTreeF reads binary files using a smart pointer defined in FilePtr.h.

Cheers
Adam

Re:[Moses-support] Reading binary phrase table from the disk

Thank you very much.

Michael.

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.