I Love Natural Language Processing

[Moses-support] Suffix arrays in Moses

Hi all,

I’m just wanting to double-check the current state of the suffix array code in Moses. Can it be used to extract translation table entries on-the-fly?

Also, has anyone written up a paper on this in Moses? I’d like to know who to cite if this has been written up.

Cheers,
Lane


Re:[Moses-support] Suffix arrays in Moses

Hi,

If I understood correctly you want to have access to the binary phrase table on your own program to extract some entries on the fly. Attached you will find a piece of code that does exactly this.

Hope it helps
Greetings
–
Felipe.

Re:[Moses-support] Suffix arrays in Moses

Felipe,

Thanks for the code. I appreciate the help. However, that’s not what I was wondering.

I know that during the 2nd MT Marathon in Wandlitz in May 2008, work was done on implementing a suffix array data structure so that Moses could extract phrase pairs directly from an aligned parallel corpus at runtime, without the necessity of first running an explicit off-line phrase table extraction process.

I don’t know what the end result of that work was, nor do I know if any followup work was performed.

My question was in regard to those issues. Specifically, does anyone know where things ended with regard to the suffix array code in Moses at the end of the MT Marathon in Wandlitz? And is functionality currently present in Moses to allow an aligned parallel corpus backed by a suffix array to act in place of a pre-computed phrase table?

Thanks,
Lane

Re:[Moses-support] Suffix arrays in Moses

Hey Lane,

I don’t know anyone developing a suffix array for Moses @ the moment but i suppose it’s paper submission time so people will be in purdah.

@ the MT Marathon, we tacked on your Joshua suffix array to Moses, but we’ve since extended the Moses phrase extraction to do hierarchical/syntax.

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Related posts:

1. Moses Support Digest: Experiment.perl publications & documentation
2. Moses Support Digest:Hierarchical rule extraction
3. Moses Support Digest:Alignment information from binary phrase table
4. Moses Support Digest: CALL FOR PAPERS – PBML
5. Moses Support Digest: moses-chart error while compiling training scripts
6. Moses Support Digest:mt3_chart compilation error
7. Moses Support Digest:Aligned phrase counts
8. Moses Support Digest:Translation from English to Foreign Language
9. Moses Support Digest:Reading binary phrase table from the disk
10. Moses Support Digest:Code monkey available,Will work for peanuts