I Love Natural Language Processing

[Moses-support] Pulling source data

I am experimenting with the Moses application now, and I have it so that it is pulling in my data from two flat, aligned text files.

My question is, can I pull in data from a mysql database table rather than a text file, or would the best approach be to dump the data on a regular basis to a text file and then process from there?

Thanks,
John


Re:[Moses-support] Pulling source data

I guess I should ask a second, related part… if my langauge corpus will be updated on a basis of at least once per hour, perhaps more, what is the best way to go here? For example, Moses will take longer than an hour to process the corpus.

– Is that assumption correct?
– Is there a way to have Moses work on a diff file?
– Is there a better way?

I am not looking for a solution per se, just trying to figure out which path to start down so I can experiment on my own.

Thanks,
John

Re:[Moses-support] Pulling source data

we have been working on streaming translation, which is what you are talkling about (ie data arrives at a high rate). as they say, stay tuned for more.

Miles

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Related posts:

1. Moses Support Digest:openTMS supports Moses as a data source
2. Moses Support Digest:Moses step 1 – data preparation step
3. Moses Support Digest: Issues with Score data
4. Moses Support Digest:Hierarchical rule extraction
5. Moses Support Digest: Experiment.perl publications & documentation
6. Moses Support Digest: experiment management system and Moses scripts
7. Moses Support Digest:GIZA++ error
8. Moses Support Digest:compiling moses 3 chart
9. Moses Support Digest:tuning tree-based models
10. Moses Support Digest:moses decoder results on cygwin and dos