[Moses-support] Moses decoder on windows freezes after 5 sentences
Dear All
I have this little python script to run quick tests on moses models (see below). The same script works on unix and windows, given the right paths. I’m also using the same basic idea as a basis for the moses thread in a translation GUI.
Essentially, the MosesCMD class starts up moses in a subprocess, and subsequent calls to translate() send data in on the subprocess’ stdin and receive translations on its stdout.
Anyway, on unix (actually MacOSX) this will run and run (haven’t tested it on 1000s of sentences, but plenty); on windows, it packs in after five sentences (sometimes fewer) and completely freezes. Automatically killing the subprocess and starting a new one after every three sentences works sometimes, but sometimes (eg when it’s powering the moses thread in a GUI) actually makes things worse.
The moses decoder I’m using on windows I compiled under cygwin. Both decoders are using the same irstlm language models.
I’m not very familiar with windows. Does anyone have any ideas what might be causing the windows moses to keel over? Is it something to do with the way windows manages threads/processes/memory? Anything I can do? Unfortunately has to run on windows.
With thanks and best wishes
Ivan
Here’s the script:
#! /usr/bin/env python
import subprocess, time
class MosesCMD(object):
def __init__(self):
self.cmd_path = ”
self.model_path = ”
self.recaser_model_path = ”
self.proc = None
def start(self):
cmd = “%s -f %s” % (self.cmd_path, self.model_path)
if self.recaser_model_path:
cmd = “%s | %s -v 0 -f %s -dl 1″
% (cmd, self.cmd_path, self.recaser_model_path)
self.proc = subprocess.Popen(cmd, shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
def translate(self, line):
line = ‘%sn’ % line.strip()
self.proc.stdin.write(line)
return self.proc.stdout.readline().capitalize()
def close(self):
self.proc.kill()
def main():
mc = MosesCMD()
mc.cmd_path = “/path/to/mosesdecoder/moses-cmd/src/moses”
mc.model_path = “/path/to/model/moses.ini”
mc.recaser_model_path = “/path/to/recase/moses.ini”
mc.start()
time.sleep(1)
print ‘Ready.’
cy = ‘*’
while cy:
cy = raw_input(‘cymraeg> ‘)
en = mc.translate(cy)
print ‘saesneg> %sn’ % en
mc.close()
if __name__ == “__main__”:
main()
Re: [Moses-support] Moses decoder on windows freezes after 5 sentences
Hi Ivan
This looks like a python/windows problem rather than a moses problem. I’m not familiar with windows, but I notice that the python documentation for subprocess recommends using the communicate() method for bidirectional interprocess communication, rather than read()/write(), in order to avoid deadlocks.
An alternative would be to try to do this in perl instead of python, again looking for documentation on bidirectional ipc. This is certainly possible on unix, not sure about windows.
A different approach would be to use the moses server, and an xmlrpc library on the client-side. I don’t know if the moses server will compile on windows – but it would be possible to run it on a separate linux box, and just run your client on windows,
hope that helps,
regards
Barry
NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.
Related posts:
- Moses Support Digest: The problem with the installation of Moses on Windows XP
- Moses Support Digest: processPhraseTable in windows
- Moses Support Digest:moses decoder results on cygwin and dos
- Moses Support Digest:RDBMS for the decoder
- Moses Support Digest:nbest list option in decoder
- Moses Support Digest:POS translation
- Moses Support Digest:Binarized SRILM
- How to start new process or run shell commands within python?
- Moses Support Digest:Translation from English to Foreign Language
- Moses Support Digest:Moses Error in training phrase