Hi all,

I ran some experiments with moses like, half a year ago. And recently I ran them for a second time. The time I got the reuslts, I got confused.
Beacause they’re so different from those I got previously.

The softwares I used was not changed, the same version. The corpus is of course the same. I just copied them. And I used the same script the run the experiments, just changed some directory. It seems I ran the same experiments on two different servers at different time, and got different results.

I checked alignment results, aligned.grow-diag-and-final, and there’re a lot of differences. I also checked moses.ini, and the parameters are greatly different.

So, has anybody ever come into this situation? I’m really confused…

Regards,

Lee Xianhua

Re:[Moses-support] different servers + different time – different result?

Giza++ and MERT both can produce different results, even when using the same code, corpora etc. This is because multiple solutions exist and each time you run Moses, you find one of these (different) optima.

Miles

Re:[Moses-support] different servers + different time – different result?

hi,

Thanks for you quick response.
But, will this cause a drop of BLEU, like, 0.5 point? I thinks that’s too much…

I have run my baseline experiments three times, and got three different results. The results for test set are: 0.2798, 0.2741, 0.2790.
The first is run on server1 previously, the second and the third are run recently, while the second is run on server2, and the third is run on server1.

Now I don’t know what is my baseline.

Regards,

Lee Xianhua

Re:[Moses-support] different servers + different time – different result?

yes, you can easily get a 1BP drop between multiple runs.

if you want to do experiments and report BLEU scores then people really need to do multiple runs and report on averages, along with variances. i think from no-one i’m going to start penalising papers i
get to review if people don’t do something about this

(and i do a lot of reviewing …)

Miles

NOTICE:This is digested from the Moses-support mailing list, which supports for the moses SMT decoder.

Related posts:

1. Moses Support Digest: Any documentation about the Multiple Decoding Path functionality
2. Moses Support Digest: Regarding moses.weight-reused.ini
3. Moses Support Digest:Call for Participation ACL WMT 2010 Machine Translation Shared Task
4. Moses Support Digest: The problem with the installation of Moses on Windows XP
5. Moses Support Digest:word lattice and multiple translation tables optimization problem
6. Moses Support Digest:The results of your email commands
7. Moses Support Digest:running giza in parts
8. Moses Support Digest:nbest list option in decoder
9. Moses Support Digest:Hierarchical rule extraction
10. Moses Support Digest:Pulling source data