52nlp's Learning Notes » NLP

A Cool Dictionary for Natural Language Processing

52nlp — Mon, 30 Nov 2009 15:53:53 +0000

I found Professor Bill Wilson’s “The Natural Language Processing Dictionary” accidentally tonight, and thought it very cool for nlpers. Except from the NLP Dictionary, you also can find the Prolog, Artificial Intelligence and Machine learning Dictionary in this web page. Below is from this Dictionary:

“You should use The NLP Dictionary to clarify or revise concepts that you have already met. The NLP Dictionary is not a suitable way to begin to learn about NLP. Further information on NLP can be found in the class web page lecture notes section.

Other places to find out about artificial intelligence include the AAAI (American Association for Artificial Intelligence) AI Overview page or AI Reference Shelf

If you wish to suggest an item or items that should be included, or if you found an item that you felt was unclear, please let me know (E-mail: billw at cse.unsw.edu.au). ”

If you are interested in NLP Dictionary and others, visit them:
The Natural Language Processing Dictionary – URL:http://www.cse.unsw.edu.au/~billw/nlpdict.html
The Prolog Dictionary – URL: http://www.cse.unsw.edu.au/~billw/prologdict.html
The Artificial Intelligence Dictionary – URL: http://www.cse.unsw.edu.au/~billw/aidict.html
The Machine Learning Dictionary – URL: http://www.cse.unsw.edu.au/~billw/mldict.html

From nlpers:Getting Started In Summarization

52nlp — Mon, 23 Nov 2009 11:37:02 +0000

“Getting Started In Summarization” is seconde post of the “GSI” series in nlpers blog. Following is from nlpers blog:

I’ll kick off the “Getting Started In” series with summarization, since it is near and dear to my heart.

Warning: I should point out at the get-go that these posts are not intended to be comprehensive or present a representative sample of work in an area. These are blog posts, not survey papers, tutorials or anything of that sort. If I don’t cite a paper, it’s not because I don’t think it’s any good. It’s also probably not a good idea to base a career move on what I say.

Summarization is the task of taking a long utterance (or a collection of long utterances) and producing a short utterance. The most common is when utterance=document, but there has also been work on speech summarization too. There are roughly three popular types of summarization: headline generation, sentence extraction and sentence compression (or document compression). In sentence extraction, a summary is created by selecting a bunch of sentences from the original document(s) and gluing them together. In headline generation, a very short summary (10 words or so) is created by selecting a bunch of words from the original document(s) and gluing them together. In sentence (document) compression a summary is created by dropping words and phrases from a sentence (document).

One of the big problems in summarization is that if you give two humans the same set of documents and ask them to write a summary, they will do wildly different things. This happens because (a) its unclear what information is important and (b) each human has different background knowledge. This is partially alleviated by moving to a task-specific setting, like the query-focused summarization model, which has grown increasingly popular in the past few years. In the query-focused setting, a document (or document collection) is provided along with a user query that serves to focus the summary on a particular topic. This doesn’t fix problem (b), but goes a long way to fixing (a), and human agreement goes up dramatically. Another advantage to the query-focused setting is that, at least with news documents, the most important information is usually presented first (this is actually stipulated by many newspaper editor’s guidelines). This means that producing a summary by just taking leading sentences often does incredibly well.

A related problem is that of evaluation. The best option is to do a human evaluation, preferably in some simulated real world setting. A reasonable alternative is to ask humans to write reference summaries and compare system output to these. The collection of Rouge metrics has been designed to automate this comparison (Rouge is essentially a collection of similarity metrics for matching human to system summaries). Overall, however, evaluation is a long-standing and not-well-solved problem in summarization.

Techniques for summarization vary by summary type (extraction, headline, compression).

The standard recipe for sentence extraction works as follows. A summary is created by first extracting the “best” sentence according to a scoring module. The second sentence is selected by finding the next-best sentence according to the scoring module, minus some redundancy penalty (we don’t want to extract the same information over and over again). This process ends when the summary is sufficiently long. The scoring component assigns to each sentence in the original document collection a score that says how important it is (in query-focused summarization, for instance, the word overlap between the sentence and the query would be a start). The redundancy component typically computes the similarity between a sentence and the previously extracted sentences.

Headline generation and sentence compression have not yet reached a point of stability in the same way that sentence extraction has. A very popular and successful approach to headline generation is to train a hidden Markov model something like what you find in statistical machine translation (similar to IBM model 1, for those familiar with it). For sentence compression, one typically parses a sentence and then attempts to summarize it by dropping words and phrases (phrases = whole constituents).

Summarization has close ties to question answering and information retrieval; in fact, having some limited background in standard IR techniques (tf-idf, vector space model, etc.) are pretty much necessary in order to understand what goes on in summarization.

Here are some papers/tutorials/slides worth reading to get one started (I’d also recommend Inderjeet Mani’s book, Automatic Summarization if you don’t mind spending a little money):

Background IR material
Sentence extraction
Headline generation: Automatic Headline Generation for Newspaper Stories
Sentence compression: Statistics-Based Summarization — Step One: Sentence Compression
Other:
1. Discussion: What might be in a summary?
2. Automatic evaluation: ROUGE: a Package for Automatic Evaluation of Summaries
3. Recent fun stuff: Cut and paste based text summarization, Learning to Paraphrase: An Unsupervised Approach Using Multiple-Sequence Alignment and A Statistical Approach for Automatic Speech Summarization

From nlpers:Getting Started in NLP

52nlp — Fri, 20 Nov 2009 13:58:35 +0000

　　nlpers blog is very famous in the natural language processing world, but it’s very pity in China we can’t visit it directly. From now I will choose some useful posts in nlpers and post them here as a mirror, hope these will be a bridge between Chinese NLP lovers and nlpers blog.
　　”Getting Started in NLP” is posted in 2006, but I think it is very useful for NLP learners, especially for the NLP beginners. I have translated in Chinese, if you are interesting it, you can find it in my Chinese blog: getting started in natural language processing.
　　Following is from nlpers blog:

　　Since starting the blog, a few people have asked me how one can get started in NLP, while residing in a department lacking NLP researchers. This is a difficult question: I fell into NLP quite naturally when I was at CMU and made an easy transition to grad school at USC, both of which have awesome NLP groups. Lacking such internal support, one has to be much more ambitious to get to the point where one could do real research in the field. The obvious avenues for support are: reading books (which ones?) and papers (from where and by whom?), going to nearby conferences (which ones?) and experimentation (on what?). (New option: read and post to this blog!)
　　The four standard books in the field are Statistical NLP (Manning + Schutze), Speech and Language Processing (Jurafsky + Martin), Statistical Language Learning (Charniak) and Natural Language Understanding (Allen). The latter two are much older, though some people prefer Charniak to Manning + Schutze. I would probably pick up Manning + Schutze if I could only buy one. From this book, I think that skimming Chapters 1, 4, 6 and 13 should give a reasonable (but not uniformly sampled) representation of background knowledge everyone should know. Unfortunately, this misses many topics: information extraction, summarization, question answering, dialog systems, discourse, morphology, ontologies, pragmatics, semantics, sentiment analysis and textual entailment.
　　Finding good papers for beginners is hard. Without guidance, skimming titles and abstracts of papers published in ACL, NAACL, HLT or COLING since 2002 or 2003 should enable someone to find out what looks interesting to them. I know many advisors take this approach with new students. The ACL anthology is great for finding old papers. I’ll probably post at a later date about what are the “must reads” for the areas I know best. Once you’ve found a few papers you like, I’d check out the respective author’s web pages and see if they have any related work (best bet is probably to look at the advisor’s page: often s/he will have multiple students working on similar topics). Also, advisor’s often have course material and slides from tutorials: these are great places to get introductory-level material.
　　If you happen to get lucky (I never have) and one of the above conferences is located nearby, I’d just go. Presentations of papers (if they’re good) are often better from the perspective of getting the high-level overview than the papers themselves, since papers have to be technically complete.
　　I’m perhaps overcommitting myself, given my promise to talk more about structured prediction, but over the next few weeks/months, I’ll work on a “Getting Starting in X” series. X will likely range over the set { summarization, sequence labeling, information extraction, machine translation, language modeling }. Requests for other topics will be heard, keeping in mind I’m not an expert in many areas.

Graphical Models and Bayesian Networks Tutorial Reading

52nlp — Wed, 18 Nov 2009 16:14:50 +0000

The following excerpt from “A Brief Introduction to Graphical Models and Bayesian Networks” by Kevin Murphy.

Books

In reverse chronological order.

Daphne Koller and Nir Friedman, “Probabilistic graphical models: principles and techniques”, MIT Press 2009
Adnan Darwiche, “Modeling and reasoning with Bayesian networks”, Cambridge 2009
F. V. Jensen. “Bayesian Networks and Decision Graphs”. Springer. 2001.
Probably the best introductory book available.
D. Edwards. “Introduction to Graphical Modelling”, 2nd ed. Springer-Verlag. 2000.
Good treatment of undirected graphical models from a statistical perspective.
J. Pearl. “Causality”. Cambridge. 2000.
The definitive book on using causal DAG modeling.
R. G. Cowell, A. P. Dawid, S. L. Lauritzen and D. J. Spiegelhalter. “Probabilistic Networks and Expert Systems”. Springer-Verlag. 1999.
Probably the best book available, although the treatment is restricted to exact inference.
M. I. Jordan (ed). “Learning in Graphical Models”. MIT Press. 1998.
Loose collection of papers on machine learning, many related to graphical models. One of the few books to discuss approximate inference.
B. Frey. “Graphical models for machine learning and digital communication”, MIT Press. 1998.
Discusses pattern recognition and turbocodes using (directed) graphical models.
E. Castillo and J. M. Gutierrez and A. S. Hadi. “Expert systems and probabilistic network models”. Springer-Verlag, 1997.
A Spanish version is available online for free.
F. Jensen. “An introduction to Bayesian Networks”. UCL Press. 1996. Out of print.
Superceded by his 2001 book.
S. Lauritzen. “Graphical Models”, Oxford. 1996.
The definitive mathematical exposition of the theory of graphical models.
S. Russell and P. Norvig. “Artificial Intelligence: A Modern Approach”. Prentice Hall. 1995.
Popular undergraduate textbook that includes a readable chapter on directed graphical models.
J. Whittaker. “Graphical Models in Applied Multivariate Statistics”, Wiley. 1990.
This is the first book published on graphical modelling from a statistics perspective.
R. Neapoliton. “Probabilistic Reasoning in Expert Systems”. John Wiley & Sons. 1990.
J. Pearl. “Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference.” Morgan Kaufmann. 1988.
The book that got it all started! A very insightful book, still relevant today.

Review articles

P. Smyth, 1998. “Belief networks, hidden Markov models, and Markov random fields: a unifying view”, Pattern Recognition Letters.
E. Charniak, 1991. “Bayesian Networks without Tears”, AI magazine.
Sam Roweis & Zoubin Ghahramani, 1999. A Unifying Review of Linear Gaussian Models, Neural Computation 11(2) (1999) pp.305-345

Exact Inference

C. Huang and A. Darwiche, 1996. “Inference in Belief Networks: A procedural guide”, Intl. J. Approximate Reasoning, 15(3):225-263.
R. McEliece and S. M. Aji, 2000.

Related posts:

Bayesian Modeling for Language Tutorial Reading

52nlp — Sat, 14 Nov 2009 15:04:01 +0000

This is reprint from Sharon Goldwater’s “Reading list on Bayesian modeling for language“.

People often ask me what they can read to learn more about recent Bayesian modeling techniques and their applications to language learning. Here is a list of the papers I have found to be most useful and relevant to my own research. I try to emphasize the papers aimed at a slightly less technical/more cognitively inclined audience. This is not intended to be a complete list, only a starting point.

General introductory material

Thomas L. Griffiths and Alan Yuille (2006). A primer on probabilistic inference. Trends in Cognitive Sciences. Supplement to special issue on Probabilistic Models of Cognition (volume 10, issue 7).

Reviews many of the basic concepts underlying probabilistic (especially Bayesian) modeling and inference, using simple examples.

Sharon Goldwater (2006). Nonparametric Bayesian Models of Lexical Acquisition. Unpublished doctoral dissertation, Brown University, 2006.

Aimed primarily at computational linguists, but should (I hope) be accessible to anyone who has a basic familiarity with generative probabilistic models. Chapters 2 and 3 cover many useful topics, including Bayesian integration in finite and infinite models (i.e., Dirichlet distribution, Dirichlet process, Chinese restaurant process) and a brief introduction to sampling techniques (Gibbs sampling and Metropolis-Hastings sampling).

Daniel J. Navarro, Thomas L. Griffiths, Mark Steyvers, and Michael D. Lee (2006). Modeling individual differences using Dirichlet processes. Journal of Mathematical Psychology, 50, 101-122.

A very nice introduction to Dirichlet processes aimed at cognitive scientists. Slightly more in-depth, covers the stick-breaking construction for the Dirichlet process (which is not in my thesis) as well as the Chinese restaurant process.

Bayesian language models for learning

Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson (2007). Distributional Cues to Word Segmentation: Context is Important. Proceedings of the 31st Boston University Conference on Language Development.

Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson (2006). Contextual Dependencies in Unsupervised Word Segmentation. Proceedings of Coling/ACL.

These two papers apply the Dirichlet process and hierarchical Dirichlet process to word segmentation. The BUCLD paper is more conceptual, the ACL paper is more technical. For a more in-depth treatment, see also Chapter 5 of my thesis (above).

Sharon Goldwater and Thomas L. Griffiths. A Fully Bayesian Approach to Unsupervised Part-of-Speech Tagging. Proceedings of the Association for Computational Linguistics.

This paper provides a direct comparison between Bayesian methods (averaging over parameters and estimation using Gibbs sampling) and standard methods (estimating parameters directly using EM) using the same underlying model (a standard finite HMM).

Mark Johnson (2007). Why Doesn’t EM Find Good HMM POS-Taggers? Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL).

Includes Variational Bayes as well as Gibbs sampling and EM as estimation procedures. Results are somewhat contradictory to Goldwater and Griffiths, possibly due to the combination of a simpler model and more training data.

Percy Liang, Slav Petrov, Michael I. Jordan, Dan Klein (2007). The infinite PCFG using hierarchical Dirichlet processes.Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL).

Jenny Rose Finkel, Trond Grenager and Christopher D. Manning (2007). The Infinite Tree. Proceedings of the Association for Computational Linguistics.

Mark Johnson, Thomas L. Griffiths, and Sharon Goldwater (2007). Adaptor Grammars: a Framework for Specifying Compositional Nonparametric Bayesian Models. Advances in Neural Information Processing Systems 19.

These three papers all deal with nonparametric models of syntax (dependency or context-free grammars). They might be a bit tough for those with less background in nonparametrics, although the exposition in Liang et al. is very nice.

Thomas L. Griffiths, Michael Steyvers, and Joshua B. Tenenbaum (2007). Topics in semantic representation. Psychological Review, 114, 211-244.

Thomas L. Griffiths, Michael Steyvers, David M. Blei, and Joshua B. Tenenbaum (2005). Integrating topics and syntax. Advances in Neural Information Processing Systems 17.

David Blei, Andrew Ng, and Michael Jordan (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993-1022. (A shorter version appeared in NIPS 2002).

These three papers are about Latent Dirichlet Allocation (a.k.a. topic models) for learning semantic structure. The Psych Review paper provides a less technical introduction and considers LDA as a cognitive model. The JMLR paper is the original one, suitable if you want more technical details. The NIPS paper is just cool.

Fei Xu and Joshua B. Tenenbaum (2007). Word learning as Bayesian inference. Psychological Review, 114, 245-272.

Develops a Bayesian model to explain how children learn words at different levels of specificity (basic-level categories versus subordinate or superordinate).

Bayesian models of language processing

This isn’t really my area, but here are a couple of interesting papers I know of:

Dennis Norris (2006). The Bayesian reader: explaining word recognition as an optimal Bayesian decision process. Psychological Review, 113(2), 327-357.

Naomi Feldman and Thomas L. Griffiths (2007). A rational account of the perceptual magnet effect. Proceedings of the Twenty-Ninth Annual Conference of the Cognitive Science Society.

Inference

A bunch of the papers mentioned above have descriptions of sampling algorithms and/or variational inference procedures for specific models. For more general information on these topics, consider reading some of the following:

Sharon Goldwater (2006). Nonparametric Bayesian Models of Lexical Acquisition. Unpublished doctoral dissertation, Brown University, 2006.

As I mentioned above, there is a brief overview of Markov chain Monte Carlo methods (Gibbs sampling and Metropolis-Hastings) in Chapter 2. Examples of Gibbs sampling algorithms are described in chapters 4 and 5.

Julian Besag (2000). Markov chain Monte Carlo for statistical inference. Working paper no. 9. University of Washington Center for Statistics and the Social Sciences.

A longer and more technical introduction to Markov chain Monte Carlo methods.

Mark Johnson, Thomas L. Griffiths, and Sharon Goldwater (2007). Bayesian Inference for PCFGs via Markov Cain Monte Carlo. Proceedings of the North American Association for Computational Linguistics.

How to do efficient sampling for PCFGs.

Matthew Beal (2003). Variational Algorithms for Approximate Bayesian Inference. PhD. Thesis, Gatsby Computational Neuroscience Unit, University College London. (Or download individual chapters from here.)

I don’t know much about variational methods myself, but I’ve been told this is a good place to start.

Further Reading

Yee Whye Teh, Michael Jordan, Matthew Beal, and David Blei (2006). Hierarchical Dirichlet processes. Journal of the American Statistical Association, 2006. 101(476):1566-1581.

The original HDP paper. Comprehensive, but I would suggest getting familiar with the ideas using some of the resources above before reading this one.

Radford Neal (1993). Probabilistic Inference Using Markov Chain Monte Carlo Methods. Technical report CRG-TR-93-1. University of Toronto Department of Computer Science.

Even more information about Markov chain Monte Carlo methods.

Statistical Machine Translation Tutorial Reading

52nlp — Mon, 09 Nov 2009 13:08:10 +0000

　　I have seen this from Dr. David Kauchak’s webpage and think it worth to keep here, so I reprint here.

　　The following is a list of papers that I think are worth reading for our discussion of machine translation. I’ve tried to give a short blurb about each of the papers to put them in context. I’ve included a number of papers that I marked “OPTIONAL” that I think are interesting, but are either supplementary or the material is more or less covered in the other papers.
　　If anyone would like more information on a particular topic or would like to discuss any of these papers, feel free to e-mail me　dkauchakcs ucsd edu

Part 1 (Jan. 19)
A Statistical MT Tutorial Workbook. Kevin Knight. 1999.
　　Very good introduction to word-based statistical machine translation.Written in an informal, understandable, tutorial oriented style.
　　Kevin Knight said about the Knight99, it is very interesting below:
　　“At the time, I was trying to align English sound sequences with Japanese sound sequences, and I knew that EM could do it for me. It was hard, though. I spent two years reading Brown et al 93. When I finally got it to work, I was pretty fired up, and I told David Yarowsky. He said, “EM is the answer to all the world’s problems.” Wow! I figured everybody should know about it, so I wrote “A Statistical MT Tutorial Workbook”.

Automating Knowledge Acquisition for Machine Translation.Kevin Knight. 1997.
　　(OPTIONAL) Another tutorial oriented paper that steps through how one can learn from bilingual data. Also introduces a number of important concepts for MT.

Foundations of Statistical NLP,chapter 13. Manning and Schutze. 1999.
　　(OPTIONAL) Must be accessed from UCSD. Overview of statistical MT. Spends a lot of time on sentence and word alignment of bilingual data.

Foundations of Statistical NLP, chapter 6. Manning and Schutze. 1999.
　　(OPTIONAL) Must be accessed from UCSD. Discusses n-gram language modeling. Language modeling is crucial for SMT and many other natural language applications. I won’t spend much time discussing language modeling, but for those that are interested this is a good introduction.

Part 2 (Jan. 26)
Word models:
The Mathematics of Statistical Machine Translation: Parameter Estimation. P. F. Brown, S. A. Della Pietra, V. J. Della Pietra and R.L. Mercer. 1993.
　　(OPTIONAL) All you ever wanted to know about word level models. Describes IBM models 1-5 and parameter estimation for these models. It’s about 50 pages and contains a lot of material for the interested reader.

Word model decoding:
Decoding Algorithm in Statistical Machine Translation.Ye-Yi Wand and Alex Waibel. 1997.
　　Early paper discussing decoding of IBM model 2. The paper provides a fairly good introduction to word-level decoding including multi-stack search (i.e. multiple beams) and rest cost estimation (heuristic functions).

An Efficient A* Search Algorithm for Statistical Machine Translation.Franz Josef Och, Nicola Ueffing, Hermann Ney. 2001.
　　(OPTIONAL) One of many papers on decoding with word-based SMT. They discuss the basic idea of viewing decoding as state space search and provide one method for doing this. They describe decoding for Model 3 and suggest a few different heuristics that are admissible, leading to few search errors.

Phrase based statistical MT:
　　 Statistical Phrase-Based Translation.Philipp Koehn, Franz Jasof Och and Daniel Marcu. 2003.
　　Good, short overview of phrased based systems. If you want more details, see the paper below.

The Alignment Template Approach to Statistical Machine Translation.Franz Josef Och and Hermann Ney. 2004.
　　(OPTIONAL) This is a journal paper discussing one phrase based statistical system including decoding. This is more or less the system used at ISI and is probably the best current system (though syntax based systems my beat these in the next few years). Requires acrobat 5 and to be at UCSD.

Part 3 (Feb. 2)
Phrase-based decoding:
　　See the previous paper.

Syntax based translation:
What’s in a Translation Rule? Galley, Hopkins, Knight and Marcu. 2004.
　　This is the current system being investigated at ISI and the hope is that these syntax based systems will perform better than phrase based systems.The paper is a bit tough to read since it’s a conference paper.

A Syntax-Based Statistical Translation Model. Yamada and Knight. 2001.
　　(OPTIONAL) Predecessor model to Galley et al., but similar.

Syntax based decoding:
Foundations of Statistical NLP, chapter 12. Manning and Schutze. 1999.
　　Must be on campus. This is a chapter on parsing (not actually decoding) However, since the above rules are very similar to PCFGs, then decoding is very similar to parsing… just with more complications.

A Decoder for Syntax-Based Statistical MT. Kenji Yamada and Kevin Knight. 2001.
　　(OPTIONAL) Decoder for the above Yamada and Knight model.

Part 4 (Feb. 9)
Discriminative Training:
Discriminative Training and Maximum Entropy Models for Statistical Machine Translation.Och and Ney. 2002.
　　Learning how the best models for combining the different models (traslation model, language model, etc.) using maximum entropy parameter estimation.This line of research is still very important and my be interesting to many of you since it’s very machine learningy.

Discriminative Reranking for Machine Translation. Shen, Sarkar and Och. 2004.
　　(OPTIONAL) Given a ranked output of possible translations from the translation system, this paper uses the perceptron algorithm to learn a reranking of the sentences to improves the top translation.

MT Evaluation:
BLEU: A Method for Automatic Evaluation of Machine Translation. Papineni, Roukos, Ward and Zhu. 2001.
　　Foundational method for evaluating MT methods and still used currently.

Maximum Entropy Model Tutorial Reading

52nlp — Wed, 04 Nov 2009 14:47:03 +0000

　　This post is reprinted from Dr Zhang’s Maximum Entropy Modeling Toolkit manul. This section lists some recommended papers for your further reference.

1. Maximum Entropy Approach to Natural Language Processing [Berger et al., 1996]
　　A must read paper on applying maxent technique to Natural Language Processing. This paper describes maxent in detail and presents an Increment Feature Selection algorithm for increasingly construct a maxent model as well as several example in statistical Machine Translation.

2.Inducing Features of Random Fields [Della Pietra et al., 1997]
　　Another must read paper on maxent. It deals with a more general frame work: Random Fields and proposes an Improved Iterative Scaling algorithm for estimating parameters of Random Fields. This paper gives theoretical background to Random Fields (and hence Maxent model). A greedy Field Induction method is presented to automatically construct a detail random elds from a set of atomic features. An word morphology application for English is developed.

3.Adaptive Statistical Language Modeling: A Maximum Entropy Approach [Rosenfeld, 1996]
　　This paper applied ME technique to statistical language modeling task. More specically, it built a conditional Maximum Entropy model that incorporated traditional N-gram, distant N-gram and trigger pair features. Significantly perplexity reduction over baseline trigram model was reported. Later, Rosenfeld and his group proposed a Whole Sentence Exponential Model that overcome the computation bottleneck of conditional ME model.

4.Maximum Entropy Models For Natural Language Ambiguity Resolution [Ratnaparkhi, 1998]
　　This dissertation discussed the application of maxent model to various Natural Language Disambiguity tasks in detail. Several problems were attacked within the ME framework: sentence boundary detection, part-of-speech tagging, shallow parsing and text categorization. Comparison with other machine learning technique (Naive Bayes, Transform Based Learning, Decision Tree etc.) are given.

5.The Improved Iterative Scaling Algorithm: A Gentle Introduction [Berger, 1997]
　　This paper describes IIS algorithm in detail. The description is easier to understand than [Della Pietra et al., 1997], which involves more mathematical notations.

6.Stochastic Attribute-Value Grammars (Abney, 1997)
　　Abney applied Improved Iterative Scaling algorithm to parameters estimation of Attribute-Value grammars, which can not be corrected calculated by ERF method (though it works on PCFG). Random Fields is the model of choice here with a general Metropolis-Hasting Sampling on calculating feature expectation under newly constructed model.

7.A comparison of algorithms for maximum entropy parameter estimation [Malouf, 2003]
　　Four iterative parameter estimation algorithms were compared on several NLP tasks. L-BFGS was observed to be the most effective parameter estimation method for Maximum Entropy model, much better than IIS and GIS. [Wallach, 2002] reported similar results on parameter estimation of Conditional Random Fields.