Departmental Papers (CIS)
Date of this Version
December 2004
Document Type
Conference Paper
Recommended Citation
John Blitzer, Kilian Q. Weinberger, Lawrence K. Saul, and Fernando C.N. Pereira, "Hierarchical Distributed Representations for Statistical Language Modeling", . December 2004.
Abstract
Statistical language models estimate the probability of a word occurring in a given context. The most common language models rely on a discrete enumeration of predictive contexts (e.g., n-grams) and consequently fail to capture and exploit statistical regularities across these contexts. In this paper, we show how to learn hierarchical, distributed representations of word contexts that maximize the predictive value of a statistical language model. The representations are initialized by unsupervised algorithms for linear and nonlinear dimensionality reduction [14], then fed as input into a hierarchical mixture of experts, where each expert is a multinomial distribution over predicted words [12]. While the distributed representations in our model are inspired by the neural probabilistic language model of Bengio et al. [2, 3], our particular architecture enables us to work with significantly larger vocabularies and training corpora. For example, on a large-scale bigram modeling task involving a sixty thousand word vocabulary and a training corpus of three million sentences, we demonstrate consistent improvement over class-based bigram models [10, 13]. We also discuss extensions of our approach to longer multiword contexts.
Date Posted: 10 September 2005
This document has been peer reviewed.
Comments
Copyright MIT Press. Postprint version. Published in Advances in Neural Information Processing Systems 17, pages 185-192. Proceedings of the 18th annual Neural Information Processing Systems (NIPS) conference, held in Vancouver, Canada, from 13-18 December 2004.