Inference and IR study group

Overview

The Inference and IR study group has the following goals:

• Develop a working understanding of the basics of advanced statistical inference techniques.
• Understand how these methods are applied in existing models that are typically applied to text.
• Discuss how these methods may be applied to other IR problems.

It will be run as a short course / tutorial, much like Victor's machine translation tutorial. There will be presentations and discussion of the material so that all involved will develop a basic/deeper understanding of statistical inference techniques and their potential applicability to information retrieval.

Venue

We will meet Thursdays from 2:30-4:00 in room 243.

Schedule

I(a) Overview and Foundations

• Date: Thursday, September 29
• Primary Material:
  • Zoubin Ghahramani's ICML '04 Tutorial on Bayesian Learning pdf
• Comments: Overview of the area of Bayesian Machine learning. We will cover many inference techniques such as sampling, variational inference, expectation propogation, etc.

I(b) Overview and Foundations: Dirichlet Processes I

• Date: Thursday, October 6
• Primary Material:
  • Zoubin Ghahramani's UAI '05 Tutorial on Nonparametric Bayesian Methods pdf
  • Teg Grenager's "An Introduction to the Dirichlet Process" ppt
• Supplemental Material:
  • TBD
• Comments: We will cover non-Parametric Bayesian models such as Dirichlet process / Chinese restaurant process which we will encounter repeatedly in this seminar.

I(c) Overview and Foundations: Dirichlet Processes II

• Date: Thursday, October 13
• Primary Material:
  • A Nonparametric Hierarchical Bayesian Framework for Information Filtering. Yu, et. al. SIGIR 2004. pdf
  • Hierarchical Dirichlet Processes, Yee Whye Teh, Michael I. Jordan, Matthew J. Beal and David M. Blei, 2004, Tech. Report.pdf
• Supplemental Material:
  • Information Retrieval Using Hierarchical Dirichlet Processes. P. J. Cowans. Proceedings of the 27th Annual International Conference on Research and Development in Information Retrieval (SIGIR '04), 2004, pp. 564-565. pdf
• Comments: This week we will look at an application of Dirichlet Processes to text, as well as some discussion on hierarchical Dirichlet processes.

II(a) Inference using Sampling: Techniques

• Date: Thursday, October 20
• Primary Material:
  • An introduction to MCMC for machine learning. C. Andrieu, N. de Freitas, A. Doucet and M. I. Jordan. Machine Learning, 50, 5-43, 2003. pdf
  • Metropolis-Hastings Algorithm Explained. pdf
• Supplemental Material:
  • Probabilistic Inference using Markov Chain Monte Carlo Methods, Radford M. Neal, Technical Report. homepage
  • Introduction to Monte Carlo Methods. David MacKay. pdf

II(b) Inference using Sampling: IR models

• Date: Thursday, November 10
• Primary Material:
  • Markov Chain Sampling Methods for Dirichlet Process Mixture Models Homepage
• Supplemental Material:
  • Hierarchical Topic Models and the Nested Chinese Restaurant Process. Blei, D.M., Griffiths, T.L., Jordan, M.I., & Tenenbaum, J.B. Advances in Neural Information Processing Systems 16, 2004. pdf
  • Probabilistic models of text and images. David Blei's Ph.D. Thesis. U.C. Berkeley (2004). pdf
• Comments: This week we will look at an application of Gibbs sampling applied to (hierarchical) Dirichlet processes w/ application to text.

III(a) Variational Inference: Techniques

• Date:
• Primary Material:
  • An introduction to variational methods for graphical models. M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. In M. I. Jordan (Ed.), Learning in Graphical Models, Cambridge: MIT Press, 1999. pdf
• Supplemental Material:
  • Tutorial on Variational Approximation Methods. T. Jaakola, 2000. pdf
  • A variational principle for graphical models. M. J. Wainwright and M. I. Jordan. New Directions in Statistical Signal Processing: From Systems to Brain. Cambridge, MA: MIT Press, 2005. pdf

III(b) Variational Inference: IR Models

• Date:
• Primary Material:
  • Latent Dirichlet Allocation. D. Blei, A. Ng, and M. Jordan. Journal of Machine Learning Research, 3:993-1022, January 2003. pdf
  • Variational methods for Dirichlet process mixtures. D. Blei and M. Jordan. To appear, Bayesian Analysis, 2005 pdf
• Supplemental Material:
  • Modeling annotated data. D. Blei and M. Jordan. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval, pages 127-134. ACM Press, 2003. pdf
  • Integrating Topics and Syntax. Griffiths, T. L., Steyvers, M., Blei, D. M., & Tenenbaum, J. B. Advances in Neural Information Processing Systems 17, 2004. pdf

IV(a) Exact Inference: Techniques

• Date:
• Primary Material:
  • Inference in Belief Networks: A Procedural Guide. Huang and Darwiche. pdf
• Supplemental Material:
  • Graphical Models: Probabilistic Inference. M. I. Jordan and Y. Weiss. In M. Arbib (Ed.), The Handbook of Brain Theory and Neural Networks, 2nd edition. Cambridge, MA: MIT Press, 2002. ps

IV(b) Exact inference: IR models

• Date:
• Primary Material:
  • Inference networks for document retrieval. In Proceedings of the 13th Annual international ACM SIGIR Conference on Research and Development in information Retrieval, 1989. pdf
• Supplemental Material:
  • Combining the Language Model and Inference Network Approaches to Retrieval. Metzler, D. and Croft, W.B. Information Processing and Management Special Issue on Bayesian Networks and Information Retrieval, 40(5), 735-750, 2004. pdf

V. Advanced Topics

• Date:
• Primary Material:
  • TBD
• Supplemental Material:
  • Minka's "A Roadmap to Research on Expectation Propagation" homepage

-- DonMetzler - 20 Sep 2005
to top