• Date: Thursday, November 10

• Supplemental Material:
  • Information Retrieval Using Hierarchical Dirichlet Processes. P. J. Cowans. Proceedings of the 27th Annual International Conference on Research and Development in Information Retrieval (SIGIR '04), 2004, pp. 564-565. pdf

• Date: Thursday, October 20

• • Metropolis-Hastings Algorithm Explained. pdf

• Date: Thursday, October 27

• Comments: This week we will look at an application of Gibbs sampling applied to (hierarchical) Dirichlet processes w/ application to text.

Schedule

I(b) Overview and Foundations: Dirichlet Processes I

I(c) Overview and Foundations: Dirichlet Processes II

• Date: Thursday, October 13
• Primary Material:
  • A Nonparametric Hierarchical Bayesian Framework for Information Filtering. Yu, et. al. SIGIR 2004. pdf
  • Hierarchical Dirichlet Processes, Yee Whye Teh, Michael I. Jordan, Matthew J. Beal and David M. Blei, 2004, Tech. Report.pdf
• Comments: This week we will look at an application of Dirichlet Processes to text, as well as some discussion on hierarchical Dirichlet processes.

• • Teg Grenager's "An Introduction to the Dirichlet Process" ppt

• Date: Thursday, September 29

• Date: Thursday, October 6

Inference and IR study group

• Comments: Overview of the area of Bayesian Machine learning. We will cover many inference techniques such as sampling, variational inference, expectation propogation, etc.

• Comments: We will cover non-Parametric Bayesian models such as Dirichlet process / Chinese restaurant process which we will encounter repeatedly in this seminar.

I(a) Overview and Foundations

• Comments: We will get an overview of the area of Bayesian Machine learning. We will cover many inference techniques such as sampling, variational inference, expectation propogation, etc.

I(b) Overview and Foundations

• Date: Friday, October 7
• Primary Material:

• Supplemental Material:
  • TBD
• Comments: We will covered non-Parametric Bayesian models such as Dirichlet process / Chinese restaurant process which we will encounter repeatedly in this seminar.

II(a) Inference using Sampling: Techniques

II(b) Inference using Sampling: IR models

III(a) Variational Inference: Techniques

III(b) Variational Inference: IR Models

IV(a) Exact Inference: Techniques

IV(b) Exact inference: IR models

• • • Markov Chain Sampling Methods for Dirichlet Process Mixture Models Homepage
    • Hierarchical Dirichlet Processes, Yee Whye Teh, Michael I. Jordan, Matthew J. Beal and David M. Blei, 2004, Tech. Report.pdf
  • Supplemental Material:

• • • Latent Dirichlet Allocation. D. Blei, A. Ng, and M. Jordan. Journal of Machine Learning Research, 3:993-1022, January 2003. pdf

• • • Modeling annotated data. D. Blei and M. Jordan. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval, pages 127-134. ACM Press, 2003. pdf

I. Overview and Foundations

• • Date: Friday, September 30
  • Primary Material:

• • • Zoubin Ghahramani's UAI '05 Tutorial on Nonparametric Bayesian Methods pdf
  • Comments: We will get an overview of the area of Bayesian Machine learning. We will cover many inference techniques such as sampling, variational inference, expectation propogation, etc. Also covered will be non-Parametric Bayesian models such as Dirichlet process / Chinese restaurant process which we will encounter repeatedly in this seminar.

II a . Inference using sampling: Techniques

• • Date:
  • Primary Material:

• • Supplemental Material:

II b. Inference using Sampling: IR models

• • Date:
  • Primary Material: pdf
    • Hierarchical Topic Models and the Nested Chinese Restaurant Process. Blei, D.M., Griffiths, T.L., Jordan, M.I., & Tenenbaum, J.B. Advances in Neural Information Processing Systems 16, 2004. pdf
  • Supplemental Material:
    • Information Retrieval Using Hierarchical Dirichlet Processes. P. J. Cowans. Proceedings of the 27th Annual International Conference on Research and Development in Information Retrieval (SIGIR '04), 2004, pp. 564-565. pdf
    • Probabilistic models of text and images. David Blei's Ph.D. Thesis. U.C. Berkeley (2004). pdf
    • Modeling annotated data. D. Blei and M. Jordan. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval, pages 127-134. ACM Press, 2003. pdf

III a. Variational Inference: Techniques

• • Date:
  • Primary Material:

• • Supplemental Material:

III b. Variational Inference: IR Models

• • Date:
  • Primary Material:
    • Latent Dirichlet Allocation. D. Blei, A. Ng, and M. Jordan. Journal of Machine Learning Research, 3:993-1022, January 2003.

• • Supplemental Material:
    • Integrating Topics and Syntax. Griffiths, T. L., Steyvers, M., Blei, D. M., & Tenenbaum, J. B. Advances in Neural Information Processing Systems 17, 2004. pdf

IV a. Exact Inference: Techniques

• • Date:
  • Primary Material:
    • Inference in Belief Networks: A Procedural Guide. Huang and Darwiche. pdf
  • Supplemental Material:
    • Graphical Models: Probabilistic Inference. M. I. Jordan and Y. Weiss. In M. Arbib (Ed.), The Handbook of Brain Theory and Neural Networks, 2nd edition. Cambridge, MA: MIT Press, 2002. ps

IV b. Exact inference: IR models

• • Date:
  • Primary Material:

• • Supplemental Material:

V. Advanced Topics

• • Date:
  • Primary Material:
    • TBD
  • Supplemental Material:
    • Minka's "A Roadmap to Research on Expectation Propagation" homepage

• • Primary Material:

• • Supplemental Material:

• • Subject: Methods I (Exact)
  • Primary Material:
    • Inference in Belief Networks: A Procedural Guide. Huang and Darwiche. pdf
  • Supplemental Material:
    • Graphical Models: Probabilistic Inference. M. I. Jordan and Y. Weiss. In M. Arbib (Ed.), The Handbook of Brain Theory and Neural Networks, 2nd edition. Cambridge, MA: MIT Press, 2002. ps

• • Subject: Methods II (Sampling)
  • Primary Material:
    • An introduction to MCMC for machine learning. C. Andrieu, N. de Freitas, A. Doucet and M. I. Jordan. Machine Learning, 50, 5-43, 2003. pdf
  • Supplemental Material:
    • Probabilistic Inference using Markov Chain Monte Carlo Methods, Radford M. Neal, Technical Report. homepage
    • Introduction to Monte Carlo Methods. David MacKay. pdf

• • Subject: Methods III (Variational)
  • Primary Material:
    • An introduction to variational methods for graphical models. M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. In M. I. Jordan (Ed.), Learning in Graphical Models, Cambridge: MIT Press, 1999. pdf
  • Supplemental Material:
    • Tutorial on Variational Approximation Methods. T. Jaakola, 2000. pdf
    • A variational principle for graphical models. M. J. Wainwright and M. I. Jordan. New Directions in Statistical Signal Processing: From Systems to Brain. Cambridge, MA: MIT Press, 2005. pdf

• • Subject: Methods IV (Advanced)
  • Primary Material:
    • TBD
  • Supplemental Material:
    • Minka's "A Roadmap to Research on Expectation Propagation" homepage

• • Subject: Non-parametric Bayesian Methods (Dirichlet Processes / Chinese Restaraunt Processes)
  • Primary Material:
    • Zoubin Ghahramani's UAI '05 Tutorial on Nonparametric Bayesian Methods pdf
  • Supplemental Material:
    • Variational methods for Dirichlet process mixtures. D. Blei and M. Jordan. To appear, Bayesian Analysis, 2005 pdf

• • Subject: Applications I (Inference Network Framework)
  • Primary Material:
    • Inference networks for document retrieval. In Proceedings of the 13th Annual international ACM SIGIR Conference on Research and Development in information Retrieval, 1989. pdf
  • Supplemental Material:
    • Combining the Language Model and Inference Network Approaches to Retrieval. Metzler, D. and Croft, W.B. Information Processing and Management Special Issue on Bayesian Networks and Information Retrieval, 40(5), 735-750, 2004. pdf

• • Subject: Applications II (Latent Dirichlet Allocation / Hierarchical Latent Dirichlet Allocation)
  • Primary Material:
    • Latent Dirichlet Allocation. D. Blei, A. Ng, and M. Jordan. Journal of Machine Learning Research, 3:993-1022, January 2003. pdf
    • Hierarchical Topic Models and the Nested Chinese Restaurant Process. Blei, D.M., Griffiths, T.L., Jordan, M.I., & Tenenbaum, J.B. Advances in Neural Information Processing Systems 16, 2004. pdf
  • Supplemental Material:
    • Information Retrieval Using Hierarchical Dirichlet Processes. P. J. Cowans. Proceedings of the 27th Annual International Conference on Research and Development in Information Retrieval (SIGIR '04), 2004, pp. 564-565. pdf
    • Probabilistic models of text and images. David Blei's Ph.D. Thesis. U.C. Berkeley (2004). pdf
    • Modeling annotated data. D. Blei and M. Jordan. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval, pages 127-134. ACM Press, 2003. pdf

• • Subject: Applications III (HMM LDA)
  • Primary Material:
    • Integrating Topics and Syntax. Griffiths, T. L., Steyvers, M., Blei, D. M., & Tenenbaum, J. B. Advances in Neural Information Processing Systems 17, 2004. pdf
  • Supplemental Material:

• • Subject: Misc. Topics
  • Primary Material:
    • TBD
  • Supplemental Material:

• • • Probabilistic Inference using Markov Chain Monte Carlo Methods, Radford M. Neal, Technical Report. Homepage

Overview

• Develop a working understanding of the basics of advanced statistical inference techniques.
• Understand how these methods are applied in existing models that are typically applied to text.
• Discuss how these methods may be applied to other IR problems.

It will be run as a short course / tutorial, much like Victor's machine translation tutorial. There will be presentations and discussion of the material so that all involved will develop a basic/deeper understanding of statistical inference techniques and their potential applicability to information retrieval.

Venue

Tentative Schedule

• Day 1
  • Subject: Overview
  • Material:
    • Zoubin Ghahramani's ICML '04 Tutorial on Bayesian Learning pdf

• Day 2
  • Subject: Sampling Methods
  • Material:
    • An introduction to MCMC for machine learning. C. Andrieu, N. de Freitas, A. Doucet and M. I. Jordan. Machine Learning, 50, 5-43, 2003. pdf

• Day 3
  • Subject: Variational Methods
  • Material:
    • An introduction to variational methods for graphical models. M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. In M. I. Jordan (Ed.), Learning in Graphical Models, Cambridge: MIT Press, 1999. pdf

• Day 4
  • Subject: Advanced Methods
  • Material: ???

• Day 5
  • Subject: Non-parametric Bayesian Methods (Dirichlet Processes / Chinese Restaraunt Processes)
  • Material:
    • Zoubin Ghahramani's UAI '05 Tutorial on Nonparametric Bayesian Methods pdf

• Day 6
  • Subject: Applications I (Latent Dirichlet Allocation / Hierarchical Latent Dirichlet Allocation)
  • Material:
    • Latent Dirichlet Allocation. D. Blei, A. Ng, and M. Jordan. Journal of Machine Learning Research, 3:993-1022, January 2003. pdf
    • Hierarchical Topic Models and the Nested Chinese Restaurant Process. Blei, D.M., Griffiths, T.L., Jordan, M.I., & Tenenbaum, J.B. Advances in Neural Information Processing Systems 16, 2004. pdf

• Day 7
  • Subject: Applications II (HMM LDA / ???)
  • Material:
    • Integrating Topics and Syntax. Griffiths, T. L., Steyvers, M., Blei, D. M., & Tenenbaum, J. B. Advances in Neural Information Processing Systems 17, 2004. pdf

• Day 8
  • Subject: Applications III ( ??? )
  • Material:
    • ???

• Day 9
  • Subject: Misc. Topics
  • Material:
    • ???

• Day 10
  • Subject: Misc. Topics.
  • Material:
    • ???

-- DonMetzler - 20 Sep 2005