CS 3750: Advanced Topics in Machine Learning (ISSP 3535)

CS 3750 Advanced Topics in Machine Learning (ISSP 3535)

Time: Monday, Wednesday 4:00-5:20pm
Location: Sennott Square, Room 5313

Instructor: Milos Hauskrecht
Computer Science Department
5329 Sennott Square
phone: x4-8845
e-mail: milos@cs.pitt.edu
office hours: Tuesday 2:30-4:00pm, Wednesday 11:00-12:00am

Announcements !!!!!

Project proposals are due on Monday, November 12, 2003 at 4pm.
Midterm project (new !!!)
List of readings for the course . The list is organized along different topics and includes additional readings and references you may find useful if you want to explore some topics in more depth.

Links

Course description
Lectures
Readings
Paper presentations
Projects

Abstract

The goal of the field of machine learning is to build computer systems that learn from experience and that are capable to adapt to their environments. Learning techniques and methods developed by researchers in this field have been successfully applied to a variety of learning tasks in a broad range of areas, including, for example, text classification, gene discovery, financial forecasting, credit card fraud detection, collaborative filtering, design of adaptive web agents and others.

The objective of the Advances Machine Learning course is to expand on the material covered in the introductory Machine Learning course (CS2750), and focus on the most recent advances in the ML field such as, kernel and variational methods. The course will consist of a mix of lectures, paper presentations and discussions. Students will be evaluated based on their participation in discussions, paper presentations and projects.

Prerequisites

CS 2750 Machine Learning , or the permission of the instructor.

Syllabus

Readings:
Readings for different topics
The primary book for the course is:

Trevor Hastie, Robert Tibshirani, Jerome Friedman. The elements of statistical learning. Springer, 2001.
Other books you may find useful:

C. Bishop. Neural networks for Pattern Recognition. Oxford University Press, 1995.
R.O. Duda, P.E. Hart, D.G. Stork. Pattern Classification. Second edition. John Wiley and Sons, 2000.
J. Han, M. Kamber. Data mining. Concepts and Techniques. Morgan Kauffman, 2001.
T. Mitchell. Machine Learning. Mc Graw Hill, 1997.
B. Schokopf and A. Smola. Learning with kernels. MIT Press, 2002. Readings

Lectures

Lectures Topic(s)

August 25 Administrivia and Course Overview.
Syllabus of the course
Readings:

Lecture notes from CS2750

August 27 Review of density estimation
Readings:

Lecture notes from CS2750

September 3 Review of density estimation and exponential family of distributions
Readings:
Lecture notes from CS2750
M. Jordan. Eponential family. In Graphical models. Chapter 7.

September 8 Multivariate normal distribution.
Readings:

Lecture notes from CS2750.
M. Jordan. The Multivariate Gaussian. In Graphical models. Chapter 12.

September 10 Subset Selection and Regularization. (Brano Kveton)
Readings:

Hastie, Tibshirani and Friedman. Elements of statistical learning. Sections: 3.4, 4.3.1, 5.6., 5.8.

September 15 PCA and SVD. (Mark Fenner)
Readings:

Lecture notes for CS2750
Hastie, Tibshirani and Friedman. Elements of statistical learning. Section 14.5.
Applications:

Jon M. Kleinberg. Authoritative Sources in a Hyperlinked Environment, Journal of ACM. 1999.
Michael W. Berry, Zlatko Drmac, Elizabeth R. Jessup. Matrices, Vector Spaces, and Information Retrieval, SIAM Review, 1999.

September 17 Overview of Bayesian belief networks (BBNs). (Milos)
Readings:

Lecture notes for CS2750
TBA

September 22 BBN Inference: Junction tree algorithm. (Mihai Rotaru)
Readings:

C. Huang, A. Darwiche. Inference in Belief Networks: A Procedural Guide.
M. Jordan. Graphical models. Chapter 16. (see CS 2750 web page from Spring 02)

September 24 BBN Inference: Pearl algorithm. (Tomas Singliar)
Readings:

Distributed during the class.

September 29 BBN Inference: Intorduction to Monte Carlo methods. (Changhe Yuan)
Readings (general intro to Monte Carlo):

Rubinstein. Simulation and the Monte Carlo Method. 1981.
David MacKay. Introduction to Monte Carlo methods.
Andrieu et al. An introduction to MCMC for Machine Learning. Machine Learning, vol. 50, pp.5-43, 2003.

October 1 BBN Inference: Monte Carlo methods (cont.).
Readings (general intro to Monte Carlo):

Rubinstein. Simulation and the Monte Carlo Method. 1981.
David MacKay. Introduction to Monte Carlo methods.
Andrieu et al. An introduction to MCMC for Machine Learning. Machine Learning, vol. 50, pp.5-43, 2003.
Monte Carlo for BBNs:
A brief summary of existing Importance Sampling Algorithms for Bayesian Networks

October 6 Learning probabilistic networks from data (Milos)
Readings:

Lecture notes for CS2750. Lecture 14.
Wray Buntine. A Guide to the Literature on Learning Probabilistic Networks From Data. 1996.
David Heckerman A Tutorial on Learning With Bayesian Networks 1996.
Wray Buntine. Operations for Learning with Graphical Models JAIR, 1994.

October 8 Expectation-maximization (EM) (Milos)
Readings:

Lecture notes for CS2750. Lecture 15.
A.P. Dempster, N.M. Laird, D.B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of Royal statistical society, vol. 39, issue 1, pp. 1-28, 1977.
Expectation-maximization in M. Jordan, C. Bishop. Introduction to graphical models.

Midterm project

October 13 Latent variable models: Probabilistic PCA (Vahan)
Readings:

Michael E. Tipping, Chris M. Bishop. Probabilistic Principal Component Analysis, 1999. (TR in 1997)
Sam Roweis. EM Algorithms for PCA and SPCA. NIPS-1998.

October 15 Latent variable models: Variational approximations and learning. (Milos)
Readings (introduction to variational approximations):

Jordan et al An Introduction to Variational Methods for Graphical Models, 1998.

Tommi Jaakkola Tutorial on variational approximation methods

Ghahramani and Beal (2000) Graphic Models and Variational Methods

October 20 Latent variable models: Variational learning. (Milos)
Readings:

Ghahramani and Beal (2000) Graphic Models and Variational Methods

October 22 Latent variable models: Variational Bayesian learning. (Milos)
Readings:

Ghahramani and Beal (2000) Graphic Models and Variational Methods
X. Lu, M. Hauskrecht, R.S. Day. Variational Bayesian learning of the cooperative vector quantizer model. Part I: The Theory. Technical Report, Center for Biomedical Informatics, CBMI-02-181, 2002.
Haigai Attias Inferring parameters and structure of latent variable models by variational Bayes (1999)

Lawrence N.D. and Bishop C.M. Variational Bayesian Independent Component Analysis 1999

Ghahramani and Beal Variational Inference for Bayesian Mixture of Factor Analyzers , 2000.

October 27 Latent variable models: Variational Bayesian learning. (Milos)
Readings:

Ghahramani and Beal (2000) Graphic Models and Variational Methods
X. Lu, M. Hauskrecht, R.S. Day. Variational Bayesian learning of the cooperative vector quantizer model. Part I: The Theory. Technical Report, Center for Biomedical Informatics, CBMI-02-181, 2002.
Haigai Attias Inferring parameters and structure of latent variable models by variational Bayes (1999)

Lawrence N.D. and Bishop C.M. Variational Bayesian Independent Component Analysis 1999

Ghahramani and Beal Variational Inference for Bayesian Mixture of Factor Analyzers , 2000.

October 29 Structural EM. (Richard)
Readings:

Nir Friedman. The Bayesian structural EM algorithm. In Fourteenth Conf. on Uncertainty in Artificial Intelligence (UAI). 1998
Singh, Moninder. Learning Bayesian Networks from Incomplete Data. In Proceedings of the 14th National Conf. on Artificial Intelligence (AAAI'97), Providence, RI, July 27-31.

November 3 Midterm projects

November 5 Support Vector Machines (Milos)
Readings:

Lecture notes from CS2750
Christopher J.C. Burges A Tutorial on Support Vector Machines for Pattern Recognition
www.kernel-machines.org

November 10 Support Vector Machines for Regression (Min)
Readings:

Alex J. Smola, Bernhard Schölkopf. A Tutorial on Support Vector Regression.

November 12 Kernel trick for distances. (Brano)
Readings:

Bernhard Scholkopf. The Kernel Trick for Distances. NIPS-2000.

November 17 Kernel PCA, ICA (Oleg)
Readings:

B. Schokopf, A. Smola, and KR. Muller. Nonlinear Component Analysis as a Kernel Eigenvalue Problem. Neural Computation 1998. (see Summary by Ian Fasel)
Francis R. Bach, Michael I. Jordan. Kernel Independent Component Analysis. 2001.

November 19 Kernels for Structures (Tomas)
Readings:

Thomas Gartner. A survey of Kernels for Structured Data. SIGKDD Explorations -- Newsletter of the ACM Special Interest Group on Knowledge Discovery and Data Mining. Vol. 5, 2003.
S. V. N. Vishwanathan, Alexander J. Smola. Fast Kernels for String and Tree Matching. NIPS-2002 (or a more recent paper )

November 24 Ensamble methods: Bagging and Boosting (Milos)
Readings:

L. Breiman. Arcing Classifiers.
Yoav Freund, Robert E. Schapire. A Short Introduction to Boosting (1999)
Yoav Freund, Robert E. Schapire.Experiments with a New Boosting Algorithm (1996)

December 1 Ensamble methods: Boosting the margin (Oleg and Min)
Readings:

Robert E. Schapire, Yoav Freund, Peter Bartlett, Wee Sun Lee. Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods (1998)

December 3 Multinomial PCA (Changhe and Mihai)
Readings:

December 5 Final projects
Readings:

Course webpage for CS2750, the introductory Machine Learning course from Spring 2003. It is the prerequisite of CS3750.

Readings
Readings will be assigned before the class at which the discussion on the topic covered by the paper takes place. Most of the readings will be electronic, however, some readings will be in the paper form or from the books. See the list of Readings for different topics

Paper discussions
Each student is expected to present 1-2 papers in the course of the semester and lead the discussion on the paper. The paper will be distributed to students the same way as other readings. The assignement of the papers will be discussed in the first week of the course.

Projects
There are no homeworks and exams in this course. However, students will be asked to prepare, submit and present two projects. The first project will be assigned and due in the middle of the semester. The final project (due at the end of the semester) and is more flexible: a student can choose his/her own topic to investigate. You will need to submit a short (one page) proposal for the purpose of approval and feedback for the final project. The final project must have a distinctive and non-trivial learning or adaptive component.

Last updated by milos on 08/13/2003

Lectures	Topic(s)
August 25	Administrivia and Course Overview. Syllabus of the course Readings: Lecture notes from CS2750
August 27	Review of density estimation Readings: Lecture notes from CS2750
September 3	Review of density estimation and exponential family of distributions Readings: Lecture notes from CS2750 M. Jordan. Eponential family. In Graphical models. Chapter 7.
September 8	Multivariate normal distribution. Readings: Lecture notes from CS2750. M. Jordan. The Multivariate Gaussian. In Graphical models. Chapter 12.
September 10	Subset Selection and Regularization. (Brano Kveton) Readings: Hastie, Tibshirani and Friedman. Elements of statistical learning. Sections: 3.4, 4.3.1, 5.6., 5.8.
September 15	PCA and SVD. (Mark Fenner) Readings: Lecture notes for CS2750 Hastie, Tibshirani and Friedman. Elements of statistical learning. Section 14.5. Applications: Jon M. Kleinberg. Authoritative Sources in a Hyperlinked Environment, Journal of ACM. 1999. Michael W. Berry, Zlatko Drmac, Elizabeth R. Jessup. Matrices, Vector Spaces, and Information Retrieval, SIAM Review, 1999.
September 17	Overview of Bayesian belief networks (BBNs). (Milos) Readings: Lecture notes for CS2750 TBA
September 22	BBN Inference: Junction tree algorithm. (Mihai Rotaru) Readings: C. Huang, A. Darwiche. Inference in Belief Networks: A Procedural Guide. M. Jordan. Graphical models. Chapter 16. (see CS 2750 web page from Spring 02)
September 24	BBN Inference: Pearl algorithm. (Tomas Singliar) Readings: Distributed during the class.
September 29	BBN Inference: Intorduction to Monte Carlo methods. (Changhe Yuan) Readings (general intro to Monte Carlo): Rubinstein. Simulation and the Monte Carlo Method. 1981. David MacKay. Introduction to Monte Carlo methods. Andrieu et al. An introduction to MCMC for Machine Learning. Machine Learning, vol. 50, pp.5-43, 2003.
October 1	BBN Inference: Monte Carlo methods (cont.). Readings (general intro to Monte Carlo): Rubinstein. Simulation and the Monte Carlo Method. 1981. David MacKay. Introduction to Monte Carlo methods. Andrieu et al. An introduction to MCMC for Machine Learning. Machine Learning, vol. 50, pp.5-43, 2003. Monte Carlo for BBNs: A brief summary of existing Importance Sampling Algorithms for Bayesian Networks
October 6	Learning probabilistic networks from data (Milos) Readings: Lecture notes for CS2750. Lecture 14. Wray Buntine. A Guide to the Literature on Learning Probabilistic Networks From Data. 1996. David Heckerman A Tutorial on Learning With Bayesian Networks 1996. Wray Buntine. Operations for Learning with Graphical Models JAIR, 1994.
October 8	Expectation-maximization (EM) (Milos) Readings: Lecture notes for CS2750. Lecture 15. A.P. Dempster, N.M. Laird, D.B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of Royal statistical society, vol. 39, issue 1, pp. 1-28, 1977. Expectation-maximization in M. Jordan, C. Bishop. Introduction to graphical models. Midterm project
October 13	Latent variable models: Probabilistic PCA (Vahan) Readings: Michael E. Tipping, Chris M. Bishop. Probabilistic Principal Component Analysis, 1999. (TR in 1997) Sam Roweis. EM Algorithms for PCA and SPCA. NIPS-1998.
October 15	Latent variable models: Variational approximations and learning. (Milos) Readings (introduction to variational approximations): Jordan et al An Introduction to Variational Methods for Graphical Models, 1998. Tommi Jaakkola Tutorial on variational approximation methods Ghahramani and Beal (2000) Graphic Models and Variational Methods
October 20	Latent variable models: Variational learning. (Milos) Readings: Ghahramani and Beal (2000) Graphic Models and Variational Methods
October 22	Latent variable models: Variational Bayesian learning. (Milos) Readings: Ghahramani and Beal (2000) Graphic Models and Variational Methods X. Lu, M. Hauskrecht, R.S. Day. Variational Bayesian learning of the cooperative vector quantizer model. Part I: The Theory. Technical Report, Center for Biomedical Informatics, CBMI-02-181, 2002. Haigai Attias Inferring parameters and structure of latent variable models by variational Bayes (1999) Lawrence N.D. and Bishop C.M. Variational Bayesian Independent Component Analysis 1999 Ghahramani and Beal Variational Inference for Bayesian Mixture of Factor Analyzers , 2000.
October 27	Latent variable models: Variational Bayesian learning. (Milos) Readings: Ghahramani and Beal (2000) Graphic Models and Variational Methods X. Lu, M. Hauskrecht, R.S. Day. Variational Bayesian learning of the cooperative vector quantizer model. Part I: The Theory. Technical Report, Center for Biomedical Informatics, CBMI-02-181, 2002. Haigai Attias Inferring parameters and structure of latent variable models by variational Bayes (1999) Lawrence N.D. and Bishop C.M. Variational Bayesian Independent Component Analysis 1999 Ghahramani and Beal Variational Inference for Bayesian Mixture of Factor Analyzers , 2000.
October 29	Structural EM. (Richard) Readings: Nir Friedman. The Bayesian structural EM algorithm. In Fourteenth Conf. on Uncertainty in Artificial Intelligence (UAI). 1998 Singh, Moninder. Learning Bayesian Networks from Incomplete Data. In Proceedings of the 14th National Conf. on Artificial Intelligence (AAAI'97), Providence, RI, July 27-31.
November 3	Midterm projects
November 5	Support Vector Machines (Milos) Readings: Lecture notes from CS2750 Christopher J.C. Burges A Tutorial on Support Vector Machines for Pattern Recognition www.kernel-machines.org
November 10	Support Vector Machines for Regression (Min) Readings: Alex J. Smola, Bernhard Schölkopf. A Tutorial on Support Vector Regression.
November 12	Kernel trick for distances. (Brano) Readings: Bernhard Scholkopf. The Kernel Trick for Distances. NIPS-2000.
November 17	Kernel PCA, ICA (Oleg) Readings: B. Schokopf, A. Smola, and KR. Muller. Nonlinear Component Analysis as a Kernel Eigenvalue Problem. Neural Computation 1998. (see Summary by Ian Fasel) Francis R. Bach, Michael I. Jordan. Kernel Independent Component Analysis. 2001.
November 19	Kernels for Structures (Tomas) Readings: Thomas Gartner. A survey of Kernels for Structured Data. SIGKDD Explorations -- Newsletter of the ACM Special Interest Group on Knowledge Discovery and Data Mining. Vol. 5, 2003. S. V. N. Vishwanathan, Alexander J. Smola. Fast Kernels for String and Tree Matching. NIPS-2002 (or a more recent paper )
November 24	Ensamble methods: Bagging and Boosting (Milos) Readings: L. Breiman. Arcing Classifiers. Yoav Freund, Robert E. Schapire. A Short Introduction to Boosting (1999) Yoav Freund, Robert E. Schapire.Experiments with a New Boosting Algorithm (1996)
December 1	Ensamble methods: Boosting the margin (Oleg and Min) Readings: Robert E. Schapire, Yoav Freund, Peter Bartlett, Wee Sun Lee. Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods (1998)
December 3	Multinomial PCA (Changhe and Mihai) Readings:
December 5	Final projects Readings: