Instructor
Course Description
Pattern Recognition (a.k.a. Machine Learning I) course focuses on the following topics in Machine Learning:
- Fundamentals:
- Probability & random variables. Univariate & multivariate.
- Maximum likelihood estimation. Risk & empirical risk minimization.
- Decision theory & Information theory for Machine Learning.
- Linear algebra for Machine Learning. Vector spaces. Matrix decomposition. Matrix calculus.
- Optimization. First & second order methods. Stochastic gradient descent. Constrained optimization.
- Linear discriminant analysis.
- Logistic regression.
- Linear regression.
- Generalized linear models.
- Exemplar-based methods. k nearest neighbors. Learning distance metrics. Kernel density estimation.
- Kernel methods. Gaussian processes. Support vector machines and relevance vector machines.
- Unsupervised learning and dimensionality reduction. Factor analysis and manifold learning.
- Clustering. k means. Mixture models. Spectral clustering.
Textbook
Kevin P. Murphy, “Probabilistic Machine Learning: An Introduction,” MIT Press, February 2022.
You can find a copy of the textbook here.
We will also occasionally use Murphy, K. P. (2022) “Probabilistic Machine Learning: Advanced Topics,” when discussing advanced topics, and Géron, A. (2019). “Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems,” O’Reilly Media, Inc., for coding and hands-on techniques.
Expected Work
Regular readings; mini-projects; in-class presentations of relevant paper; final course project.
You can find examples of some of the past projects here.
Course Schedule
| Lecture | Date | Topic | Reading |
|---|---|---|---|
| 1 | 2021-09-02 | Intorduction. Foundations I (Probability Univariate, Probability Multivariate, Statistics, Decision Theory) | PML I.1 - PML I.5 M.I.Jordan and T.M.Mitchell, "Machine learning: Trends, perspectives, and prospects," Science, 17 July 2015, Vol 349 Issue 6245 |
| 2 | 2021-09-09 | Foundations II (Information Theory, Linear Algebra & Matrix Calculus; Optimization) | PML I.6 - PML I.8 |
| 3 | 2021-09-16 | Linear Models I (LDA, Logistic regression - Binary, Multinomial, Robust, Bayesian) | PML II.9 - II.10 |
| 4 | 2021-09-23 | Linear Models II (Linear Regression - Ridge, Lasso, Robust, Bayesian); Generalized Linear Models | PML II.11 - PML II.12 |
| 5 | 2021-09-30 | Exemplar-based methods. k-NN; Distance metric learning; Kernel Density Estimation | PML IV.16 |
| 6 | 2021-10-07 | Kernel Methods. Mercer's theorem. Gaussian Processes. Support Vector Machines. | PML IV.17 |
| 7 | 2021-10-14 | Trees, Forests, Bagging & Boosting. | PML IV.18 |
| 8 | 2021-10-21 | Recap and discussions. Project proposals. | |
| 9 | 2021-10-28 | Learning with Fewer Labeled Examples. Data augmentation. Transfer Learning. Semi-supervised Learning. Active Learning. Meta Learning. Few-shot Learning. | PML IV.19 |
| 10 | 2021-11-04 | Dimensionality reduction I. Factor analysis. Autoencoders. | PML 20.1 - 20.3 |
| 11 | 2021-11-11 | Dimensionality reduction II. Manifold Learning. Word embeddings. | PML 20.4 - 20.5 |
| 12 | 2021-11-18 | Clustering. K-means. Mixture models. Spectral clustering. | PML 21 |
| 2021-11-25 | Thanksgiving recess | ||
| 13 | 2021-12-02 | Recommender systems. Dynamic models I. Linear dynamical models. Hidden Markov models. | PML 22. Additional material. |
| 14 | 2021-12-09 | Recap and discussions. Project follow-up. | |
| 2021-12-16 | Final Project Presentations | ||
| 2021-12-17 | Final Projects Due |
Other Important Readings
| Abbreviation | Textbook Title | Author | Publisher | Year |
|---|---|---|---|---|
| PRML | Pattern Recognition and Machine Learning | Christopher C. Bishop | Springer | 2006 |
| MLPP | Machine Learning: A Probabilistic Perspective | Kevin P. Murphy | MIT Press | 2012 |
| CVMLI | Computer vision: models, learning and inference | Prince, Simon J D | Cambridge University Press | 2012 |
| DL | Deep Learning | Goodfellow, Ian and Bengio, Yoshua and Courville, Aaron | MIT Press | 2016 |
| FML | Foundations of Machine Learning | Mohri, Mehryar and Rostamizadeh, Afshin and Talwalkar, Ameet | MIT Press | 2012 |
| DHS | Pattern Classification, 2nd ed | Duda, Richard O. and Hart, Peter E. and Stork, David G. | Wiley Interscience | 2004 |
| ML | Machine Learning | Mitchell, Tom | McGraw Hill | 1997 |
| I2ML | Introduction to Machine Learning, 2nd ed | Alpaydin, Ethem | MIT Press | 2012 |
| MLAP | Machine Learning: An Algorithmic perspective | Marsland, Stephen | CRC press | 2009 |
| PTPR | A Probabilistic Theory of Pattern Recognition | Devroye, Luc and Gyorfi, Laszlo and Lugosi, Gabor | Springer | 1997 |
| ESL | The elements of statistical learning: Data mining, inference, and prediction | Friedman, J and Hastie, T and Tibshirani, R | Springer | 2009 |
| NAPR | Netlab: Algorithms for Pattern Recognition | Nabney, Ian | Springer | 2002 |
| DMPMLTT | Data Mining: Practical Machine Learning Tools and Techniques | Witten, Ian H and Frank, Eibe | Morgan Kaufmann | 2005 |
| LAA | Linear Algebra and Its Applications | Strang, Gilbert | Elsevier Science | 2014 |
| MC | Matrix computations, 4th ed | Golub, Gene H and Van Loan, Charles F | JHU Press | 2013 |
| CO | Convex Optimization | Boyd, Steven P and Vandenberghe, Lieven | Cambridge University Press | 2004 |
| ILCO | Introductory lectures on convex optimization: a basic course_ | Nestorov, Yurii | Springer | 2004 |
| GPML | Gaussian Processes for Machine Learning | Rasmussen, Carl Edward and Williams, Christopher K. I. | MIT Press | 2006 |
| ITILA | Information Theory, Inference, and Learning Algorithms | MacKay, David | Cambridge University Press | 2003 |
| HOML | Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems | Géron, Aurélien | O’Reilly Media, Inc. | 2019 |
| PMLI | Probabilistic Machine Learning: An Introduction | Murphy, Kevin P. | MIT Press | 2022 |
| PMLA | Probabilistic Machine Learning: Advanced Topics | Murphy, Kevin P. | MIT Press | 2022 |