**Instructor**

**Course Description**

Pattern Recognition (a.k.a. Machine Learning I) course focuses on the following topics in Machine Learning:

- Fundamentals:
- Probability & random variables. Univariate & multivariate.
- Maximum likelihood estimation. Risk & empirical risk minimization.
- Decision theory & Information theory for Machine Learning.
- Linear algebra for Machine Learning. Vector spaces. Matrix decomposition. Matrix calculus.
- Optimization. First & second order methods. Stochastic gradient descent. Constrained optimization.

- Linear discriminant analysis.
- Logistic regression.
- Linear regression.
- Generalized linear models.
- Exemplar-based methods. k nearest neighbors. Learning distance metrics. Kernel density estimation.
- Kernel methods. Gaussian processes. Support vector machines and relevance vector machines.
- Unsupervised learning and dimensionality reduction. Factor analysis and manifold learning.
- Clustering. k means. Mixture models. Spectral clustering.

**Textbook**

Kevin P. Murphy, “Probabilistic Machine Learning: An Introduction,” MIT Press, February 2022.

You can find a copy of the textbook here.

We will also occasionally use Murphy, K. P. (2022) “*Probabilistic Machine Learning: Advanced Topics*,” when discussing advanced topics, and Géron, A. (2019). “*Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems,*” O’Reilly Media, Inc., for coding and hands-on techniques.

**Expected Work**

Regular readings; mini-projects; in-class presentations of relevant paper; final course project.

You can find examples of some of the past projects here.

**Course Schedule**

Lecture | Date | Topic | Reading |
---|---|---|---|

1 | 2021-09-02 | Intorduction. Foundations I (Probability Univariate, Probability Multivariate, Statistics, Decision Theory) | PML I.1 - PML I.5 M.I.Jordan and T.M.Mitchell, "Machine learning: Trends, perspectives, and prospects," Science, 17 July 2015, Vol 349 Issue 6245 |

2 | 2021-09-09 | Foundations II (Information Theory, Linear Algebra & Matrix Calculus; Optimization) | PML I.6 - PML I.8 |

3 | 2021-09-16 | Linear Models I (LDA, Logistic regression - Binary, Multinomial, Robust, Bayesian) | PML II.9 - II.10 |

4 | 2021-09-23 | Linear Models II (Linear Regression - Ridge, Lasso, Robust, Bayesian); Generalized Linear Models | PML II.11 - PML II.12 |

5 | 2021-09-30 | Exemplar-based methods. k-NN; Distance metric learning; Kernel Density Estimation | PML IV.16 |

6 | 2021-10-07 | Kernel Methods. Mercer's theorem. Gaussian Processes. Support Vector Machines. | PML IV.17 |

7 | 2021-10-14 | Trees, Forests, Bagging & Boosting. | PML IV.18 |

8 | 2021-10-21 | Recap and discussions. Project proposals. | |

9 | 2021-10-28 | Learning with Fewer Labeled Examples. Data augmentation. Transfer Learning. Semi-supervised Learning. Active Learning. Meta Learning. Few-shot Learning. | PML IV.19 |

10 | 2021-11-04 | Dimensionality reduction I. Factor analysis. Autoencoders. | PML 20.1 - 20.3 |

11 | 2021-11-11 | Dimensionality reduction II. Manifold Learning. Word embeddings. | PML 20.4 - 20.5 |

12 | 2021-11-18 | Clustering. K-means. Mixture models. Spectral clustering. | PML 21 |

2021-11-25 | Thanksgiving recess | ||

13 | 2021-12-02 | Recommender systems. Dynamic models I. Linear dynamical models. Hidden Markov models. | PML 22. Additional material. |

14 | 2021-12-09 | Recap and discussions. Project follow-up. | |

2021-12-16 | Final Project Presentations | ||

2021-12-17 | Final Projects Due |

**Other Important Readings**

Abbreviation | Textbook Title | Author | Publisher | Year |
---|---|---|---|---|

PRML | Pattern Recognition and Machine Learning | Christopher C. Bishop | Springer | 2006 |

MLPP | Machine Learning: A Probabilistic Perspective | Kevin P. Murphy | MIT Press | 2012 |

CVMLI | Computer vision: models, learning and inference | Prince, Simon J D | Cambridge University Press | 2012 |

DL | Deep Learning | Goodfellow, Ian and Bengio, Yoshua and Courville, Aaron | MIT Press | 2016 |

FML | Foundations of Machine Learning | Mohri, Mehryar and Rostamizadeh, Afshin and Talwalkar, Ameet | MIT Press | 2012 |

DHS | Pattern Classification, 2nd ed | Duda, Richard O. and Hart, Peter E. and Stork, David G. | Wiley Interscience | 2004 |

ML | Machine Learning | Mitchell, Tom | McGraw Hill | 1997 |

I2ML | Introduction to Machine Learning, 2nd ed | Alpaydin, Ethem | MIT Press | 2012 |

MLAP | Machine Learning: An Algorithmic perspective | Marsland, Stephen | CRC press | 2009 |

PTPR | A Probabilistic Theory of Pattern Recognition | Devroye, Luc and Gyorfi, Laszlo and Lugosi, Gabor | Springer | 1997 |

ESL | The elements of statistical learning: Data mining, inference, and prediction | Friedman, J and Hastie, T and Tibshirani, R | Springer | 2009 |

NAPR | Netlab: Algorithms for Pattern Recognition | Nabney, Ian | Springer | 2002 |

DMPMLTT | Data Mining: Practical Machine Learning Tools and Techniques | Witten, Ian H and Frank, Eibe | Morgan Kaufmann | 2005 |

LAA | Linear Algebra and Its Applications | Strang, Gilbert | Elsevier Science | 2014 |

MC | Matrix computations, 4th ed | Golub, Gene H and Van Loan, Charles F | JHU Press | 2013 |

CO | Convex Optimization | Boyd, Steven P and Vandenberghe, Lieven | Cambridge University Press | 2004 |

ILCO | Introductory lectures on convex optimization: a basic course_ | Nestorov, Yurii | Springer | 2004 |

GPML | Gaussian Processes for Machine Learning | Rasmussen, Carl Edward and Williams, Christopher K. I. | MIT Press | 2006 |

ITILA | Information Theory, Inference, and Learning Algorithms | MacKay, David | Cambridge University Press | 2003 |

HOML | Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems | Géron, Aurélien | O’Reilly Media, Inc. | 2019 |

PMLI | Probabilistic Machine Learning: An Introduction | Murphy, Kevin P. | MIT Press | 2022 |

PMLA | Probabilistic Machine Learning: Advanced Topics | Murphy, Kevin P. | MIT Press | 2022 |