cs535 – Pattern Recognition

Instructor

Vladimir Pavlovic

Course Description

Pattern Recognition (a.k.a. Machine Learning I) course focuses on the following topics in Machine Learning:

  • Fundamentals:
    • Probability & random variables.  Univariate & multivariate.
    • Maximum likelihood estimation.  Risk & empirical risk minimization.
    • Decision theory & Information theory for Machine Learning.
    • Linear algebra for Machine Learning. Vector spaces.  Matrix decomposition.  Matrix calculus.
    • Optimization.  First & second order methods.  Stochastic gradient descent.  Constrained optimization.
  • Linear discriminant analysis.
  • Logistic regression.
  • Linear regression.
  • Generalized linear models.
  • Exemplar-based methods.  k nearest neighbors. Learning distance metrics. Kernel density estimation.
  • Kernel methods.  Gaussian processes.  Support vector machines and relevance vector machines.
  • Unsupervised learning and dimensionality reduction.  Factor analysis and manifold learning.
  • Clustering.  k means.  Mixture models.  Spectral clustering.

Textbook

Kevin P. Murphy, “Probabilistic Machine Learning: An Introduction,” MIT Press, February 2022.

You can find a copy of the textbook here.

We will also occasionally use Murphy, K. P. (2022) “Probabilistic Machine Learning: Advanced Topics,” when discussing advanced topics, and Géron, A. (2019). “Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems,” O’Reilly Media, Inc., for coding and hands-on techniques.

Expected Work

Regular readings; mini-projects; in-class presentations of relevant paper;  final course project.

You can find examples of some of the past projects here.

Course Schedule

LectureDateTopicReading
12021-09-02 Intorduction.
Foundations I (Probability Univariate, Probability Multivariate, Statistics, Decision Theory)
PML I.1 - PML I.5
M.I.Jordan and T.M.Mitchell, "Machine learning: Trends, perspectives, and prospects," Science, 17 July 2015, Vol 349 Issue 6245
22021-09-09 Foundations II (Information Theory, Linear Algebra & Matrix Calculus; Optimization)PML I.6 - PML I.8
32021-09-16 Linear Models I (LDA, Logistic regression - Binary, Multinomial, Robust, Bayesian)PML II.9 - II.10
42021-09-23Linear Models II (Linear Regression - Ridge, Lasso, Robust, Bayesian); Generalized Linear ModelsPML II.11 - PML II.12
52021-09-30Exemplar-based methods. k-NN; Distance metric learning; Kernel Density EstimationPML IV.16
62021-10-07Kernel Methods. Mercer's theorem. Gaussian Processes. Support Vector Machines.PML IV.17
72021-10-14Trees, Forests, Bagging & Boosting.PML IV.18
82021-10-21Recap and discussions. Project proposals.
92021-10-28Learning with Fewer Labeled Examples. Data augmentation. Transfer Learning. Semi-supervised Learning. Active Learning. Meta Learning. Few-shot Learning.PML IV.19
102021-11-04Dimensionality reduction I. Factor analysis. Autoencoders.PML 20.1 - 20.3
112021-11-11Dimensionality reduction II. Manifold Learning. Word embeddings.PML 20.4 - 20.5
122021-11-18Clustering. K-means. Mixture models. Spectral clustering.PML 21
2021-11-25Thanksgiving recess
132021-12-02Recommender systems. Dynamic models I. Linear dynamical models. Hidden Markov models.PML 22. Additional material.
142021-12-09Recap and discussions. Project follow-up.
2021-12-16Final Project Presentations
2021-12-17Final Projects Due

Other Important Readings

AbbreviationTextbook TitleAuthorPublisherYear
PRMLPattern Recognition and Machine LearningChristopher C. BishopSpringer2006
MLPPMachine Learning: A Probabilistic PerspectiveKevin P. MurphyMIT Press2012
CVMLIComputer vision: models, learning and inferencePrince, Simon J DCambridge University Press2012
DLDeep LearningGoodfellow, Ian and Bengio, Yoshua and Courville, AaronMIT Press2016
FMLFoundations of Machine LearningMohri, Mehryar and Rostamizadeh, Afshin and Talwalkar, AmeetMIT Press2012
DHSPattern Classification, 2nd edDuda, Richard O. and Hart, Peter E. and Stork, David G.Wiley Interscience2004
MLMachine LearningMitchell, TomMcGraw Hill1997
I2MLIntroduction to Machine Learning, 2nd edAlpaydin, EthemMIT Press2012
MLAPMachine Learning: An Algorithmic perspectiveMarsland, StephenCRC press2009
PTPRA Probabilistic Theory of Pattern RecognitionDevroye, Luc and Gyorfi, Laszlo and Lugosi, GaborSpringer1997
ESLThe elements of statistical learning: Data mining, inference, and predictionFriedman, J and Hastie, T and Tibshirani, RSpringer2009
NAPRNetlab: Algorithms for Pattern RecognitionNabney, IanSpringer2002
DMPMLTTData Mining: Practical Machine Learning Tools and TechniquesWitten, Ian H and Frank, EibeMorgan Kaufmann2005
LAALinear Algebra and Its ApplicationsStrang, GilbertElsevier Science2014
MCMatrix computations, 4th edGolub, Gene H and Van Loan, Charles FJHU Press2013
COConvex OptimizationBoyd, Steven P and Vandenberghe, LievenCambridge University Press2004
ILCOIntroductory lectures on convex optimization: a basic course_Nestorov, YuriiSpringer2004
GPMLGaussian Processes for Machine LearningRasmussen, Carl Edward and Williams, Christopher K. I. MIT Press2006
ITILAInformation Theory, Inference, and Learning AlgorithmsMacKay, DavidCambridge University Press2003
HOMLHands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent SystemsGéron, AurélienO’Reilly Media, Inc.2019
PMLIProbabilistic Machine Learning: An IntroductionMurphy, Kevin P.MIT Press2022
PMLAProbabilistic Machine Learning: Advanced TopicsMurphy, Kevin P.MIT Press2022