Instructor
Course Description
Pattern Recognition (a.k.a. Machine Learning I) course focuses Unsupervised Learning methods for data analysis. In particular, we cover topics such as:
- Association rules
- Density modeling
- Clustering (k-means, k-medoids, hierarchical models, spectral methods)
- Mixture models
- Distance metric learning
- Factor analysis (PCA, CCA, etc.) and other latent variable models
- Probabilistic topic models
- Matrix factorization
- Tensor models
- Clustering evaluation
Expected Work
Regular readings; mini-projects; in-class presentations; midterm and/or a final course project.
You can find examples of some of the past projects here.
Course Schedule
| Lec. # | Topic | Readings |
|---|---|---|
| 1 | Introduction & Overview | _ http://www.cs.cmu.edu/~tom/pubs/MachineLearning.pdf |
| _ M.I.Jordan and T.M.Mitchell, "Machine learning: Trends, perspectives, and prospects," Science, 17 July 2015, Vol 349 Issue 6245 | ||
| 1a | Probability & Linear Algebra | _ https://content.sakai.rutgers.edu/access/content/group/c626f8f2-9099-4059-b4b9-5794524e759d/algebra.pdf |
| _ https://content.sakai.rutgers.edu/access/content/group/c626f8f2-9099-4059-b4b9-5794524e759d/probability.pdf | ||
| 2 | Association Rules & | _ Chapter 14.1 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ |
| Frequent Itemsets | _ Chapter 14.2-14.2.3 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ | |
| 3 | _ Chapter 6 of http://www.mmds.org/#book | |
| 4 | Density Estimation | _ Chapters 6.6 through 6.9 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ |
| _ http://ned.ipac.caltech.edu/level5/March02/Silverman/Silver_contents.html | ||
| 5 | _ Chapters 4.1 - 4.4 of Duda, Hart, and Stork | |
| — | Homework #1 assigned | |
| Project proposals and in-class pitches assigned | ||
| 6 | K -means | _ Chapters 9.1 and 9.2 of http://robotics.stanford.edu/~nilsson/MLBOOK.pdf |
| _ Chapters 14.3.1 through 14.3.6 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ | ||
| _ Chapter 7 of http://www.mmds.org/#book | ||
| _ https://www.cs.rutgers.edu/~mlittman/courses/lightai03/jain99data.pdf | ||
| _ Chapters 13.1 and 13.2 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ | ||
| _ Chapters 10.1 – 10.4 and 10.7 of Duda, Hart, and Stork | ||
| 7 | Gaussian Mixtures & Expectation Maximization & Factor Analysis | _ Mixture of Gaussians: http://cs229.stanford.edu/notes/cs229-notes7b.pdf |
| _ The EM Algorithm: http://cs229.stanford.edu/notes/cs229-notes8.pdf | ||
| _ Factor Analysis: http://cs229.stanford.edu/notes/cs229-notes9.pdf | ||
| _ Chapters 14.3.7 through 14.3.9 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ | ||
| 8 | K -medoids & Hierarchical Clustering | _ Chapter 14.3.10 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ |
| _ Chapter 14.3.12 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ | ||
| _ Chapter 9.3 of http://robotics.stanford.edu/~nilsson/MLBOOK.pdf | ||
| _ Chapter 10.9 of Duda, Hart, and Stork | ||
| 9 | Evaluation Metrics & Practical Issues | _ http://web.itu.edu.tr/sgunduz/courses/verimaden/paper/validity_survey.pdf |
| _ Chapter 14.3.11 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ | ||
| — | Homework #1 due at 11:59 PM Eastern | |
| 10 | Distance/Similarity Measures & Metric Learning | _ http://web.cse.ohio-state.edu/~kulis/pubs/ftml_metric_learning.pdf |
| _ Check out the Encyclopedia of Distances on this course’s Sakai site (under Resources). | ||
| — | Homework #2 assigned | |
| 11 | Principal Component Analysis (PCA) & Singular Value Decomposition (SVD) | _ Chapter 14.5 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ |
| _ Chapter 11 of http://www.mmds.org/#book | ||
| 12 | Spectral Clustering & Graph Clustering | _ http://ai.stanford.edu/~ang/papers/nips01-spectral.pdf |
| _ http://www.cs.columbia.edu/~jebara/4772/papers/Luxburg07_tutorial.pdf | ||
| _ [Optional] http://arxiv.org/pdf/0906.0612.pdf | ||
| — | Homework #1 graded | Note: Warning grades will be issued by Fri Oct 16. |
| 13 | Kernel Principal Components & Independent Component Analysis (ICA) & Canonical Correlation Analysis (CCA) & PageRank | _ Chapter 14.5 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ |
| _ ICA: Chapter 14.7 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ | ||
| _ CCA: https://www.cs.cmu.edu/~tom/10701_sp11/slides/CCA_tutorial.pdf | ||
| _ PageRank: | ||
| o Chapter 5 of http://www.mmds.org/#book | ||
| o Chapter 14.10 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ | ||
| 14 | o https://www.cs.purdue.edu/homes/dgleich/publications/Gleich%202015%20-%20prbeyond.pdf | |
| — | Homework #2 due at 11:59 PM Eastern | |
| — | Two-page project proposals due at 11:59 PM Eastern | |
| 15 | In-class project pitches | |
| 16 | Recommendation Systems | _ http://infolab.stanford.edu/~ullman/mmds/ch9.pdf |
| _ http://eliassi.org/papers/chaney-recsys15.pdf | ||
| 17 | Midterm exam | |
| — | Project proposals & pitches graded | |
| Project presentations and reports assigned | ||
| 18 | Latent Variable Models & | _ http://research.microsoft.com/pubs/67187/bishop-latent-erice-99.pdf |
| Probabilistic Topic Models | _ http://www.cs.columbia.edu/~blei/papers/Blei2012.pdf | |
| _ http://www.cs.princeton.edu/~blei/papers/Blei2011.pdf | ||
| 19 | _ http://www.cs.columbia.edu/~blei/papers/BleiLafferty2009.pdf | |
| — | Homework #2 graded | |
| 20 | Latent Variable Models & | _ http://www.cs.berkeley.edu/~jordan/papers/variational-intro.pdf |
| Probabilistic Topic Models (continued) | _ http://www.cs.ubc.ca/~arnaud/andrieu_defreitas_doucet_jordan_intromontecarlomachinelearning.pdf | |
| 21 | _ https://www.ee.washington.edu/techsite/papers/documents/UWEETR-2010-0006.pdf | |
| 22 | Matrix Factorization | _ Chapter 14.6 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ |
| _ http://papers.nips.cc/paper/1861-algorithms-for-non-negative-matrix-factorization.pdf | ||
| 23a | Tensor Factorization | _ http://www.sandia.gov/~tgkolda/pubs/pubfiles/TensorReview.pdf |
| 23b | Midterm exam graded and returned at the end of lecture | |
| Thanksgiving Holiday – no class | ||
| 24 | Sequence Models | |
| 25 | Model Selection | _ Chapter 7 of http://statweb.stanford.edu/~tibs/ElemStatLearn/ |
| 26 | Theory of Clustering | _ http://www.cs.cornell.edu/home/kleinber/nips15.pdf |
| _ http://papers.nips.cc/paper/3491-measures-of-clustering-quality-a-working-set-of-axioms-for-clustering.pdf | ||
| 27 | In-class project presentations | |
| 27 | In-class project presentations | |
| Last day of classes | ||
| — | Project presentations graded | |
| — | Project reports due at 11:59 PM Eastern | |
| — | Project reports graded and final grades released. | |
Textbooks
| Abbreviation | Textbook Title | Author | Publisher | Year |
|---|---|---|---|---|
| PRML | Pattern Recognition and Machine Learning | Christopher C. Bishop | Springer | 2006 |
| MLPP | Machine Learning: A Probabilistic Perspective | Kevin P. Murphy | MIT Press | 2012 |
| CVMLI | Computer vision: models, learning and inference | Prince, Simon J D | Cambridge University Press | 2012 |
| DL | Deep Learning | Goodfellow, Ian and Bengio, Yoshua and Courville, Aaron | MIT Press | 2016 |
| FML | Foundations of Machine Learning | Mohri, Mehryar and Rostamizadeh, Afshin and Talwalkar, Ameet | MIT Press | 2012 |
| DHS | Pattern Classification, 2nd ed | Duda, Richard O. and Hart, Peter E. and Stork, David G. | Wiley Interscience | 2004 |
| ML | Machine Learning | Mitchell, Tom | McGraw Hill | 1997 |
| I2ML | Introduction to Machine Learning, 2nd ed | Alpaydin, Ethem | MIT Press | 2012 |
| MLAP | Machine Learning: An Algorithmic perspective | Marsland, Stephen | CRC press | 2009 |
| PTPR | A Probabilistic Theory of Pattern Recognition | Devroye, Luc and Gyorfi, Laszlo and Lugosi, Gabor | Springer | 1997 |
| ESL | The elements of statistical learning: Data mining, inference, and prediction | Friedman, J and Hastie, T and Tibshirani, R | Springer | 2009 |
| NAPR | Netlab: Algorithms for Pattern Recognition | Nabney, Ian | Springer | 2002 |
| DMPMLTT | Data Mining: Practical Machine Learning Tools and Techniques | Witten, Ian H and Frank, Eibe | Morgan Kaufmann | 2005 |
| LAA | Linear Algebra and Its Applications | Strang, Gilbert | Elsevier Science | 2014 |
| MC | Matrix computations, 4th ed | Golub, Gene H and Van Loan, Charles F | JHU Press | 2013 |
| CO | Convex Optimization | Boyd, Steven P and Vandenberghe, Lieven | Cambridge University Press | 2004 |
| ILCO | Introductory lectures on convex optimization: a basic course_ | Nestorov, Yurii | Springer | 2004 |
| GPML | Gaussian Processes for Machine Learning | Rasmussen, Carl Edward and Williams, Christopher K. I. | MIT Press | 2006 |
| ITILA | Information Theory, Inference, and Learning Algorithms | MacKay, David | Cambridge University Press | 2003 |