[expand]

[expand all]

[collapse all]

Overview

This course covers a selected set of topics in machine learning and data mining, with an emphasis on good methods and practices for deployment of real systems. The majority of sections are related to commonly used supervised learning techniques, and to a lesser degree unsupervised methods. This includes fundamentals of algorithms on linear and logistic regression, decision trees, support vector machines, clustering, neural networks, as well as key techniques for feature selection and dimensionality reduction, error estimation and empirical validation.
Prerequisites [click to expand the list]
This course requires programming skills (python) and basic knowledge of probabilities, calculus and linear algebra. For more information see the course prerequisites and restrictions at McGill's webpage.

Textbooks

• [Bishop] Pattern Recognition and Machine Learning by Christopher Bishop (2007)
• [Goodfellow] Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville (2016)
• [Murphy] Machine Learning: A Probabilistic Perspective by Kevin Murphy (2012)
• [Murphy'22] Probabilistic Machine Learning: An Introduction, by Kevin P. Murphy (2022)
Chapters from these four books are cited as optional reference materials for the slides.
There are several other related references. [click to expand the list]
The Elements of Statistical Learning: Data Mining, Inference, and Prediction by Trevor Hastie, Robert Tibshirani and Jerome Friedman (2009)
Information Theory, Inference, and Learning Algorithms, by David MacKay (2003)
Bayesian Reasoning and Machine Learning, by David Barber (2012).
Understanding Machine Learning: From Theory to Algorithms, by Shai Shalev-Shwartz and Shai Ben-David (2014)
Foundations of Machine Learning, by Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar (2018)
Dive into Deep Learning, by Aston Zhang, Zachary Lipton, Mu Li, and Alexander J. Smola (2019)
Mathematics for Machine Learning, by Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong (2019)
A Course in Machine Learning, by Hal Daume III (2017)
Hands-on Machine Learning with Scikit-Learn and TensorFlow, by Aurelien Geron (2017)
Machine Learning, by Tom Mitchell (1997)
Introduction to Data Mining, by Pang-Ning Tan, Michael Steinbach, Anuj Karpatne, and Vipin Kumar (2020)
Machine Learning, Dynamical Systems and Control, by Steven L. Brunton and J. Nathan Kutz (2019)

Schedule

Thu., Jan. 5
Syllabus
Tue., Jan. 10
Thu., Jan. 12
Tue., Jan. 17
Note: Add/Drop deadline
Thu., Jan. 19
Tue., Jan. 24
Thu., Jan. 26
Tue., Jan. 31
Thu., Feb. 2
Tue., Feb. 7
Thu., Feb. 9
Tue., Feb. 14
Thu., Feb. 16
Tue., Feb. 21
Thu., Feb. 23
Tue., Feb. 28
Note: Study break, No class [no quiz], see McGill's Academic calendar
Thu., Mar. 2
Note: Study break, No class [no quiz], see McGill's Academic calendar
Tue., Mar. 7
Thu., Mar. 9
Tue., Mar. 14
Thu., Mar. 16
Midterm Exam
Tue., Mar. 21
Thu., Mar. 23
Tue., Mar. 28
Thu., Mar. 30
Tue., Apr. 4
Thu., Apr. 6
Tue., Apr. 11

Outline

Introduction
slides, reference: 1 [Murphy22]
Parameter Estimation
slides, notebook (Colab), reference: 4 [Murphy22], 2-2.3 [Bishop], 3-3.5 [Murphy]
Linear regression
slides, notebook (Colab), reference: 7-7.3.3 [Murphy], 3-3.1.2 [Bishop]
Logistic and softmax regression
slides, notebook (Colab), reference: 8.1-8.3.3 [Murphy], 4.1-4.1.3 + 4.3-4.3.3 [Bishop]
Gradient descent methods
slides, notebook (Colab), reference: 8.3.2 [Murphy] and this overview by S. Ruder (in pdf )
Regularization
slides, notebook (Colab), reference: 3.1.4-3.3 [Bishop]
Generalization
slides, notebook for model selection (Colab), optional notebook for curse of dimensionality (Colab)
Perceptrons & multilayer perceptrons
slides, Perceptrons Colab, MLP demo, reference: 4.1.1-4.1.3 + 4.1.7 [Bishop], 6-6.5 + parts of 7 [Goodfellow]
Gradient computation and automatic differentiation
slides, notebook (Colab), reference: 6.5 + 8.2 [Goodfellow], blog post, visualization
Convolutional neural networks
slides, reference: 9 [Goodfellow], blog post, optional reading
Neural Networks for Sequences
slides, reference: 15 [Murphy'22], optional reading
Naive Bayes
slides, notebook (Colab), reference: 3.5-3.5.4 [Murphy]
Nearest neighbours
slides, notebook (Colab), reference: chapter 1 [Murphy]
Classification and regression trees
slides, notebook (Colab), reference: 16.1-16.2.6 [Murphy], 14.4 [Bishop]
Linear support vector machines
slides, notebook (Colab), reference: 4.1.1-4.1.3 + 4.1.7 + 7.1-7.1.4 excluding kernels [Bishop]
Bagging & boosting
slides, notebook (Colab), reference: 3.2 [Bishop], demos for Bias-Variance Tradeoff, Gradient Boosting explanation, and Interactive playground
Unsupervised learning
slides, notebook (Colab), reference: 25.5 [Murphy] and 9.1 [Bishop], demos for K-Means and DB-SCAN
Dimensionality reduction
slides, notebook (Colab), reference: 12.2 [Murphy], 12.1 [Bishop], demo

Evaluation

Regular Practice Quizzes and Assignments [10%]
One quiz per week to check the key topics discussed
Late Mid-term Exam [40%]
In person, during class on March 16th
Closed book, but you can bring 5 pages of your hand written notes
Hands-on Mini-Projects [50%]
Four programming assignments to be done in groups of three*, *no exception to this given the grading load on TAs
Groups can stay the same between projects, you can also regroup when needed
All group members receive the same mark unless there are major complains on not contributing, responding, etc. from group-mates, which will be resolved in a case by case basis. If a significant difficulty/conflict arises, please send an email to the course email, put 'Group-Issue' in the title
Work submitted for evaluation as part of this course may be checked with text-matching software within myCourses
Late submission policy
All due dates are 11:59 pm in Montreal, unless specified otherwise [e.g. check the due dates for quizzes].
No make-up quizzes will be given.
For mini-projects, 2^k% percent will be deducted per k days of delay.
If you experience barriers (including a covid related issue) to learning in this course, submitting the projects, etc., please do not hesitate to discuss them with me directly, and please make sure to put "551 special" in the header to make sure I see your email [for general course correspondence, please use the course email: comp551mcgill@gmail.com]. As a point of reference, you can reach the Office for Students with Disabilities at 514-398-6009.

Academic Integrity

The ``McGill University values academic integrity. Therefore, all students must understand the meaning and consequences of cheating, plagiarism and other academic offenses under the Code of Student Conduct and Disciplinary Procedures'' (see McGill's webpage for more information). (Approved by Senate on 29 January 2003)

Online Resources

Learning plan
metacademy
Video Playlists
StatQuest
FreeCodeCamp
Essence of linear algebra and Neural Networks by 3Blue1Brown
Mathematics for ML by David Rolnick
Courses with Playlist and/or Code
Introduction to Machine Learning by Google
Machine Learning by Stanford
Deep Learning by UC Berkeley
Hinton's Lectures on Neural Networks for Machine Learning
Deep Learning & Linear Algebra courses by fastai
Learning from Data by Caltech
Deep Learning (with PyTorch) playlist and course by NYU
Deep Learning by Stanford
Deep Learning by deeplearning.ai
Introduction to Deep Learning by MIT
Information Theory, Pattern Recognition, and Neural Networks by David MacKay
Books with Code
Probabilistic Machine Learning: An Introduction by Kevin Murphy (book 1)
Dive into Deep Learning BY by Aston Zhang, Zachary Lipton, Mu Li, and Alexander J. Smola
Machine Learning Notebooks for O'Reilly book Hands-on Machine Learning with Scikit-Learn and TensorFlow
Similar Courses - Graduate Level
https://www.cs.toronto.edu/~rgrosse/courses/csc2515_2019/
https://www.cs.cornell.edu/courses/cs4780/2019fa/
Similar Courses - Undergraduate Level
hhttps://cs.mcgill.ca/~wlh/comp451/schedule.html
https://www.cs.toronto.edu/~rgrosse/courses/csc311_f20/
https://www.cs.toronto.edu/~rgrosse/courses/csc411_f18/
http://cs229.stanford.edu/syllabus-fall2020.html
https://cs230.stanford.edu/lecture/
Cheatsheets: https://stanford.edu/~shervine/teaching/
Similar Courses - Last Versions
Fall 2019
Winter 2020
Fall 2020

FAQ

• Class/waitlist is full, can I still register?
Unfortunately you will have to wait for the next semester. As an alternative, consider Fundamental of Machine Learning Course, Comp 451, and please check the list of free online courses below.
• Who to contact for department approval required for taking the course?
Please contact teresa.pian@mcgill.ca.
• Do I have the prerequisites to take the course?
This course requires strong Python programming skills and basic knowledge of probabilities, [multivariate] calculus and linear algebra. Please check this quiz to test if your background is strong enough for taking the course. It can also be used to diagnose where your background might be lacking and be used to self-study before taking the course. Most concepts covered in these questions will be used throughout the course in the slides.
• How similar is it to the last years?
Very similar, please check last year's websites to get a glimpse of the slides, expectations, etc. We will have an updated version and not exactly the same materials but very similar overall.
• Will there be lecture recordings?
Yes, and class participation is not mandatory.
• What do I learn in this course?
You will learn how the most common machine learning algorithms are designed, how they are implemented, and how to apply them in practice. This course has a heavy theory component, since it is important to understand the inner-workings of the algorithms in order to effectively utilize them in practice. Please check below for more information and note that everything below is tentative.