COMP 551: Applied Machine Learning
Winter 2021, delivered though McGill's MyCoursesInstructor: Reihaneh Rabbany
please make sure to use this email to receive a timely response
Overview
 This course covers a selected set of topics in machine learning and data mining, with an emphasis on good methods and practices for deployment of real systems. The majority of sections are related to commonly used supervised learning techniques, and to a lesser degree unsupervised methods. This includes fundamentals of algorithms on linear and logistic regression, decision trees, support vector machines, clustering, neural networks, as well as key techniques for feature selection and dimensionality reduction, error estimation and empirical validation.
 Prerequisites [click to expand the list]
 This course requires programming skills (python) and basic knowledge of probabilities, calculus and linear algebra. For more information see the course prerequisites and restrictions at McGill's webpage.
Textbooks
 [Bishop] Pattern Recognition and Machine Learning by Christopher Bishop (2007)
 [Goodfellow] Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville (2016)
 [Murphy] Machine Learning: A Probabilistic Perspective by Kevin Murphy (2012)
 Chapters from these three books are cited as optional reference materials for the slides.
There are several other related references. [click to expand the list] 
 The Elements of Statistical Learning: Data Mining, Inference, and Prediction by Trevor Hastie, Robert Tibshirani and Jerome Friedman (2009)
 Information Theory, Inference, and Learning Algorithms, by David MacKay (2003)
 Bayesian Reasoning and Machine Learning, by David Barber (2012).
 Understanding Machine Learning: From Theory to Algorithms, by Shai ShalevShwartz and Shai BenDavid (2014)
 Foundations of Machine Learning, by Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar (2018)
 Dive into Deep Learning, by Aston Zhang, Zachary Lipton, Mu Li, and Alexander J. Smola (2019)
 Mathematics for Machine Learning, by Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong (2019)
 A Course in Machine Learning, by Hal Daume III (2017)
 Handson Machine Learning with ScikitLearn and TensorFlow, by Aurelien Geron (2017)
 Machine Learning, by Tom Mitchell (1997)
 Introduction to Data Mining, by PangNing Tan, Michael Steinbach, Anuj Karpatne, and Vipin Kumar (2020)
 Machine Learning, Dynamical Systems and Control, by Steven L. Brunton and J. Nathan Kutz (2019)
 Probabilistic Machine Learning: An Introduction, by Kevin P. Murphy (2021)
Schedule
 Mon., Jan. 11: Syllabus
 Wed., Jan. 13: Introduction
 Mon., Jan. 18: Nearest Neighbours
 Wed., Jan. 20: Decision Trees

covered slides: 112.3
Note: last lecture before the Add/Drop deadline  Mon., Jan. 25: Decision Trees continued & Core ML concepts
 Core ML concepts slides: 13.3
 Wed., Jan. 27: Core ML concepts continued
 Core ML concepts slides up to the curse of dimensionality: slide 5.1
 Mon., Feb. 1: Core ML concepts continued & Maximum likelihood and Bayesian Reasoning
 Bayesian Reasoning up to posterior predictive, slide 5.1
 Wed., Feb. 3: Bayesian Reasoning continued & Naive Bayes
 Bayesian Reasoning from slide 5.1 & Naive Bayes up to until slide 8.2
 Mon., Feb. 8: Naive Bayes continued
 Wed., Feb. 10: Linear regression
 Mon., Feb. 15: Logistic and softmax regression
 Wed., Feb. 17: Gradient descent methods
 until slide 6.5: MiniBatch SGD
 Mon., Feb. 22: Gradient descent methods continued & Regularization
 Wed., Feb. 24: Regularization continued with Biasvariance decomposition
Mon., Mar. 1 Note: Study break, No class [no quiz], see McGill's Academic calendar
Wed., Mar. 3 Note: Study break, No class [no quiz], see McGill's Academic calendar
 Mon., Mar. 8: Multilayer Perceptrons
 Wed., Mar. 10: Multilayer Perceptrons & Gradient computation and automatic differentiation
 until slide 4.2
 Mon., Mar. 15: Gradient computation and automatic differentiation
 Wed., Mar. 17: Gradient computation and automatic differentiation Cont.
 Mon., Mar. 22: Convolutional neural networks
 Wed., Mar. 24: Support vector machines
 Mon., Mar. 29: Bagging & Boosting
 Wed., Mar. 31: Unsupervised learning
Mon., Apr. 5 Note: Easter Monday, No class [no quiz], see McGill's Academic calendar
 Wed., Apr. 7: Dimensionality reduction
 Mon., Apr. 12: Learning with graphs
 Wed., Apr. 14: Ethics in ML, A guest lecture by Abhishek Gupta
 Abhishek Gupta is the founder of Montreal AI Ethics Institute
 Thr., Apr. 15: RL in practice, A guest lecture by by Marc Bellemare
 Marc Bellemare leads the reinforcement learning efforts of the Google Research team in Montreal
Note: Makeup Day, see McGill's Academic calendar
Tutorials
 Fri., Jan. 15: Probability and Linear Algebra
 By Safa Alver, slides for linear algebra and probability, see Mycourses for the recording
 Tue., Jan. 19: Python / NumPy
 By Manoj
 Mid Feb.: ScikitLearn
 By Nishant
 Early Mar.: Pytorch
 by Sumana
Outline
 *** quick note ***
 Please note that this list of topics is tentative, we might add/drop some topics or change the order as the course progresses. The slides will be posted here right after the corresponding lecture or sooner when possible. All the course materials (slides, codes, etc.) are adapted based on the last year's offering of the same course by Prof. Siamak Ravanbakhsh. You can look ahead into the materials, by checking that outline here.
 Introduction
 slides
 Nearest Neighbours
 slides, notebook (Colab), reference: chapter 1 [Murphy]
 Classification and regression trees
 slides, notebook (Colab), reference: 16.116.2.6 [Murphy], 14.4 [Bishop]
 Core concepts
 slides, notebook for model selection (Colab), notebook for curse of dimensionality (Colab)
 Maximum likelihood and Bayesian Reasoning
 slides, notebook (Colab), reference: 22.3 [Bishop], 33.5 [Murphy]
 Naive Bayes
 slides, notebook (Colab), reference: 3.53.5.4 [Murphy]
 Linear regression
 slides, notebook (Colab), reference: 77.3.3 [Murphy], 33.1.2 [Bishop]
 Logistic and softmax regression
 slides, notebook (Colab), reference: 8.18.3.3 [Murphy], 4.14.1.3 + 4.34.3.3 [Bishop]
 Gradient descent methods
 slides, notebook (Colab), reference: 8.3.2 [Murphy] and this overview by S. Ruder (in pdf )
 Regularization
 slides, notebook (Colab), reference: 3.1.43.3 [Bishop]
 Perceptrons & Multilayer Perceptrons
 slides, Perceptrons Colab, MLP demo, reference: 4.1.14.1.3 + 4.1.7 [Bishop], 66.5 + parts of 7 [Goodfellow]
 Gradient computation and automatic differentiation
 slides, notebook (Colab), reference: 6.5 + 8.2 [Goodfellow], blog post, visualization
 Convolutional neural networks
 slides, reference: 9 [Goodfellow], blog post, optional reading
 Linear support vector machines
 slides, notebook (Colab), reference: 4.1.14.1.3 + 4.1.7 + 7.17.1.4 excluding kernels [Bishop]
 Bagging & Boosting
 slides, reference: 3.2 [Bishop], demos for BiasVariance Tradeoff, Gradient Boosting explanation, and Interactive playground
 Unsupervised learning
 slides, notebook (Colab), reference: 25.5 [Murphy] and 9.1 [Bishop], demos for KMeans and DBSCAN
 Dimensionality reduction
 slides, notebook (Colab), reference: 12.2 [Murphy], 12.1 [Bishop], demo
 Learning with graphs
 slides,
 Frontiers
Evaluation
 Regular Practice Quizzes [20%]

 One per lecture to check the key concepts discussed in the last lecture
 Timed to be done in 1 hour after starting the quiz
 Available until the start of the next lecture
 Four lectures don't have practice quizzes and instead we have the checkpoint quizzes
 From the two subsequent quizzes, the best one will be considered for your final grade
 Starts from January 13th
 The first 2 practice quizzes check the prerequisites for the course and have a different setting than the regular practice quizzes. In particular, unlimited attempts are allowed and due date is set to after add/drop instead of the next lecture
 Knowledge Checkpoint Quizzes [30%]

 One per month from all the topics discussed up to the date of the quiz
 Timed to be done in 3 hours, within a 3 days availability period
 Mark the due dates: Jan 2123rd [3%], Feb 1214th [7%], March 1214th [10%], April 1517th [10%]; timing (12:01 am to 11:59 pm)
 Handson MiniProjects [50%]

 Four programming assignments to be done in groups of three*, *no exception to this given the grading load on TAs
 Groups can stay the same between projects, you can also regroup when needed
 All group members receive the same mark unless there are major complains on not contributing, responding, etc. from groupmates, which will be resolved in a case by case basis. If a significant difficulty/conflict arises, please send an email to the course email, cc the groupTA and put 'GroupTA' in the title
 Mark the due dates: Feb 5th [10%], Feb 28th [15%], March 25th [15%], April 25th [10%]
 Work submitted for evaluation as part of this course may be checked with textmatching software within myCourses
 Late submission policy
 All due dates are 11:59 pm in Montreal, unless specified otherwise [e.g. check the due dates for quizzes].
No makeup quizzes will be given.
For miniprojects, 2^k% percent will be deducted per k days of delay.
If you experience barriers (including a covid related issue) to learning in this course, submitting the projects, etc., please do not hesitate to discuss them with me directly, and please make sure to put "551 special" in the header to make sure I see your email [for general course correspondence, please use the course email: comp551mcgill@gmail.com]. As a point of reference, you can reach the Office for Students with Disabilities at 5143986009.
Academic Integrity
 The ``McGill University values academic integrity. Therefore, all students must understand the meaning and consequences of cheating, plagiarism and other academic offenses under the Code of Student Conduct and Disciplinary Procedures'' (see McGill's webpage for more information). (Approved by Senate on 29 January 2003)
Online Resources
 Learning plan
 metacademy
 Video Playlists

 StatQuest
 FreeCodeCamp
 Essence of linear algebra and Neural Networks by 3Blue1Brown
 Mathematics for ML by David Rolnick
 Courses with Playlist and/or Code

 Introduction to Machine Learning by Google
 Machine Learning by Stanford
 Deep Learning by UC Berkeley
 Hinton's Lectures on Neural Networks for Machine Learning
 Deep Learning & Linear Algebra courses by fastai
 Learning from Data by Caltech
 Deep Learning (with PyTorch) playlist and course by NYU
 Deep Learning by Stanford
 Deep Learning by deeplearning.ai
 Introduction to Deep Learning by MIT
 Information Theory, Pattern Recognition, and Neural Networks by David MacKay
 Books with Code

 Probabilistic Machine Learning: An Introduction by Kevin Murphy (book 1)
 Dive into Deep Learning BY by Aston Zhang, Zachary Lipton, Mu Li, and Alexander J. Smola
 Machine Learning Notebooks for O'Reilly book Handson Machine Learning with ScikitLearn and TensorFlow
 Similar Courses  Graduate Level

 https://www.cs.toronto.edu/~rgrosse/courses/csc2515_2019/
 https://www.cs.cornell.edu/courses/cs4780/2019fa/
 Similar Courses  Undergraduate Level

 hhttps://cs.mcgill.ca/~wlh/comp451/schedule.html
 https://www.cs.toronto.edu/~rgrosse/courses/csc311_f20/
 https://www.cs.toronto.edu/~rgrosse/courses/csc411_f18/
 http://cs229.stanford.edu/syllabusfall2020.html
 https://cs230.stanford.edu/lecture/
 Cheatsheets: https://stanford.edu/~shervine/teaching/
 Similar Courses  Last Versions

 Fall 2019
 Winter 2020
 Fall 2020