Overview
Course description: This course will cover the basics of modern deep neural networks. The first part of the course will introduce neural network architectures, activation functions, and operations. It will present different loss functions and describe how training is performed via backpropagation. In the second part, the course will describe specific types of neural networks, e.g. convolutional, recurrent, and transformers, as well as their applications in computer vision and natural language processing. The course will also briefly discuss foundation models, self-supervised and deep reinforcement learning, and generation approaches.Prerequisites: Math 220 (Calculus I), Math 280 or 1180 (Linear Algebra), CS 1501 (Algorithm Implementation).
Programming language/framework: We will use Python, NumPy/SciPy, and PyTorch.
Textbooks: We will have readings from the following textbooks. Readings will be posted about one week in advance of the lecture.
- Ian Goodfellow, Yoshua Bengio, Aaron Courville. Deep Learning. online version
- Richard Szeliski. Computer Vision: Algorithms and Applications. online version (second edition)
- Aston Zhang, Zack C. Lipton, Mu Li, and Alex J. Smola. Dive into Deep Learning. online version
- Michael Nielsen. Neural Networks and Deep Learning. 2019. online version
- Eugene Charniak. Introduction to Deep Learning. MIT Press, 2019. link
- Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006. free pdf
- Kevin P. Murphy. Machine Learning: A Probabilistic Perspective. MIT Press, 2012.
- Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer, 2009. link
- David Barber. Bayesian Reasoning and Machine Learning. Cambridge University Press, 2012. link
- Dan Jurafsky and James H. Martin. Speech and Language Processing. online version
Policies
Grading
Grading for CS 1678 will be based on the following components:- Homework (4 assignments x 13.75% each = 55%)
- First exam (20%)
- Second exam (20%)
- Participation (5%)
- Homework (4 assignments x 11.25% each = 45%)
- First exam (15%)
- Second exam (15%)
- Participation (5%)
- Project (20%)
- Proposal (5%)
- Status report (5%)
- Presentation (5%)
- Final report (5%)
Assignment Submission Mechanics
Homework, project reports and presentations slides are due at 11:59pm on the due date. You will submit your homework using Canvas, under "Assignments" and the corresponding homework ID. You should submit a single zip file with source/results/report/slides, as requested. Name the file YourFirstName_YourLastName.zip. Please comment your code! Grades will be posted on Canvas.Note that Canvas will also contain an automatically computed running average column that you can use to gauge how you're doing in the class based on grades that are already available. Generally, Overall scores over 90% map to some type of A, over 80% to B, and over 70% to C.
Exams and Quizzes
There will be two in-class exams. The second exam is not cumulative and will not cover material from the first exam. There will be no make-up exams unless you or a very close friend/relative is seriously ill!Participation
Students are expected to regularly attend the class lectures, and should actively engage in in-class discussions. Attendance will not be taken, but keep in mind that if you don't attend, you cannot participate. You can actively participate by, for example, responding to the instructor's or others' questions, asking questions or making meaningful remarks and comments about the lecture, or bringing in relevant articles you saw in the news. The grading rubric will be as follows: 1 = you attended infrequently, 2 = you attended frequently but did not speak in class, 3 = you attended frequently and spoke a few times, 4 = you participated frequently, 5 = you participated every other week or more.Homework Late Policy
For homework assignments only (not for project components), you get 5 "free" late days counted in minutes, i.e., you can submit a total of 120 hours late. For example, you can submit one homework 24 hours late, and another 96 hours late. The 120-hour "budget" is total for all assignments, NOT per assignment. Once you've used up your free late days, you will incur a penalty of 25% from the total assignment credit possible for each late day. A late day is anything from 1 minute to 24 hours. Note that because of the generous late days policy, the instructor will be reluctant to give extra days to individual students due to illness that is not severe/prolonged, due to travel, etc. However, if you are facing an especially difficult situation or exceptional circumstances, please speak with the instructor (ahead of the assignment deadline, if possible) and provide documentation.Collaboration Policy and Academic Honesty
You will do your work (homework and quizzes) individually. The work you turn in must be your own work. You are allowed to discuss the assignments with your classmates, but do not look at code they might have written for the assignments, or at their written answers. You are also not allowed to search for code on the internet, use solutions posted online unless you are explicitly allowed to look at those. When in doubt about what you can or cannot use, ask the instructor! Plagiarism will cause you to fail the class and receive disciplinary penalty. Please consult the University Guidelines on Academic Integrity. All project components involve group work.Note on Disabilities
If you have a disability for which you are or may be requesting an accommodation, you are encouraged to contact both your instructor and Disability Resources and Services (DRS), 140 William Pitt Union, (412) 648-7890, drsrecep@pitt.edu, (412) 228-5347 for P3 ASL users, as early as possible in the term. DRS will verify your disability and determine reasonable accommodations for this course.Note on Medical Conditions
If you have a medical condition which will prevent you from doing a certain assignment, you must inform the instructor of this before the deadline. You must then submit documentation of your condition within a week of the assignment deadline.Statement on Classroom Recording
To ensure the free and open discussion of ideas, students may not record classroom lectures, discussion and/or activities without the advance written permission of the instructor, and any such recording properly approved in advance can be used solely for the student's own private use.[top]
Project (CS 2078 only)
Students will complete a project which studies in more depth one of the topics we cover in class. Students should work in groups of two or three. These projects should focus on one of the following:- a novel approach which addresses one of the problems covered in class, properly evaluated
- a definition of a new problem, along with detailed argumentation of why this problem is important and challenging, an approach to solve this problem, and an evaluation of this approach
- an extensive analysis and experimental evaluation of one or more of the approaches covered in class
The mid-semester project status report will describe the students' progress on the project, and any problems encountered along the way. The status report should use a known conference format (e.g. ICLR, AAAI, NeurIPS, CVPR, ACL), but can be more informal than a conference paper. The progress report should include the following sections: Introduction, Related Work, Approach, and Results. In Results, include your experimental setup (this can change later). If you have results but they do not yet look great, include them anyway. Comment on any challenges encountered as well.
The project presentation, scheduled for the last day of class, will describe the students' approach and their experimental findings in a clear and engaging fashion. This will be a chance to get feedback from the class before final submission of your report. Presentations will be about 10-15 minutes long. Please submit a copy of your slides to Canvas on the same day as your presentation.
The project final report should be formatted and should read like a conference paper, with clear problem definition and argumentation of why this problem is important, overview of related work, detailed explanation of the approach, and well-motivated experimental evaluation. Еach student should document what part of the project they did, and how duties and tasks were divided. Below are some tips:
- The project should include some amount of novelty. For example, you cannot just re-implement an existing paper or project. You are allowed to use existing code for known methods, but your project is expected to be a significant amount of work and not just a straight-up run of some package.
- You must show that your method is in some sense better (quantitatively) than at least some relatively recent existing methods. For example, you can show that your method achieves superior accuracy in some prediction task compared to prior methods, or that it achieves comparable accuracy but is faster. This outcome is not guaranteed to come out the way you intended during the limited timespan of a course project, so whether or not your outperform the state of the art will only be a small component of your grade. Further, if you propose a sufficiently interesting method, rather than an extremely simple method, it will be less of a problem if your method does not outperform other existing approaches to the problem.
- You are encouraged to use any external expertise you might have (e.g. biology, physics, etc.) so that your project makes the best use of areas you know well, and is as interesting as possible.
- To get some ideas for projects, browse websites/proceedings for some of the latest conferences, e.g. ICLR, AAAI, NeurIPS, CVPR, ACL, etc.
- What do you propose to do?
- What have others attempted in this space, i.e. what is the relevant literature?
- Why is what you are proposing interesting?
- Why is it challenging?
- Why is it important?
- What data do you plan to use?
- What is your high-level idea of how your method will work?
- In what ways is this method novel?
- How will you evaluate the method, i.e. what metrics are you going to use, and what baselines are you going to compare to?
- Give a (1) conservative and (2) an ambitious schedule of milestones for your project.
- What is the prior work in this space? Describe in detail (2-3 pages, in your own words).
- How is your work novel, in the context of this prior work?
- What is your proposed approach? Describe in detail (3-4 pages).
- Why should your approach work well for this task?
- In what ways is your proposed approach ambitious?
- How do you plan to evaluate your method, against what baselines? (It's possible these will have changed after the proposal stage.)
- What progress have you made since the first report? Describe in detail (1-2 pages).
- What challenges have you encountered along the way?
- What are your next steps? Describe in detail (1-2 pages).
- What metrics are you going to use to evaluate your method? What datasets are you going to use? What are your experimental results so far, if any?
- How well did the authors (presenters) explain what problem they are trying to solve?
- How well did they explain why this problem is important and/or challenging?
- How clearly was prior work described? How well did the authors explain how their proposed work is different than prior work?
- How clearly did the authors describe their proposed approach?
- How novel and ambitious is the proposed approach?
- How well did the authors describe their experimental validation? How informative were the figures used?
- Were all/most relevant experimental settings (e.g. datasets, tasks) and baselines (competitor methods) included in the experimental validation?
- To what extent is the performance of the proposed method satisfactory?
- How informative were the conclusions the authors drew about their method’s performance relative to other methods? How sensible was the discussion of limitations? How interesting was the discussion of future work?
Schedule
| Date | Topic | Readings | Lecture slides | Due |
| 1/8 | Introduction, administrivia, ML overview/review | Goodfellow Ch.
1-5, Bishop PRML Sec. 1.1, Bishop PRML Sec. 7.1, Zhang Ch. 1, 2, 22, Sec. 23.1, 23.4 |
pptx pdf | |
| 1/10 | ||||
| 1/17 | ||||
| 1/22 | Neural network basics (tasks, operations, training) | Goodfellow Sec. 4.2, 4.3, Ch. 6, Bishop Ch. 5, Zhang Sec. 4.1-4.3, Ch. 5 | pptx pdf exercise answers | |
| 1/24 | ||||
| 1/29 | ||||
| 1/31 | PyTorch tutorial (TA) [link on Canvas] | Zhang Ch. 6 | ||
| 2/5 | Training part 2 (alternative notation, optimization, properties, convergence) |
Goodfellow Ch. 7, 8, 11, Zhang Ch. 12 | pptx pdf | HW1 |
| 2/7 | ||||
| 2/12 | proposal | |||
| 2/14 | Convolutional neural networks (architectures, visualization, applications) |
Goodfellow Ch. 9, Szeliski Sec. 5.4.3-4, 6.3.3, 6.4, Zhang Ch. 7, 8, 14 | pptx pdf | |
| 2/19 | ||||
| 2/21 | ||||
| 2/26 | Recurrent neural networks (architectures, training, attention) |
Goodfellow Ch. 10, Szeliski Sec. 5.5.2, Zhang Ch. 9, 10, 15, RNN blog 1, RNN blog 2, RNN vis | pptx pdf | HW2 |
| 2/28 | ||||
| 3/4 | ||||
| 3/6 | First exam | |||
| 3/18 | Transformers | Szeliski Sec. 5.5.3, Zhang Ch. 11, Mikolov NIPS 2013, Vaswani NIPS 2017, Devlin NAACL 2019, Illustrated Transformer and BERT, Dosovitskiy ICLR 2021, Lu NeurIPS 2019, Kipf GCNs | pptx pdf | |
| 3/20 | HW3; status report (3/22) |
|||
| 3/25 | Self-supervised learning | Szeliski Sec. 5.5.4, 5.4.7, Doersch ICCV 2015, Chen ICML 2020 | pptx pdf | |
| 3/27 | ||||
| 4/1 | Foundation models and prompting | Hu ICLR 2022, Radford ICML 2021, Zhou IJCV 2022, Menon ICML 2023 | pptx pdf | |
| 4/3 | ||||
| 4/8 | Reinforcement learning | Zhang Ch. 17, Karpathy RL blog, Mnih NIPS-W 2013 | pptx pdf | |
| 4/10 | Image/video generation methods | Zhang Ch. 20, Goodfellow NIPS 2014, Rombach CVPR 2022 | pptx pdf | HW4 |
| 4/15 | Bias, domain shifts, attacks | Peng ICCV 2019, Buettner CVPR 2024, Burns ECCV 2018 | pptx pdf | |
| 4/17 | Project presentations | presentation; final report (4/19) | ||
| 4/24 | Second exam (2-3:15pm, 358 Cathedral of Learning) | participation notes | ||
[top]
Resources
This course was inspired by the following courses:- Convolutional Neural Networks for Visual Recognition by Fei-Fei Li, Justin Johnson, and Serena Young, some content by Andrej Karpathy, Stanford University, Spring 2023
- Natural Language Processing with Deep Learning by Chris Manning, Abigail See, based on an earlier course by Richard Socher, Stanford University, Winter 2019
- Introduction to Deep Learning by Bhiksha Raj, Carnegie Mellon University, Fall 2019
- Deep Learning for Computer Vision by Yong Jae Lee, University of Wisconsin, Spring 2023
- Introduction to Machine Learning by Rebecca Hwa, University of Pittsburgh, Fall 2015
- Machine Learning by Milos Hauskrecht, University of Pittsburgh, Spring 2015
- Introduction to Machine Learning by Dhruv Batra, Virginia Tech, Spring 2015
- Machine Learning by Tommi Jaakkola, MIT
- Machine Learning by Subhransu Maji, UMass Amhrest, Spring 2015
- Machine Learning by Erik Sudderth, Brown University, Fall 2015
- Computer Vision by Kristen Grauman, UT Austin, Spring 2011
- Computer Vision by Derek Hoiem, UIUC, Spring 2015
- Natural Language Processing by Ray Mooney, UT Austin