Overview
Course description: This course will cover the basics of modern deep neural networks. The first part of the course will introduce neural network architectures, activation functions, and operations. It will present different loss functions and describe how training is performed via backpropagation. In the second part, the course will describe specific types of neural networks, e.g. convolutional, recurrent, and graph networks, as well as their applications in computer vision and natural language processing. The course will also briefly discuss reinforcement learning and unsupervised learning, in the context of neural networks. In addition to attending lectures and completing bi-weekly homework assignments, students will also carry out and present a project.Prerequisites: Math 220 (Calculus I), Math 280 or 1180 (Linear Algebra), CS 1501 (Algorithm Implementation)
Piazza: Sign up for it here. Please use Piazza rather than email so everyone can benefit from the discussion-- you can post in such a way that only the instructor sees your name. Please try to answer each others' questions whenever possible. The best time to ask the instructor or TA questions is during office hours.
Programming language/framework: We will use Python, NumPy/SciPy, and PyTorch.
Textbooks: We will have required readings from the following textbook.
- Ian Goodfellow, Yoshua Bengio, Aaron Courville. Deep Learning. MIT Press, 2016. online version
- Michael Nielsen. Neural Networks and Deep Learning. online version
- Eugene Charniak. Introduction to Deep Learning. MIT Press, 2019. link
- Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
- Kevin P. Murphy. Machine Learning: A Probabilistic Perspective. MIT Press, 2012.
- Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer, 2009. link
- David Barber. Bayesian Reasoning and Machine Learning. Cambridge University Press, 2012. link
- Dan Jurafsky and James H. Martin. Speech and Language Processing. online version
- Richard Szeliski. Computer Vision: Algorithms and Applications. online version
Policies
Grading
Grading will be based on the following components:- Homework (5 assignments x 7% each = 35%)
- Project (5% proposal + 10% first presentation + 15% second presentation = 30%)
- First exam (15%)
- Second exam (15%)
- Participation (5%)
Homework Submission Mechanics
Homework is due at 11:59pm on the due date. You will submit your homework using CourseWeb. Navigate to the CourseWeb page for CS1699, then click on "Assignments" (on the left) and the corresponding homework ID. You should submit a single zip file with source and results, as requested. Name the file YourFirstName_YourLastName.zip. Please comment your code! Grades will be posted on CourseWeb.Note that CourseWeb will also contain an automatically computed running average column ("Overall") that you can use to gauge how you're doing in the class based on grades that are already available. Generally, "Overall" scores over 90% map to some type of A, over 80% to B, and over 70% to C.
Exams
There will be two in-class exams. The second exam is not cumulative and will not cover material from the first exam. There will be no make-up exams unless you or a close relative is seriously ill!Participation
Students are expected to regularly attend the class lectures, and should actively engage in in-class discussions. Attendance will not be taken, but keep in mind that if you don't attend, you cannot participate. You can actively participate by, for example, responding to the instructor's or others' questions, asking questions or making meaningful remarks and comments about the lecture, answering others' questions on Piazza, or bringing in relevant articles you saw in the news. The grading rubric will be as follows: 1 = you attended infrequently, 2 = you attended frequently but did not speak in class, 3 = you attended frequently and spoke a few times, 4 = you participated frequently, 5 = you participated every other week or more.Homework Late Policy
You get 3 "free" late days counted in minutes, i.e., you can submit a total of 72 hours late. For example, you can submit one homework 12 hours late, and another 60 hours late. The 72-hour "budget" is total for all assignments, NOT per assignment. Once you've used up your free late days, you will incur a penalty of 25% from the total assignment credit possible for each late day. A late day is anything from 1 minute to 24 hours.Collaboration Policy and Academic Honesty
You will do your work (exams and homework) individually. The work you turn in must be your own work. You are allowed to discuss the assignments with your classmates, but do not look at code they might have written for the assignments, or at their written answers. You are also not allowed to search for code on the internet, use solutions posted online unless you are explicitly allowed to look at those. When in doubt about what you can or cannot use, ask the instructor! Plagiarism will cause you to fail the class and receive disciplinary penalty. Please consult the University Guidelines on Academic Integrity.Note on Disabilities
If you have a disability for which you are or may be requesting an accommodation, you are encouraged to contact both your instructor and Disability Resources and Services (DRS), 140 William Pitt Union, (412) 648-7890, drsrecep@pitt.edu, (412) 228-5347 for P3 ASL users, as early as possible in the term. DRS will verify your disability and determine reasonable accommodations for this course.Note on Medical Conditions
If you have a medical condition which will prevent you from doing a certain assignment, you must inform the instructor of this before the deadline. You must then submit documentation of your condition within a week of the assignment deadline.Statement on Classroom Recording
To ensure the free and open discussion of ideas, students may not record classroom lectures, discussion and/or activities without the advance written permission of the instructor, and any such recording properly approved in advance can be used solely for the student's own private use.[top]
Project
Please enter your team information and topic here by January 30.The project is expected to be a new method for an existing problem or an application of techniques we studied in class (or another method) to a new problem that we have not discussed in class. Below are some tips:
- The project should include some amount of novelty. For example, you cannot just re-implement an existing paper or project. You are allowed to use existing code for known methods, but your project is expected to be a significant amount of work and not just a straight-up run of some package.
- Do not rely on data collection to be the novel component of your work. If you are proposing to tackle a new problem, you might need to collect data, but while this is a contribution, it will not be enough to earn a good project grade. You still have to come up with a solid method idea, i.e. your project has to have sufficient technical novelty.
- You must show that your method is in some sense better (quantitatively) than at least some relatively recent existing methods. For example, you can show that your method achieves superior accuracy in some prediction task compared to prior methods, or that it achieves comparable accuracy but is faster. This outcome is not guaranteed to come out the way you intended during the limited timespan of a course project, so whether or not your outperform the state of the art will only be a small component of your grade. Further, if you propose a sufficiently interesting method, rather than an extremely simple method, it will be less of a problem if your method does not outperform other existing approaches to the problem.
- You are encouraged to use any external expertise you might have (e.g. biology, physics, etc.) so that your project makes the best use of areas you know well, and is as interesting as possible.
- Students are required to work in groups of three or four for their final project.
- You are strongly encouraged to use a topic from the list of suggested topics (see CourseWeb). The instructor has suggested these based on her familiarity with the literature, and these contain sufficient amount of novelty, with a risk level appropriate for this course. At the very list you should read through this list carefully to get a sense of what an appropriate topic might be.
- You will submit a project proposal that describes what you propose to do; why what you propose to do is interesting, important, and challenging; what data and resources you are going to use; what your high-level idea of the method is; and how you plan to evaluate your method, including against what baselines. A grading rubric is provided below.
- There will be two presentations per team. The first will focus on motivation, related work, and your proposed approach. The goal is to have carefully designed your method conceptually, and to have made notable progress in terms of implementing this proposed method. During the first presentations, you will receive feedback from your classmates. Do comment on challenges you are facing, if you think this will be useful. You should present a literature review during your first presentation, so your classmates know the space in which you are working. A good source for learning about what work has been done in your domain of interest are search engines, Google Scholar, and arxiv.org.
- The second presentation will focus on your experimental validation and findings. You only need to briefly refresh your classmates' memory about the motivation for your work and the proposed method.
- Your slides are due on CourseWeb on the indicated due date, at 11:59pm.
- You are allowed to change your slides for the second presentation between presenting and submitting them, but make sure to change the slide background for any new slides to yellow so the instructor can easily tell what content is new compared to the version that was presented.
- Your grades will be based on three rubrics, available below. For each presentation, your grade will be an average of (1) your classmates' scores, and (2) the instructor's scores. You will be able to see your score for each item in the rubric.
- If you are interested in receiving feedback from the instructor on any of these components (proposal or presentations), please come to office hours after your grades are posted.
- There will be no written report, but you will practice your ability to describe your work in clear and memorable fashion through your presentation.
- Combining your final project for this class and another class is generally permitted, but the project proposal and final report should clearly outline what part of the work was done to get credit in this class, and the instructor should approve the proposed breakdown of work between this and another class.
- The first presentation should take about 10 minutes, with 2-3 minutes for questions and discussion. The second presentation should take 12 minutes, with 1 minute for questions. Make sure to test your laptops before class as we'll have an extremely tight schedule! The time budgeted for laptop switching is on the order of seconds.
- The order in which you present will be decided by a random draw. The order of teams for the first presentation will be inversely related to the order for the second presentation. For example, if you present on the second day for the first presentation, you'll present on the first day for the second.
- What do you propose to do?
- What have others attempted in this space, i.e. what is the relevant literature?
- Why is what you are proposing interesting?
- Why is it challenging?
- Why is it important?
- What data do you plan to use?
- What is your high-level idea of how your method will work?
- In what ways is this method novel?
- How will you evaluate the method, i.e. what metrics are you going to use, and what baselines are you going to compare to?
- Give a (1) conservative and (2) an ambitious schedule of milestones for your project.
- How well did the authors (presenters) explain what problem they are trying to solve?
- How well did they explain why this problem is important?
- How well did they explain why the problem is challenging?
- How thorough was the literature review?
- How clearly was prior work described?
- How well did the authors explain how their proposed work is different than prior work?
- How clearly did the authors describe their proposed approach?
- How novel is the proposed approach?
- How challenging and ambitious is the proposed approach? (1-10)
- To what extent did the authors develop the method as described in the first presentation? (1-10)
- How well did the authors describe their experimental validation?
- How informative were the figures used?
- Were all/most relevant baselines and competitor methods included in the experimental validation?
- Were sufficient experimental settings (e.g. datasets) tested?
- To what extent is the performance of the proposed method satisfactory?
- How informative were the conclusions the authors drew about their method’s performance relative to other methods?
- How sensible was the discussion of limitations?
- How interesting was the discussion of future work?
- Suggested list on CourseWeb.
- Look at the datasets and tasks below.
- Look at the topics in the programs of some of the recent computer vision conferences: CVPR 2019 (with papers downloadable here) and ICCV 2019 (with papers here).
- Check out project resources and topics in a related class here.
- Read some paper abstracts on this page.
Schedule
| Date | Topic | Readings | Lecture slides | Due |
| 1/7 | Introduction, administrivia, review, Numpy/SciPy | DL Ch 1-5 | pptx pdf | |
| 1/9 | ||||
| 1/14 | ||||
| 1/16 | Neural network basics (tasks, operations, training) | DL Ch 4.2, 4.3, 6, Bishop Ch 5 |
pptx pdf examples | |
| 1/21 | ||||
| 1/23 | HW1 | |||
| 1/28 | ||||
| 1/30 | PyTorch tutorial (guest lecture: Mingda Zhang) | tutorial demo | proposals | |
| 2/4 | Training part 2 (alternative notation, optimization, properties, convergence) | DL Ch 7, 8, 11 | pptx pdf | |
| 2/6 | ||||
| 2/11 | HW2 | |||
| 2/13 | ||||
| 2/18 | First exam | |||
| 2/20 | Convolutional neural networks (architectures, visualization, applications) | DL Ch 9 | pptx pdf | |
| 2/25 | ||||
| 2/27 | HW3 | |||
| 3/3 | Project first presentations | |||
| 3/5 | slides | |||
| 3/24 | Recurrent neural networks (architectures, training, applications) | DL Ch 10, Mikolov | pptx pdf | |
| 3/26 | ||||
| 3/31 | HW4 | 4/2 | ||
| 4/7 | Advanced/recent topics: Graph networks, self-supervised and reinforcement learning, generation, bias and ethics; Project second presentations | Kipf, Doersch, Karpathy, Mnih, Goodfellow, Bolukbasi | pptx pdf | |
| 4/9 | ||||
| 4/14 | ||||
| 4/16 | HW5 | |||
| 4/23 | Second exam (2-3:50pm) | |||
[top]
Resources
This course was inspired by the following courses:- Convolutional Neural Networks for Visual Recognition by Fei-Fei Li, Justin Johnson, and Serena Young, some content by Andrej Karpathy, Stanford University, Spring 2019
- Natural Language Processing with Deep Learning by Chris Manning, Abigail See, based on an earlier course by Richard Socher, Stanford University, Winter 2019
- Introduction to Deep Learning by Bhiksha Raj, Carnegie Mellon University, Fall 2019
- Introduction to Machine Learning by Rebecca Hwa, University of Pittsburgh, Fall 2015
- Machine Learning by Milos Hauskrecht, University of Pittsburgh, Spring 2015
- Introduction to Machine Learning by Dhruv Batra, Virginia Tech, Spring 2015
- Machine Learning by Tommi Jaakkola, MIT
- Machine Learning by Subhransu Maji, UMass Amhrest, Spring 2015
- Machine Learning by Erik Sudderth, Brown University, Fall 2015
- Computer Vision by Kristen Grauman, UT Austin, Spring 2011
- Computer Vision by Derek Hoiem, UIUC, Spring 2015
- Natural Language Processing by Ray Mooney, UT Austin
- Microsoft COCO (Common Objects in Context) (object recognition, segmentation, image description)
- ImageNet (object recognition)
- SUN Database (scenes)
- Caltech-UCSD Birds 200 (fine-grained object recognition)
- MSRC Annotations (active learning)
- Animals with Attributes (attribute-based recognition)
- a-Pascal + a-Yahoo (attribute-based recognition)
- Shoes (attribute-based search)
- INRIA Movie Actions (action recognition)
- ADL (ego-centric action recognition)
- Action Quality (evaluating action quality)
- CarDb Historical Cars (style classification of cars)
- Recognizing Image Style (photographic style classification)
- Judd gaze (visual saliency prediction)
- Visual Persuasion (predicting subtle messages in images)
- Advertisements: Images and Videos (understanding what the ad prompts of the viewer and why)
- VQA (visual question-answering)
- Recognition datasets list compiled by Kristen Grauman
- Human activity datasets list compiled by Chao-Yeh Chen
- TensorFlow (deep learning framework by Google)
- Caffe (deep learning framework by Yangqing Jia et al.)
- PyTorch (another popular deep learning framework)
- Keras (deep learning library)
- LIBSVM (by Chih-Chung Chang and Chih-Jen Lin)
- SVM Light (by Thorsten Joachims)
- VLFeat (feature extraction, tutorials and more, by Andrea Vedaldi)