CS1699: Homework 3

CS 1699: Homework 3

Due: Feb. 27, 2020, 11:59pm

This assignment is worth 50 points.

Part I: Building a Custom Data Loader (20 points)

In our PyTorch tutorial, we show how to use the torchvision library to load common dataset such as MNIST. However, in real world applications, it's more common for you to deal with custom data. In this exercise, you are provided with a bunch of images and you need to write a data loader to feed these images to your model in PyTorch.

Note: You are not allowed to use torchvision in this exercise. You may find the following tutorial helpful: Writing Custom Datasets, Dataloaders and Transforms

Instructions:

Please download the data from here. There are two directories within the given zip file, one for training samples and one for test. In each directory, there are 10 subdirectories each containing the images that belong to that category. The structure looks like:
cifar10_train/airplane/airplane_00001.png
cifar10_train/airplane/airplane_00002.png
... ...
cifar10_train/truck/truck_04999.png

cifar10_test/airplane/airplane_00001.png
cifar10_test/airplane/airplane_00002.png
... ...
cifar10_test/truck/truck_00999.png
In your hw3.py you need to implement a class CifarDataset, which represents the given dataset. Your custom dataset should inherit torch.utils.data.Dataset. This class can be used for both training set and test set, which depends on the argument `root_dir` that passes to the constructor.
Once you set up your custom dataset, you can feed it to a torch.utils.data.DataLoader so it helps to iterate through your dataset.

Here is a sample code snippet that you could use as a starter:

class CifarDataset(torch.utils.data.Dataset):
  def __init__(self, root_dir):
    """Initializes a dataset containing images and labels."""
    super().__init__()
    raise NotImplementedError

  def __len__(self):
    """Returns the size of the dataset."""
    raise NotImplementedError

  def __getitem__(self, index):
    """Returns the index-th data item of the dataset."""
    raise NotImplementedError

train_dataset = CifarDataset(TRAIN_DIRECTORY_PATH)
train_dataloader = torch.utils.data.DataLoader(train_dataset,
                                               batch_size=BATCH_SIZE,
                                               shuffle=True)

Part II: Training a neural network in PyTorch (20 points)

In this exercise you need to implement a 3-layer MLP model (one input layer, one hidden layer with tanh activation and one output layer) in PyTorch (named MultiLayerPerceptronModel, which will be used to classify the images from the dataset in Part I.
You can use the built-in modules in PyTorch to build your model, such as Linear, Dropout, Tanh, etc..
You also need to write the training function (training), and should explore the following hyperparameter settings:

Batch size: Number of examples per training iteration.
Hidden size: Try using different number of hidden nodes in your model and compare the performances.
Dropout: Dropout is an effective strategy to defend against overfitting. Adding a dropout layer after the hidden layer, and try using different dropout rate to compare the performances.
Optimizer: Try using different optimizers such as SGD, Adam, RMSProp.
Regularization (weight decay): L2 regularization can be specified by setting the weight_decay parameter in optimizer. Try using different regularization factor and check the performance.
Learning rate, Learning rate scheduler: Learning rate is key hyperparameter in model training, and you can gradually decreasing the learning rate to further improve your model. Try using different learning rate and different learning rate scheduler to compare the performance.

To get full credit, you should explore at least 4 different type of hyperparameters (from listed above), and choose at least 3 different values for each hyperparameters. For simplicity, you could analyze one hyperparameter at a time (i.e. fixing all others to some reasonable value), rather than perfoming grid search.
If you use TensorBoard to monitor your training, you can directly attach the screenshots of the training curves (accuracy) in your report.
To evaluate the performance of trained model, you also need to write a function (evaluation) which loads the trained model and evaluate its performance on train/test set. In your report, please clearly state what hyperparameters you explored, and what accuracy the model achieved on train/test set.

Part III: Transfer Learning (10 points)

In this exercise you can take an ImageNet pre-trained MobileNetV2 model and finetune it on the dataset in Part I.
Specifically, you can load a MobileNetV2 model with pretrained weights using torchvision library, and replace the final classification layer using a new randomly initialized fully-connected layer and finetune on the Cifar10 Dataset. You should try the two flavors of transfer learning, i.e. (1) freezing all the MobileNetV2 layers (feature extraction) and only train the final classification layer; (2) finetuning all MobileNetV2 layers together with the final classification layer. Report the performance of the two models on train/test set.
Note: By "finetuning" we mean the model is further trained with small learning rate, thus the weights do not change significantly but it will hopefully lead to improved performance.
Note: You may find the following tutorial helpful: Finetuning torchvision Models

Hints:

MobileNetV2 by default takes 224x224 image input, so you may need to resize your image from 32x32 to 224x224 (e.g. using skimage.transform.resize for this model. )
PyTorch takes input image in the form of NCHW, which means the four dimension in the input tensor represents Batch, Channel, Height, Width. However by convention the image is saved as NHWC. You may need to swap the dimensions in your input, e.g. using x.permute(0, 3, 1, 2). You can check the Stack Overflow answer or PyTorch documentations.

Submission: Please include the following files:

report.pdf/docx
hw3.py with the following class/functions:
- class CifarDataset
- class MultiLayerPerceptronModel
- function training
- function evaluation