CS 1678: Homework 5

1. Read the code in rnn_datasets.py and sentiment_analysis.py, then run the training with provided GRU cell. You should be able to achieve at least 85% accuracy in 50 epochs with default hyperparameters.
   Attach the figures (either TensorBoard screenshot or plot on your own) of (1) training loss, (2) training accuracy per epoch and (3) validation accuracy per epoch in your report.
2. Implement the following variant in rnn_modules.py, details in the blog, the section named Variants on Long Short Term Memory.
   • LSTMCell: classical LSTM with input gate and forget gate, also cell state and hidden state.
   Please note that the class is already provided for you, and you only need to complete the __init__ and forward functions. Please do NOT change the signatures.
   
3. Print number of model parameters for different module type (GRUCell and LSTMCell) using count_parameters, and include the comparison in your notebook.
   Use the following hyperparameters for comparison: input_size=128, hidden_size=100, bias=True.
4. Run experiments with your custom implementation on sentiment analysis with IMDB dataset, and compare the results with the GRU, including both speed and performance.
   Attach the training loss and training/validation accuracy plot (or include it in your notebook).

1. Read the code in rnn_datasets.py and sentence_generation.py, and complete the SentenceGeneration class which is a character-level RNN.
   You can reuse (by copy/paste) most of the codes from SentimentClassification in Part A, just note that instead of predicting positive or negative, now your task is to predict the next character given a sequence of chars (history).
   
2. Train the model with the GRU module on Shakespeare books. You should be able to achieve loss value of 1.2 in 10 epochs with default hyperparameters. You probably need to use an embedding_dim of 256 and hidden_size of 512 to get the above loss value. If you are interested you could try with your own LSTM variant, but experiments with GRU is required.
3. Complete the function in sentence_generation.py to load your trained model and generate new sentence from it.
   Basically, once a language model is trained, it is able to predict the next character after a sequence, and this process can be continued (predicted character serve as history for predicting the next).
   More specifically, your model should be able to predict the probability distribution over the vocabulary for the next character, and we have implemented a sampler sample_next_char_id which samples according to the probability. By repeating this process, your model is able to write arbitrarily long paragraphs.
   For example the following passage is written by a GRU trained on Shakespeare:
   

   ROMEO:Will't Marcius Coriolanus and envy of smelling!
   
   DUKE VINCENTIO:
   He seems muster in the shepherd's bloody winds;
   Which any hand and my folder sea fast,
   Last vantage doth be willing forth to have.
   Sirraher comest that opposite too?
   
   JULIET:
   Are there incensed to my feet relation!
   Down with mad appelate bargage! troubled
   My brains loved and swifter than edwards:
   Or, hency, thy fair bridging courseconce,
   Or else had slept a traitors in mine own.
   Look, Which canst thou have no thought appear.
   
   ROMEO:
   Give me them: for that I put us empty.
   
   RIVERS:
   The shadow doth not live: and he would not
   From thee for his that office past confusion
   Is their great expecteth on the wheek;
   But not the noble fathom were an poison
   Here come to make a dukedom: therefore--
   But O, God grant! for Signior HERY
   
   VI:
   Soft love, that Lord Angelo: then blaze me all;
   And slept not without a Calivan Us.
   

   Note that the model learns how to spell each word and write the sentence-like paragraphs all by itself, even including the punctuations and line breaks.
   Please use ROMEO and JULIET as history to begin the generation for 1000 characters each, and attach the generated text (or include it in your notebook).