CMSC 678 Fall 2022 - Homework 3

Due by 11:59PM on Tuesday October 25th

Part 1

Get the example PyTorch CNN code that learns to classify CIFAR 10 images from here:

https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html

Here are your tasks and what you'll need to turn in for each of them:

• Run the notebook once without any modifications up to, but not including, the cell that explains how to run on a GPU. Running on the CPU will be fine for this homework. The last cell you'll run shows the accuracy of the CNN for each of the 10 classes on the test data. Submit those accuracies as the first thing in your writeup.
• Experiment with several (at least 4) ways of improving the accuracy of the CNN. Some things you can consider include:
  • Running training longer
  • Changing the learning rate and other hyperparameters
  • Increase or decrease the number of convolutional filters
  • Change the number or sizes of the fully connected layers
  • Change the size/stride of the filters
  • Add more convolutional layers
  • Try a nonlinearity other than ReLIU
  • Add dropout
  Make at least 4 changes to the network one at a time. That is, try one change, revert back to the original network, try another change, and repeat. The goal is to see the impact of one thing by itself. For each change, write up why you think it will help or hurt, submit the test accuracy for all of the classes, and explain what happened (did it help or hurt accuracy and why?). Then, create a final network using as many modifications as you like, say what they are, and submit the final test accuracy over all of the classes.

  At least one change must involve modifying the structure of the network, i.e., the number of layers, the sizes of layers, the numbers of filters, etc.

Part 2

The figure below shows a 2-layer, feed-forward neural network with two hidden-layer nodes and one output node. x1 and x2 are the two inputs. For the following questions, assume the learning rate is 0.1. Each node also has a bias input value of +1. Assume there is a sigmoid activation function at the hidden layer nodes and at the output layer node.

• Calculate the output values at nodes h1, h2 and y of this network for input {x1 = 0, x2 = 1}. Each unit produces as its output the real value computed by the unit's associated sigmoid function. Show all steps in your calculation.
• Compute one (1) step of the backpropagation algorithm for a given example with input {x1 = 0, x2 = 1} and target output y = 1. The network output is the real-valued output of the sigmoid function. The error on the given example is defined as 1/2 times the squared difference between the target output and the actual output. Perform one step of gradient descent to compute the updated weights for both the hidden layer and the output layer. There are nine updated weights in total - the three incoming weights to node h1, the three incoming weights to node h2, and the three incoming weights to node y. Show all steps in your calculation. You may use any resources you like to determine the equations for gradient descent, but you must cite them in your homework submission.