CS2750: Homework 1

• What features would you use?
• What would the labels be?
• How would you collect data?
• Why might the problem turn out to be challenging?

1. First, split the data into three groups: A training set (40%), a validation set (20%), and a test set (40%).
2. You will train a multi-way SVM classifier. We will talk about SVM classifiers in great length later in the class. The goal in this assignment is to use SVMs as a black box, so that you can experiment with the machine learning pipeline we discussed in the introduction. SVM is one of the most popular classifiers; you might need to use it for your project, and it might be one of the few things you remember how to use after this class is over.
3. Pick an SVM software package to use among LIBSVM, LIBLINEAR, SVM Light, the SVM built-in to Matlab, or the SVM in scikit-learn for Python. Look at different packages to see which one you would feel most comfortable using. Read the documentation and find which function in the package you are using performs learning/fitting/training, and which function performs prediction/classification/testing. You can also look for examples of how these functions are used.
4. Include a text file part2.txt in your submission in which you describe why you chose the package that you chose to use. Then copy-paste the parts of the documentation that show how to train and use an SVM.
5. Pick some SVM parameter that the package allows you to tune or specify values for. For now, you won't know what these parameters do, but they will affect the success of your learning algorithm in some way. Your goal is to pick the best value of the parameter, by tuning your model on a validation set. In other words, you will train one model for every value of the parameter of your choice (try 5 different values) and pick which value is best, according to the accuracy on the validation set. You will then use the chosen value of the parameter and apply it to classify the samples in the test set.
6. In the text file mentioned above, report the final accuracy on (1) the training set, and (2) the test set. The difference between these two is called the generalization error. If the code you are using does not include a function to compute accuracy, you should write such a function yourself (it will only be a few lines). To compute accuracy, you will compare the predicted labels for your test samples to the ground-truth (provided with the dataset) labels for those same samples, and compute the fraction of samples that were labeled correctly.
7. Finally, experiment with different amounts of training data, and report how the error on the (1) training data and (2) test data, changes as you add more training data. Include a plot in your write-up with at least 5 different values for the size of the training data on the x-axis, and accuracy on the y-axis, for the training and test sets separately (i.e. show two curves). In the text file, explain what you are observing, and why it might be happening.

1. Loading and splitting the data into train/test/validation and X/Y parts: 5 pts
2. Package choice and documentation excerpts: 5 pts
3. Training with different values of a parameter and using the validation set to pick the best value of that parameter: 10 pts
4. Demonstrating how train/test error changes as more training data is added, including plot: 10 pts

1. Write code to perform clustering over a an NxD data matrix (where N is the number of samples and D is the dimensionality of your feature representation) that you receive as input from the user. Your code should output (1) an Nx1 output containing the data memberships of each sample (denoted by an index from 1 to K, where K is the number of clusters); (2) a KxD matrix containing the mean/center for each cluster; and (3) the final SSD error of the clustering, i.e. the sum of the squared distances between points and their assigned means, summed over all clusters.
2. In your K-means function, try 10 random restarts and return the clustering with the lowest SSD error.
3. You will next test your implementation by applying clustering to segment and recolor an image. Download 10 images from the The Berkeley Segmentation Dataset and Benchmark. To make sure running your method doesn't take a long time, downsample (reduce the size) of your 10 chosen images.
4. To perform segmentation, you need a representation for every image pixel. For simplicity, you will use a three-dimensional feature representation for each pixel, consisting of the R, G and B values of each pixel. You can also include the (x, y) location of each pixel in the feature representation if you wish.
5. Perform clustering over the pixels of the image. Then recolor the pixels of each image according to their cluster membership. In particular, replace each pixel with the average R, G, B values for the center to which the pixel belongs. Include the recoloring result for 10 images and 3 different values of K for each image, in your submission.

1. The correctness of your clustering method implementation: 20 pts
2. Applying your clustering method on images and recoloring depending on cluster membership: 10 pts

1. Use the linear system of equations / least squares solution for the language of your choice. In Matlab, that's the backslash operator, Ax=b => x = A\b. You need to decide what are A, x and b in the case of linear regression.
2. Use the resulting solution to find the wine quality scores on the test data. Then measure and report (in a file part4.txt) the L2 distance between the true and predicted scores.
3. Now implement the gradient descent solution. For this, you will need to initialize the weights in some way (use either random values or all zeros). Then you repeat the following some number of times (for this problem, repeat 10 times). In each iteration, compute the error function gradient using all training data points, then adjust the weights in the direction opposite to the gradient.
4. Apply the solution to the test set, then compute and report the L2 distance as above.
5. Experiment with different learning rates (e.g. ones in the range 10.^(-5:-3) i.e. 0.00001, 0.0001, 0.001) and report your observations.

1. Data set up and least squares solution: 10 pts
2. Gradient descent solution: 20 pts