Computational Machine Learning & Data Science

Students will be introduced to the Julia programming language. By the end, students will be familiar with variables, arrays, functions, and everything else in Julia that they need to succeed in this course.

An introduction to matrix math and linear algebra.
Students will learn about vectors, matrices, arrays, and various operations on these objects.

An introduction to the concept of convolution and expressing convolution as a matrix-vector product.

An introduction to normal and non-normal matrices and the spectral theorem for normal matrices, as well as the eigenvalue decomposition and the singular value decomposition and their variational characterizations via eigshow and svdshow.

An introduction to vector spaces, subspaces, and the four fundamental subspaces of a matrix, including a discussion of basis vectors for subspaces and how the SVD of a matrix reveals these bases, as well as orthogonal projection matrices and how to efficiently compute the projection of a vector onto a matrix subspace without first computing and storing the associated projection matrix.

First difference matrix construction and the role of the Kronecker product and sparse constructions thereof.

How to setup and solve least squares problems of the form $Ax = b$. Applications include fitting data to a higher order polynomial function, predicting search query time series results after an appropriate non-linear transformation.

Stochastic descent, Nesterov's accelerated method. Application: photometric stereo reconstruction using these algorithms.

How and why we need to regularize the solution of a system of equations of the form $Ax = b$. Applications include better fitting data to a higher order polynomial function, image in-painting/graffiti removal with a first difference regularizer. Discussion of how the optimal regularization coefficient is selected.

Learning to recognize sparsity in its canonical and transformed manifestations and seeing (computationally) how that helps regularize the solution of a system of equations of the form $Ax = b$ in a regime where minimum norm least squares does not work well. Applications include compressed sensing, image in-painting/graffiti removal with a first difference regularizer and a discussion of how the optimal regularization coefficient is selected.

Convolution plays a role, directly or indirectly, in many data science techniques. Many seemingly complex image filters, for example, may be expressed elegantly using convolution.

Application illustrating how we need to split matrices into blocks and write a matrix-matrix multiplication routine from scratch for settings where the matrices are too big to fit into memory.

Learning to tell apart two classes.

Learning to tell apart multiple classes.

Learning to tell apart multiple classes with logistic regression. What logistic regression is about, why it is different than least squares and what the underlying loss function better captures than the mean squared error loss function in least squares.

Learning to tell apart multiple classes with the nearest subspace classification algorithm.

Fisher's linear discriminant analysis, the generalized eigen-problem and the $Ax = b$ equivalent formulation

First principles derivation of the support vector machine and how its loss function is different from least squares classification. Learning to tell apart multiple classes with the SVM.

False positive, false negatives and how we can change the classification output to weight one or the other. How decision theory informs the choice of decision thresholds.

How to generate inputs that can cause a classifier to give erroneous outputs; how to train an algorithm to be robust to such errors.

What is unsupervised learning and how the k-means algorithm can be used for that.

How to use the singular value decomposition (SVD) to isolate moving objects from the background of a video. Many real-world datasets consist of dynamic signals and noise on top of some static backdrop or baseline. The SVD can isolate these components, allowing us to focus on what we are interested in and disregard the rest.

How to use the SVD to fill in missing elements of a matrix with known rank. Random matrix theory to predict sparsity and incoherence conditions when near-perfect recovery is possible. Applications to low rank image inpainting and collaborative filtering.

The Eckart-Young theorem and its consequences. Applications include image compression and image denoising.

Implement Principal Component Analysis (PCA) and Independent Component Analysis (ICA). These methods reason about covariance and kurtosis to emphasize important data components.

Cumulant Theory of ICA

Viewing PCA and ICA as cumulant maximization problems. Understanding limits of each and finding ways to encode the error of higher order ICA and why kurtosis based ICA is the one used in practice.

Learn2Embed

Spectral algorithms for finding communities in a graph via modularity maximization. Applications include finding communities in a karate club dataset and inferring baseball team division structure from a dataset of how frequently they play each other. Investigation of limits of finding community structure in the stochastic block model.

Phase transitions in PCA and examination of how to reason about unlearnability of structure due outlier-induced phase transitions in the low-rank-plus-noise-plus-sparse outliers data model.

Synchronized Waveforms

Use Flux.jl to quickly design and train single input, single output shallow neural networks for 1-D function approximation

Use Flux.jl to quickly design and train 1-D and 2-D deep feedforward neural nets. See the power of deep nets and critically examine what it means for conv. nets to be "shift-invariant". Applications include handwritten digit recognition and 1-D signal recognition.

Use Flux.jl to quickly design and train convolutional neural nets. Applications include handwritten digit recognition.

Use Flux.jl to quickly design and train recurrent neural nets. Learn why they are used for sequence modeling, what they can do that Conv and Deep Feedforward nets cannot. Applications include image captioning, predicting the Fibonacci sequence and time series.

How to use neural networks in the context of regression and function approximation

Using methods learned to classify red wine versus white wine

Using methods learned to classify varieties of fruit

Using methods learned to tell apart gesures for "rock", "paper" and "scissors"

How to train a network to be robust to adversarial attacks

Deep, denoising and variational autoencoders. Application includes auto-encoding of handwritten digits.

Theory and practice include the choice of the loss function. Application includes generating realistic handwritten digits.