Iterative stochastic optimization for large-scale machine learning and statistical inverse problems

Abstract

The twentieth century saw mathematical modeling transition from an intellectual pursuit to an invaluable tool for governments and other large organizations, enabling previously unimaginable feats and reshaping the global landscape. The technologies behind space flight, GPS, globalized financial markets, worldwide logistics, cryptography, and many other achievements of the last century all rely on mathematical models. The twenty-first century has seen a further democratization of mathematics with its utility broadening from large organizations to small businesses and even personal use. Over the course of twenty years, optical character recognition went from a research lab to something a high school student can code in an afternoon; smart grids now efficiently coordinate power from utilities to households; cell phones transcribe a person's voice or translate the words to a foreign language in real-time; self-driving cars are expected this decade. The ubiquity of mathematical models in modern life has led to an increase in the complexity of the models as they are expected to provide results that are more accurate, more precise, and faster. The purpose of this thesis is to balance the requirement of having more expressive and complex models that use more data against the need for these models to be optimized quickly, and sometimes in real-time. To this end develop stochastic iterative methods for optimization, providing novel algorithms along with analysis of the convergence and rate of convergence of these algorithms. We focus specifically on models that require tremendous amounts of data or have many parameters that must be estimated. In the latter case we developed algorithms that are as broadly applicable as possible by focusing on the case where the gradient or even derivative of the objective function can not be computed. The methods developed herein can be used for optimization of machine learning models, statistical inverse problem models, and even physical systems such as robotics.