Stochastic Optimization for Big Data Analytics: Algorithms and Library

SIAM-SDM 2014 Tutorial

Tianbao Yang, Rong Jin and Shenghuo Zhu

Overview

Recent years have witnessed an unprecedented growth of data, from gigabyte to terabyte and even larger, in data analytics. Analyzing and Mining such big data have become increasingly important. However, a concomitant big challenge faced by the community is how to scale up traditional analytic algorithms. Among all, large-scale optimization of big data analytics is a centre theme that attracts a lot of attention.

In this tutorial, we shall cover the most recent advances in optimization for big data analytics. In particular, we will present various stochastic optimization algorithms for some fundamental learning problems including classification and regression.

The first part gives an introduction to machine learning and stochastic optimization that motivates stochastic optimization for big data analytics. The second part presents several start-of-the-art stochastic optimization algorithms for solving big data classification and regression problems. The third part presents general strategies of stochastic optimization, including stochastic gradient descent for a variety of objective functions, accelerated stochastic gradient descent for composite optimization, variance reduced stochastic optimization algorithms, parallel and distributed optimization algorithms. In the fourth part, we discuss some implementation issues and introduce a practical library of distributed stochastic optimization for solving big data classification and regression problems.

Target Audience

This tutorial is intended for researchers, graduate students, and practitioners who are interested in solving big data analytic problems. Through this tutorial, the students will be able to master the understanding of the basic theories and tools for improving the scalability of learning algorithms. The audience will be able to appreciate the library for applying the presented techniques to real-world big data problems such as SVM, Logistic Rregression and Least Square Regression.

The audience is expected to have the basic knowledge of machine learning and convex optimization. Experience with stochastic optimization for solving big data analytic problems will be a plus.

Content

Slides

Slides presented at the tutorial are here.

References