This page provides more info about the code and simulations presented in (Toulis et. al., 2014) on implicit stochastic gradient descent (SGD) for large Generalized Linear Models (GLM). Implicit stochastic gradient descent is a modification of the typical SGD algorithm that has very attractive properties. First, it is a shrinked version of the typical (explicit) SGD and the shrinkage factor depends on the observed Fisher information. Thus, implicit SGD is significantly more stable than explicit SGD. In the limit, both methods are statistically equivalent, but in small-to-moderate samples, implicit SGD is more stable, and robust to misspecifications of the learning rate, or outliers.
The following are the main parts of this code:
- Extensive theory and methods are in the paper,
Panos Toulis, Edoardo M. Airoldi, "Implicit stochastic gradient descent for principled estimation with large datasets" (under review) ( pdf)
- Implicit stochastic gradient descent was originally introduced for generalized linera models (GLM)
in the following paper.
Panos Toulis, Jason Rennie, Edoardo Airoldi, "Statistical analysis of stochastic gradient methods for generalized linear models", ICML, Beijing, China, 2014 ( pdf)
- R code implementing SGD and implicit SGD for any Generalized Linear Model (GLM) is
currently maintained in the following Github link.
For the experiments presented in the paper, look for the file
- C++ code implementing implicit online learning methods for SVM is available as a standalone C++ source file and a MAKEFILE. To compile you will need Leon Bottou's SGD 2.0 code. Then download the implicit C++ file and its MAKEFILE and save them in the "svm/" folder in Bottou's codebase. You should be able to compile and run the "svmimplicit" program.