This page provides more info about the code and simulations presented in (Toulis et. al., 2014) on implicit stochastic gradient descent (SGD) for large Generalized Linear Models (GLM). The implicit method is a modification of the typical SGD algorithm that has very attractive properties: (i) asymptotically it has similar guarantees for bias/variance and (ii) in small samples is more stable and more robust to misspecifications of the learning rate and/or outliers.
The following are the main parts of this code:
- Theory and methods are in the paper, Panos Toulis, Jason Rennie, Edoardo Airoldi, "Statistical analysis of stochastic gradient methods for generalized linear models", ICML, Beijing, China, 2014 ( pdf)
- R code implementing SGD and implicit SGD for any Generalized Linear Model (GLM) is
currently maintained in the following Github link.
For the experiments presented in the paper, look for the file
- C++ code implementing implicit online learning methods for SVM is available as a standalone C++ source file and a MAKEFILE. To compile you will need Leon Bottou's SGD 2.0 code. Then download the implicit C++ file and its MAKEFILE and save them in the "svm/" folder in Bottou's codebase. You should be able to compile and run the "svmimplicit" program.