Session II.2 - Continuous Optimization
Friday, June 16, 14:30 ~ 15:00
Homogenization of SGD in High-dimensions
Courtney Paquette
McGill University/Google Brain, Canada - This email address is being protected from spambots. You need JavaScript enabled to view it.
We develop a stochastic differential equation, called homogenized SGD, for analyzing the dynamics of stochastic gradient descent (SGD) on a high-dimensional random generalized linear models. We show that homogenized SGD is the high-dimensional equivalence of SGD– for any $C^3$- statistic (e.g., population risk ), the statistic under the iterates of SGD converges to the statistic under homogenized SGD when the number of samples $n$ and number of features $d$ are polynomially related ($d^c \le n \le d^{1/c}$ for some $c \ge 0$). Several motivating applications are provided including phase retrieval, least-squares, and logistic regression.
Joint work with Elliot Paquette (McGill University), Inbar Seroussi (Tel-Aviv University) and Elizabeth Collins-Woodfin (McGill University.