Dear all,
Next week’s statphys seminar will be given by Dr Takashi Mori, starting at 1pm, Monday, Dec 6.
Please send an e-mail to
hatano@iis.u-tokyo.ac.jp
for a zoom link.
best regards,
Naomichi Hatano
———————
Speaker: Dr. Takashi Mori
Date and Time: 1pm, Monday December 6, 2021
Title: Stochastic differential equation approach to machine learning dynamics
Abstract
In recent unparalleled success of deep learning, stochastic gradient descent (SGD) or its variants plays a crucial role as an efficient training algorithm. Although the loss landscape is highly nonconvex, the SGD often succeeds in finding a global minimum. It has been argued that the SGD noise plays a key role in escaping from local minima. It has also been suggested that SGD has an implicit bias that is beneficial for generalization. That is, SGD may help the network to find flat minima, which are considered to imply good generalization. How and why does SGD help the network escape from bad local minima and find flat minima? These questions have been addressed in several works, and it is now recognized that the SGD noise strength and structure importantly affect the efficiency of escape from local minima.
In this talk, I explain our recent work [1] following this line of research. We derived a stochastic differential equation (SDE) as a continuous-time approximation of SGD, and investigated the property of SGD noise. It turns out that SGD noise strength significantly depends on the position in the parameter space, and is proportional to the loss function that should be minimized. By using this property, we introduced a random time change that transforms the original SDE with complicated multiplicative noise into a simple SDE with additive noise. I discuss the Kramers escape problem by using this simplified SDE.
[1] Takashi Mori, Liu Ziyin, Kangqiao Liu, Masahito Ueda, “Logarithmic landscape and power-law escape rate of SGD”, arXiv:2105.09557 <arxiv.org/abs/2105.09557>
———————–
Naomichi Hatano
Institute of Industrial Science, University of Tokyo
Kashiwanoha 5-1-5, Kashiwa, Chiba 277-8574, JAPAN
Phone: +81-4-7136-6961
Fax: +81-4-7136-6978
hatano-lab.iis.u-tokyo.ac.jp/hatano/ <hatano-lab.iis.u-tokyo.ac.jp/hatano/>
—
To unsubscribe from this group and stop receiving emails from it, send an email to statphys-seminar+unsubscribe@phys.s.u-tokyo.ac.jp <mailto:statphys-seminar+unsubscribe@phys.s.u-tokyo.ac.jp>.
————————————————-
Computational Material Physics Mailing List
home: www.issp.u-tokyo.ac.jp/public/cmp/
archive: cmp-ml.issp.u-tokyo.ac.jp
twitter: https://twitter.com/cmp_ml
————————————————-