Convergence of Stochastic Gradient Descent in Deep Neural Network

• 作 者: Bai-cun ZHOU Cong-ying HAN Tian-de GUO
• 机 构: School of Mathematical Sciences,University of Chinese Academy of Sciences,Beijing 100049,China;Key L
• 出 处: 应用数学学报（英文版）
• 发表日期: 2021年1期
• 关键词: stochastic gradient descent deep neural network convergence

Stochastic gradient descent(SGD)is one of the most common optimization algorithms used in pattern recognition and machine learning.This algorithm and its variants are the preferred algorithm while optimizing parameters of deep neural network for their advantages of low storage space requirement and fast computation speed.Previous studies on convergence of these algorithms were based on some traditional assump-tions in optimization problems.However,the deep neural network has its unique properties.Some assumptions are inappropriate in the actual optimization process of this kind of model.In this paper,we modify the assump-tions to make them more consistent with the actual optimization process of deep neural network.Based on new assumptions,we studied the convergence and convergence rate of SGD and its two common variant algorithms.In addition,we carried out numerical experiments with LeNet-5,a common network framework,on the data set MNIST to verify the rationality of our assumptions.