http://yunpengtai.top/tags/back-propagation/ Recent content in back-propagation on Tai's Blog http://yunpengtai.top/%3Clink%20or%20path%20of%20image%20for%20opengraph,%20twitter-cards%3E http://yunpengtai.top/%3Clink%20or%20path%20of%20image%20for%20opengraph,%20twitter-cards%3E Hugo -- gohugo.io en Wed, 07 Sep 2022 18:10:00 +0800 http://yunpengtai.top/posts/deep-back-propagation/ Wed, 07 Sep 2022 18:10:00 +0800 http://yunpengtai.top/posts/deep-back-propagation/ 1. Gradient descent optimization Gradient-based methods make use of the gradient information to adjust the parameters. Among them, gradient descent can be the simplest. Gradient descent makes the parameters to walk a small step in the direction of the negative gradient. $$ \boldsymbol{w}^{\tau + 1} = \boldsymbol{w}^{\tau} - \eta \nabla_{\boldsymbol{w}^{\tau}} E \tag{1.1} $$ where $\eta, \tau, E$ label learning rate ($\eta > 0$), the iteration step and the loss function. Wait!