Batch Normalization

If you like our work, please consider supporting us so we can keep doing what we do. And as a current subscriber, enjoy this nice discount!

Also: if you haven’t yet, follow us on Twitter, TikTok, or YouTube!

Batch Normalization is a technique used to improve the performance and stability of neural networks. It was first introduced in a paper by Sergey Ioffe and Christian Szegedy in 2015.

Batch Normalization works by normalizing the input of each layer to have mean 0 and a standard deviation of 1. This is done by first computing the mean and standard deviation of the input for each batch, and then using those values to normalize the input.

Batch Normalization has a number of benefits:

First, it helps to stabilize the training of neural networks by reducing the internal covariate shift. This is the phenomenon where the distribution of the input to a layer changes during training, as the weights of the previous layer are updated. This can slow down training or even cause training to fail. By normalizing the input, Batch Normalization helps to reduce the internal covariate shift and make training more stable.

Second, Batch Normalization helps to improve the performance of neural networks. This is because it allows the network to learn faster and achieve a higher accuracy. Batch Normalization also helps to reduce overfitting.

If you're interested in learning more about Batch Normalization, I recommend reading the original paper or watching this video.

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Training Deep Neural Networks is complicated by the fact that thedistribution of each layer’s inputs changes during training, as the parametersof the previous layers change. This slows down the training by requiring lowerlearning rates and careful parameter initialization, and makes it notoriousl…

arXiv.orgSergey Ioffe

Do you like our work?
Consider becoming a paying subscriber to support us!