Training

Concept2016

Layer Normalization

Ba, Kiros & Hinton

Normalizing each example across its features

Paper

Concept2015

Batch Normalization

Ioffe & Szegedy

Normalizing layer inputs to accelerate deep network training

Paper

Concept2014

Adam Optimizer

Kingma & Ba

Adaptive learning rates with momentum for deep learning

Paper

Concept2012

Dropout: Regularization for Neural Networks

Hinton et al.

Randomly dropping units during training to prevent overfitting

Paper

Paper2019

GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism

Huang et al.

Training giant neural networks by pipelining micro-batches across devices

Paper

Concept

Pre-training

The stage where a model learns broad patterns from a very large dataset

Paper