AIpedia

The 2017 paper that introduced the Transformer architecture

A recent idea for training models on pass-fail tasks when sampling matters

Adaptive learning rates with momentum for deep learning

The algorithm that enables neural networks to learn by computing gradients efficiently

Pre-training deep bidirectional representations for NLP

Normalizing layer inputs to accelerate deep network training

Eliciting step-by-step reasoning in language models for complex problem solving

Learning visual concepts from natural language supervision

Generative models that learn to denoise, enabling high-quality image and video synthesis

Combining Q-learning with deep neural networks for Atari-level game playing

Randomly dropping units during training to prevent overfitting

Two neural networks compete to generate realistic data

Autoregressive language models that learn to predict the next token

How large language models learn from examples in the prompt without weight updates

Normalizing each example across its features

High-resolution image generation by diffusing in learned latent spaces

A sequence model that keeps a running state instead of attending to every token pair

Directly optimizing policies through gradient ascent on expected returns

A stable, sample-efficient policy gradient algorithm for reinforcement learning

A paradigm where LLMs treat context as an environment and recursively call themselves on sub-problems

Learning by trial and error through rewards

Teaching language models to prefer responses that people rank higher

Encoder-decoder architecture for mapping sequences to sequences

Applying Transformers directly to image patches for visual recognition

Learning dense vector representations of words from text

The deep CNN that won ImageNet 2012 and sparked the deep learning revolution

Line-by-line PyTorch implementation of the Transformer architecture

The paper that introduced the attention mechanism for sequence-to-sequence models

The Coffee Automaton paper formalizing how complexity peaks then declines

Why complexity rises then falls while entropy only increases

Stanford's foundational course on deep learning for computer vision

Scaling up end-to-end speech recognition with RNNs and CTC

Expanding receptive fields exponentially without losing resolution or adding parameters

Training giant neural networks by pipelining micro-batches across devices

Measuring how short the best description of an object can be

Shane Legg's PhD thesis formalizing universal intelligence and the AIXI agent

Grünwald's comprehensive guide to MDL for model selection and learning

Hinton's MDL approach to neural network regularization through noisy weights

A unified framework for graph neural networks applied to molecular property prediction

Neural networks augmented with external memory and attention-based read/write heads

How input and output ordering affects seq2seq learning on set-structured data

Neural architecture that outputs pointers to input positions, enabling variable-size outputs

Relation Networks for learning to reason about object relationships

Pre-activation ResNet design that enables training of 1000+ layer networks

RNNs with relational memory that enables reasoning across time

Deep residual learning with skip connections that enabled training of 152+ layer networks

Andrej Karpathy's influential blog post demonstrating RNN capabilities through character-level generation

How to apply dropout to LSTMs without disrupting memory dynamics

Why bigger models, more data, and more compute lead to predictable gains

Self-attention models that process sequences in parallel

Christopher Olah's visual guide to Long Short-Term Memory networks

Probabilistic generative model with structured latent space

Understanding VAEs as compression systems with a rate-distortion trade-off

Sequential tree ensembles optimized via gradient descent

Differentiable decision trees and oblivious ensembles for tabular learning

The stage where a model learns broad patterns from a very large dataset

Finding a stable matching with the Gale-Shapley deferred acceptance algorithm

Open-domain platform for web-based reinforcement learning agents