Paper2017
Attention Is All You Need
Vaswani et al.
The 2017 paper that introduced the Transformer architecture
The AI Encyclopedia
The things worth knowing on the way into the singularity. A growing collection of time-tested cool things in AI: papers, software, interviews, positions, demos, and blogs, each explained with an interactive demo and linked to the originals.
58 entries 57 interactive demos 159 resources
Share it with your AI: llms.txt llms-full.txt Ask Claude Ask ChatGPT
Start here
From gradients to policies, in eight steps.
58 entries
Paper2026
Maximum Likelihood Reinforcement Learning (MaxRL)
Tajwar et al.
A recent idea for training models on pass-fail tasks when sampling matters
Paper2025
Recursive Language Models
Zhang, Kraska & Khattab
A paradigm where LLMs treat context as an environment and recursively call themselves on sub-problems
Paper2023
Mamba: State Space Models
Gu & Dao
A sequence model that keeps a running state instead of attending to every token pair
Concept2022
Chain-of-Thought Prompting
Wei et al.
Eliciting step-by-step reasoning in language models for complex problem solving
Paper2021
CLIP: Contrastive Language-Image Pre-training
Radford et al.
Learning visual concepts from natural language supervision
Paper2021
Latent Diffusion Models
Rombach et al.
High-resolution image generation by diffusing in learned latent spaces
Concept2020
Diffusion Models
Ho, Jain & Abbeel
Generative models that learn to denoise, enabling high-quality image and video synthesis
Concept2020
In-Context Learning
Brown et al.
How large language models learn from examples in the prompt without weight updates
Paper2020
Vision Transformer (ViT)
Dosovitskiy et al.
Applying Transformers directly to image patches for visual recognition
Paper2018
BERT: Bidirectional Transformers
Devlin et al.
Pre-training deep bidirectional representations for NLP
Paper2018
GPT: Generative Pre-Training
Radford et al.
Autoregressive language models that learn to predict the next token
Concept2017
Proximal Policy Optimization (PPO)
Schulman et al.
A stable, sample-efficient policy gradient algorithm for reinforcement learning
Concept2017
RLHF: Reinforcement Learning from Human Feedback
Christiano et al.
Teaching language models to prefer responses that people rank higher
Concept2015
Batch Normalization
Ioffe & Szegedy
Normalizing layer inputs to accelerate deep network training
Concept2014
Generative Adversarial Networks
Goodfellow et al.
Two neural networks compete to generate realistic data
Paper2014
Sequence to Sequence Learning
Sutskever, Vinyals & Le
Encoder-decoder architecture for mapping sequences to sequences
Paper2013
Deep Q-Networks (DQN)
Mnih et al.
Combining Q-learning with deep neural networks for Atari-level game playing
Concept2013
Word2Vec: Word Embeddings
Mikolov et al.
Learning dense vector representations of words from text
Concept2012
Dropout: Regularization for Neural Networks
Hinton et al.
Randomly dropping units during training to prevent overfitting
Concept1992
Policy Gradient Methods
Williams
Directly optimizing policies through gradient ascent on expected returns
Concept1986
Backpropagation
Rumelhart, Hinton & Williams
The algorithm that enables neural networks to learn by computing gradients efficiently
Paper2020
Scaling Laws for Neural Language Models
Kaplan et al.
Why bigger models, more data, and more compute lead to predictable gains
Paper2019
GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism
Huang et al.
Training giant neural networks by pipelining micro-batches across devices
Paper2018
Relational Recurrent Neural Networks
Santoro et al.
RNNs with relational memory that enables reasoning across time
Blog2018
The Annotated Transformer
Sasha Rush
Line-by-line PyTorch implementation of the Transformer architecture
Paper2017
A Simple Neural Network Module for Relational Reasoning
Santoro et al.
Relation Networks for learning to reason about object relationships
Misc2017
Kolmogorov Complexity and Algorithmic Randomness
Shen, Uspensky & Vereshchagin
Measuring how short the best description of an object can be
Paper2017
Neural Message Passing for Quantum Chemistry
Gilmer et al.
A unified framework for graph neural networks applied to molecular property prediction
Paper2016
Identity Mappings in Deep Residual Networks
He et al.
Pre-activation ResNet design that enables training of 1000+ layer networks
Paper2016
Variational Lossy Autoencoder
Chen et al.
Understanding VAEs as compression systems with a rate-distortion trade-off
Misc2015
CS231n: CNNs for Visual Recognition
Stanford
Stanford's foundational course on deep learning for computer vision
Paper2015
Deep Speech 2: End-to-End Speech Recognition
Amodei et al.
Scaling up end-to-end speech recognition with RNNs and CTC
Paper2015
Multi-Scale Context Aggregation by Dilated Convolutions
Yu & Koltun
Expanding receptive fields exponentially without losing resolution or adding parameters
Paper2015
Order Matters: Sequence to Sequence for Sets
Vinyals, Bengio & Kudlur
How input and output ordering affects seq2seq learning on set-structured data
Paper2015
Pointer Networks
Vinyals, Fortunato & Jaitly
Neural architecture that outputs pointers to input positions, enabling variable-size outputs
Blog2015
The Unreasonable Effectiveness of Recurrent Neural Networks
Andrej Karpathy
Andrej Karpathy's influential blog post demonstrating RNN capabilities through character-level generation
Blog2015
Understanding LSTM Networks
Chris Olah
Christopher Olah's visual guide to Long Short-Term Memory networks
Paper2014
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, Cho & Bengio
The paper that introduced the attention mechanism for sequence-to-sequence models
Paper2014
Neural Turing Machines
Graves, Wayne & Danihelka
Neural networks augmented with external memory and attention-based read/write heads
Misc2014
Quantifying the Rise and Fall of Complexity in Closed Systems
Aaronson, Carroll & Ouellette
The Coffee Automaton paper formalizing how complexity peaks then declines
Paper2014
Recurrent Neural Network Regularization
Zaremba, Sutskever & Vinyals
How to apply dropout to LSTMs without disrupting memory dynamics
Concept2013
Variational Autoencoder (VAE)
Kingma & Welling
Probabilistic generative model with structured latent space
Blog2011
The First Law of Complexodynamics
Scott Aaronson
Why complexity rises then falls while entropy only increases
Misc2008
Machine Super Intelligence
Shane Legg
Shane Legg's PhD thesis formalizing universal intelligence and the AIXI agent
Misc2004
A Tutorial Introduction to the Minimum Description Length Principle
Peter Grünwald
Grünwald's comprehensive guide to MDL for model selection and learning
Misc1993
Keeping Neural Networks Simple by Minimizing the Description Length of the Weights
Hinton & van Camp
Hinton's MDL approach to neural network regularization through noisy weights
Paper2019
NODE (Neural Oblivious Decision Ensembles)
Popov et al.
Differentiable decision trees and oblivious ensembles for tabular learning
Misc2001
Gradient Boosted Decision Trees
Jerome Friedman
Sequential tree ensembles optimized via gradient descent
Misc1962
Stable Marriage Problem
Gale & Shapley
Finding a stable matching with the Gale-Shapley deferred acceptance algorithm
No entries match.