# AIpedia

> The AI Encyclopedia: a growing collection of time-tested cool things in AI (papers, software, interviews, positions, demos, and blogs), the things worth knowing on the way into the singularity. Each entry has a plain-language explanation, an interactive demo on the website, and links to the originals.

Each link below is a Markdown version of an entry. Everything in one file: https://aipedia.org/llms-full.txt

## Papers

- [AlexNet](https://aipedia.org/alexnet.md): The deep CNN that won ImageNet 2012 and sparked the deep learning revolution (Krizhevsky, Sutskever & Hinton, 2012)
- [Deep Q-Networks (DQN)](https://aipedia.org/dqn.md): Combining Q-learning with deep neural networks for Atari-level game playing (Mnih et al., 2013)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://aipedia.org/bahdanau-attention.md): The paper that introduced the attention mechanism for sequence-to-sequence models (Bahdanau, Cho & Bengio, 2014)
- [Neural Turing Machines](https://aipedia.org/neural-turing-machines.md): Neural networks augmented with external memory and attention-based read/write heads (Graves, Wayne & Danihelka, 2014)
- [Recurrent Neural Network Regularization](https://aipedia.org/rnn-regularization.md): How to apply dropout to LSTMs without disrupting memory dynamics (Zaremba, Sutskever & Vinyals, 2014)
- [Sequence to Sequence Learning](https://aipedia.org/seq2seq.md): Encoder-decoder architecture for mapping sequences to sequences (Sutskever, Vinyals & Le, 2014)
- [Deep Speech 2: End-to-End Speech Recognition](https://aipedia.org/deep-speech-2.md): Scaling up end-to-end speech recognition with RNNs and CTC (Amodei et al., 2015)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://aipedia.org/dilated-convolutions.md): Expanding receptive fields exponentially without losing resolution or adding parameters (Yu & Koltun, 2015)
- [Order Matters: Sequence to Sequence for Sets](https://aipedia.org/order-matters.md): How input and output ordering affects seq2seq learning on set-structured data (Vinyals, Bengio & Kudlur, 2015)
- [Pointer Networks](https://aipedia.org/pointer-networks.md): Neural architecture that outputs pointers to input positions, enabling variable-size outputs (Vinyals, Fortunato & Jaitly, 2015)
- [ResNet](https://aipedia.org/resnet.md): Deep residual learning with skip connections that enabled training of 152+ layer networks (He et al., 2015)
- [Identity Mappings in Deep Residual Networks](https://aipedia.org/resnet-identity.md): Pre-activation ResNet design that enables training of 1000+ layer networks (He et al., 2016)
- [Variational Lossy Autoencoder](https://aipedia.org/variational-lossy-autoencoder.md): Understanding VAEs as compression systems with a rate-distortion trade-off (Chen et al., 2016)
- [A Simple Neural Network Module for Relational Reasoning](https://aipedia.org/relational-reasoning.md): Relation Networks for learning to reason about object relationships (Santoro et al., 2017)
- [Attention Is All You Need](https://aipedia.org/attention-is-all-you-need.md): The 2017 paper that introduced the Transformer architecture (Vaswani et al., 2017)
- [Neural Message Passing for Quantum Chemistry](https://aipedia.org/neural-message-passing.md): A unified framework for graph neural networks applied to molecular property prediction (Gilmer et al., 2017)
- [World of Bits](https://aipedia.org/world-of-bits.md): Open-domain platform for web-based reinforcement learning agents (Shi et al., 2017)
- [BERT: Bidirectional Transformers](https://aipedia.org/bert.md): Pre-training deep bidirectional representations for NLP (Devlin et al., 2018)
- [GPT: Generative Pre-Training](https://aipedia.org/gpt.md): Autoregressive language models that learn to predict the next token (Radford et al., 2018)
- [Relational Recurrent Neural Networks](https://aipedia.org/relational-rnn.md): RNNs with relational memory that enables reasoning across time (Santoro et al., 2018)
- [GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism](https://aipedia.org/gpipe.md): Training giant neural networks by pipelining micro-batches across devices (Huang et al., 2019)
- [NODE (Neural Oblivious Decision Ensembles)](https://aipedia.org/node.md): Differentiable decision trees and oblivious ensembles for tabular learning (Popov et al., 2019)
- [Scaling Laws for Neural Language Models](https://aipedia.org/scaling-laws.md): Why bigger models, more data, and more compute lead to predictable gains (Kaplan et al., 2020)
- [Vision Transformer (ViT)](https://aipedia.org/vision-transformer.md): Applying Transformers directly to image patches for visual recognition (Dosovitskiy et al., 2020)
- [CLIP: Contrastive Language-Image Pre-training](https://aipedia.org/clip.md): Learning visual concepts from natural language supervision (Radford et al., 2021)
- [Latent Diffusion Models](https://aipedia.org/latent-diffusion.md): High-resolution image generation by diffusing in learned latent spaces (Rombach et al., 2021)
- [Mamba: State Space Models](https://aipedia.org/mamba.md): A sequence model that keeps a running state instead of attending to every token pair (Gu & Dao, 2023)
- [Recursive Language Models](https://aipedia.org/recursive-language-models.md): A paradigm where LLMs treat context as an environment and recursively call themselves on sub-problems (Zhang, Kraska & Khattab, 2025)
- [Maximum Likelihood Reinforcement Learning (MaxRL)](https://aipedia.org/maximum-likelihood-rl.md): A recent idea for training models on pass-fail tasks when sampling matters (Tajwar et al., 2026)

## Concepts

- [Backpropagation](https://aipedia.org/backpropagation.md): The algorithm that enables neural networks to learn by computing gradients efficiently (Rumelhart, Hinton & Williams, 1986)
- [Policy Gradient Methods](https://aipedia.org/policy-gradient.md): Directly optimizing policies through gradient ascent on expected returns (Williams, 1992)
- [Dropout: Regularization for Neural Networks](https://aipedia.org/dropout.md): Randomly dropping units during training to prevent overfitting (Hinton et al., 2012)
- [Variational Autoencoder (VAE)](https://aipedia.org/vae.md): Probabilistic generative model with structured latent space (Kingma & Welling, 2013)
- [Word2Vec: Word Embeddings](https://aipedia.org/word2vec.md): Learning dense vector representations of words from text (Mikolov et al., 2013)
- [Adam Optimizer](https://aipedia.org/adam.md): Adaptive learning rates with momentum for deep learning (Kingma & Ba, 2014)
- [Generative Adversarial Networks](https://aipedia.org/gan.md): Two neural networks compete to generate realistic data (Goodfellow et al., 2014)
- [Batch Normalization](https://aipedia.org/batch-normalization.md): Normalizing layer inputs to accelerate deep network training (Ioffe & Szegedy, 2015)
- [Layer Normalization](https://aipedia.org/layer-normalization.md): Normalizing each example across its features (Ba, Kiros & Hinton, 2016)
- [Proximal Policy Optimization (PPO)](https://aipedia.org/ppo.md): A stable, sample-efficient policy gradient algorithm for reinforcement learning (Schulman et al., 2017)
- [RLHF: Reinforcement Learning from Human Feedback](https://aipedia.org/rlhf.md): Teaching language models to prefer responses that people rank higher (Christiano et al., 2017)
- [Transformer](https://aipedia.org/transformer.md): Self-attention models that process sequences in parallel (Vaswani et al., 2017)
- [Diffusion Models](https://aipedia.org/diffusion-models.md): Generative models that learn to denoise, enabling high-quality image and video synthesis (Ho, Jain & Abbeel, 2020)
- [In-Context Learning](https://aipedia.org/in-context-learning.md): How large language models learn from examples in the prompt without weight updates (Brown et al., 2020)
- [Chain-of-Thought Prompting](https://aipedia.org/chain-of-thought.md): Eliciting step-by-step reasoning in language models for complex problem solving (Wei et al., 2022)
- [Pre-training](https://aipedia.org/pre-training.md): The stage where a model learns broad patterns from a very large dataset
- [Reinforcement Learning](https://aipedia.org/reinforcement-learning.md): Learning by trial and error through rewards

## Blogs

- [The First Law of Complexodynamics](https://aipedia.org/complexodynamics.md): Why complexity rises then falls while entropy only increases (Scott Aaronson, 2011)
- [The Unreasonable Effectiveness of Recurrent Neural Networks](https://aipedia.org/rnn-effectiveness.md): Andrej Karpathy's influential blog post demonstrating RNN capabilities through character-level generation (Andrej Karpathy, 2015)
- [Understanding LSTM Networks](https://aipedia.org/understanding-lstms.md): Christopher Olah's visual guide to Long Short-Term Memory networks (Chris Olah, 2015)
- [The Annotated Transformer](https://aipedia.org/annotated-transformer.md): Line-by-line PyTorch implementation of the Transformer architecture (Sasha Rush, 2018)

## Misc

- [Stable Marriage Problem](https://aipedia.org/stable-marriage.md): Finding a stable matching with the Gale-Shapley deferred acceptance algorithm (Gale & Shapley, 1962)
- [Keeping Neural Networks Simple by Minimizing the Description Length of the Weights](https://aipedia.org/mdl-weights.md): Hinton's MDL approach to neural network regularization through noisy weights (Hinton & van Camp, 1993)
- [Gradient Boosted Decision Trees](https://aipedia.org/gbdt.md): Sequential tree ensembles optimized via gradient descent (Jerome Friedman, 2001)
- [A Tutorial Introduction to the Minimum Description Length Principle](https://aipedia.org/mdl-tutorial.md): Grünwald's comprehensive guide to MDL for model selection and learning (Peter Grünwald, 2004)
- [Machine Super Intelligence](https://aipedia.org/machine-superintelligence.md): Shane Legg's PhD thesis formalizing universal intelligence and the AIXI agent (Shane Legg, 2008)
- [Quantifying the Rise and Fall of Complexity in Closed Systems](https://aipedia.org/coffee-automaton.md): The Coffee Automaton paper formalizing how complexity peaks then declines (Aaronson, Carroll & Ouellette, 2014)
- [CS231n: CNNs for Visual Recognition](https://aipedia.org/cs231n.md): Stanford's foundational course on deep learning for computer vision (Stanford, 2015)
- [Kolmogorov Complexity and Algorithmic Randomness](https://aipedia.org/kolmogorov-complexity.md): Measuring how short the best description of an object can be (Shen, Uspensky & Vereshchagin, 2017)

## Optional

- [All entries](https://aipedia.org/llms-full.txt): every entry in full, in one file