Topics
33 articles across 65 tags
Tags
agi aixi algorithm architecture attention autoencoder automata benchmark boosting chemistry cnn combinatorial complexity compression computation computer-vision course ctc deep-learning differentiable distributed ensemble entropy foundation-models game-theory generative gnn graphs ilya-sutskever imagenet information-theory intelligence latent-space lstm machine-learning matching mdl memory model-selection multimodal nlp optimization parallelism physics reasoning regularization reinforcement-learning residual rnn scaling segmentation seq2seq sets sota speech tabular-data theory training transformer transformers translation trees vae visual-qa web-agents
All Articles
AlexNet
The deep CNN that won ImageNet 2012 and sparked the deep learning revolution
The Annotated Transformer
Line-by-line PyTorch implementation of the Transformer architecture
Neural Machine Translation by Jointly Learning to Align and Translate
The paper that introduced the attention mechanism for sequence-to-sequence models
Quantifying the Rise and Fall of Complexity in Closed Systems
The Coffee Automaton paper formalizing how complexity peaks then declines
The First Law of Complexodynamics
Why complexity rises then falls while entropy only increases
CS231n: CNNs for Visual Recognition
Stanford's foundational course on deep learning for computer vision
Deep Speech 2: End-to-End Speech Recognition
Scaling up end-to-end speech recognition with RNNs and CTC
Multi-Scale Context Aggregation by Dilated Convolutions
Expanding receptive fields exponentially without losing resolution or adding parameters
Gradient Boosted Decision Trees
Sequential tree ensembles optimized via gradient descent
GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism
Training giant neural networks by pipelining micro-batches across devices
Kolmogorov Complexity and Algorithmic Randomness
The mathematical foundation for measuring information content and randomness
Machine Super Intelligence
Shane Legg's PhD thesis formalizing universal intelligence and the AIXI agent
A Tutorial Introduction to the Minimum Description Length Principle
Grünwald's comprehensive guide to MDL for model selection and learning
Keeping Neural Networks Simple by Minimizing the Description Length of the Weights
Hinton's MDL approach to neural network regularization through noisy weights
Neural Message Passing for Quantum Chemistry
A unified framework for graph neural networks applied to molecular property prediction
Neural Turing Machines
Neural networks augmented with external memory and attention-based read/write heads
NODE (Neural Oblivious Decision Ensembles)
Differentiable decision trees and oblivious ensembles for tabular learning
Order Matters: Sequence to Sequence for Sets
How input and output ordering affects seq2seq learning on set-structured data
Pointer Networks
Neural architecture that outputs pointers to input positions, enabling variable-size outputs
Pre-training
The initial phase of training foundation models on vast amounts of data
A Simple Neural Network Module for Relational Reasoning
Relation Networks for learning to reason about object relationships
Relational Recurrent Neural Networks
RNNs with relational memory that enables reasoning across time
Identity Mappings in Deep Residual Networks
Pre-activation ResNet design that enables training of 1000+ layer networks
ResNet
Deep residual learning with skip connections that enabled training of 152+ layer networks
The Unreasonable Effectiveness of Recurrent Neural Networks
Andrej Karpathy's influential blog post demonstrating RNN capabilities through character-level generation
Recurrent Neural Network Regularization
How to apply dropout to LSTMs without disrupting memory dynamics
Scaling Laws for Neural Language Models
Empirical laws governing how language model performance scales with compute, data, and parameters
Stable Marriage Problem
Finding stable matchings via the Gale–Shapley algorithm
Attention Is All You Need
The Transformer architecture that replaced recurrence with self-attention
Understanding LSTM Networks
Christopher Olah's visual guide to Long Short-Term Memory networks
Variational Autoencoder (VAE)
Probabilistic generative model with structured latent space
Variational Lossy Autoencoder
Connecting VAEs to lossy compression and the bits-back coding argument
World of Bits
Open-domain platform for web-based reinforcement learning agents