Reading list

Module 1: Foundations of scalable learning

SGD and machine learning

• Optimization Methods for Large-Scale Machine Learning. SIAM Review.

Optimizers beyond SGD

• Adam: A Method for Stochastic Optimization. ICLR.
• Muon: An Optimizer for Hidden Layers in Neural Networks. Blog post.
• On the Importance of Initialization and Momentum in Deep Learning. ICML.
• Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. JMLR.
• On the Convergence of Adam and Beyond. ICLR.
• Shampoo: Preconditioned Stochastic Tensor Optimization. ICML.

Parallel training in distributed SGD

• Large Scale Distributed Deep Networks. NeurIPS.
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv.
• Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent. NeurIPS.
• Scaling Distributed Machine Learning with the Parameter Server. OSDI.
• Can Decentralized Algorithms Outperform Centralized Algorithms?. NeurIPS.
• Local SGD Converges Fast and Communicates Little. ICLR.

Optimizer efficiency at scale (memory, communication, mixed precision)

• Mixed Precision Training. ICLR.
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. NeurIPS.
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML.
• Training Deep Nets with Sublinear Memory Cost. arXiv.
• signSGD: Compressed Optimisation for Non-Convex Problems. ICML.
• Deep Gradient Compression. ICLR.

Scaling bottlenecks in distributed SGD

• An Empirical Model of Large-Batch Training. arXiv.
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. SC.
• Measuring the Effects of Data Parallelism on Neural Network Training. JMLR.
• Roofline: An Insightful Visual Performance Model for Multicore Architectures. CACM.
• PipeDream: Generalized Pipeline Parallelism for DNN Training. SOSP.
• Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes. ICLR.

Module 2: Practical ML systems beyond accuracies

Federated and collaborative learning

• Communication-Efficient Learning of Deep Networks from Decentralized Data. AISTATS.
• Towards Federated Learning at Scale: System Design. MLSys.
• Federated Optimization in Heterogeneous Networks. MLSys.
• Deep Leakage from Gradients. NeurIPS.

ML pipeline in practice and model compression

• Hidden Technical Debt in Machine Learning Systems. NeurIPS.
• Distilling the Knowledge in a Neural Network. arXiv.
• Deep Compression. ICLR.
• Challenges in Deploying Machine Learning: A Survey of Case Studies. ACM Computing Surveys.
• The Lottery Ticket Hypothesis. ICLR.

Random hashing in large-scale ML

• Feature Hashing for Large Scale Multitask Learning. ICML.
• An Improved Data Stream Summary: The Count-Min Sketch and its Applications. J. Algorithms.
• Compressing Neural Networks with the Hashing Trick. ICML.
• SLIDE: In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems. MLSys.
• Reformer: The Efficient Transformer. ICLR.

Robustness, distribution shift, and privacy

• Towards Deep Learning Models Resistant to Adversarial Attacks. ICLR.
• WILDS: A Benchmark of in-the-Wild Distribution Shifts. ICML.
• Deep Learning with Differential Privacy. CCS.
• Certified Adversarial Robustness via Randomized Smoothing. ICML.
• Extracting Training Data from Large Language Models. USENIX Security.

Module 3: LLM systems in practice

LLM pipeline, language models and transformers

• Attention Is All You Need. NeurIPS.
• Language Models are Few-Shot Learners. NeurIPS.
• Language Models are Unsupervised Multitask Learners. OpenAI tech report.
• BERT: Pre-training of Deep Bidirectional Transformers. NAACL.

Data selection, labeling, and infrastructure

• DataComp-LM: In Search of the Next Generation of Training Sets for Language Models. NeurIPS D&B.
• DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. NeurIPS.
• Dolma: An Open Corpus of Three Trillion Tokens. ACL.
• Data Selection for Language Models via Importance Resampling. NeurIPS.
• Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning. NeurIPS.
• Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. ICML.
• Selection via Proxy: Efficient Data Selection for Deep Learning. ICLR.
• Snorkel: Rapid Training Data Creation with Weak Supervision. VLDB.

Optimizers and distributed training for language models

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS.
• Muon is Scalable for LLM Training. arXiv.
• Fantastic Pretraining Optimizers and Where to Find Them. ICLR.

Supervised finetuning and RL

• Training Language Models to Follow Instructions with Human Feedback. NeurIPS.
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS.
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv.
• Finetuned Language Models Are Zero-Shot Learners. ICLR.
• LoRA: Low-Rank Adaptation of Large Language Models. ICLR.
• Deep Reinforcement Learning from Human Preferences. NeurIPS.

Efficient inference for language models

• Efficiently Scaling Transformer Inference. MLSys.
• Fast Inference from Transformers via Speculative Decoding. ICML.
• SGLang: Efficient Execution of Structured Language Model Programs. NeurIPS.
• Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. ICLR.

Multi-agent systems: overview and applications

• ReAct: Synergizing Reasoning and Acting in Language Models. ICLR.
• Building Effective Agents. Blog post.
• Why Do Multi-Agent LLM Systems Fail?. arXiv.