Schedule

Readings:

• Attention Is All You Need
• The Illustrated Transformer
• What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?

Readings:

• What's In My Big Data?
• A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity

Readings:

• DataDecide: How to predict best pretraining data with small experiments
• Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
• DataComp-LM: In search of the next generation of training sets for language models

Readings:

• Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data
• Deduplicating Training Data Makes Language Models Better

Readings:

• Scalable Influence and Fact Tracing for Large Language Model Pretraining
• OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens

Readings:

• 2 OLMo 2 Furious
• Midtraining Bridges Pretraining and Posttraining Distributions

Readings:

• Finetuned Language Models are Zero-Shot Learners
• Scaling Instruction-Finetuned Language Models

Readings:

• Training Language Models to Follow Instructions with Human Feedback (InstructGPT)
• Constitutional AI: Harmlessness from AI Feedback
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Readings:

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training

Readings:

• Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models
• Scalable Extraction of Training Data from (Production) Language Models
• The Files are in the Computer: On Copyright, Memorization, and Generative AI
• Generative AI meets copyright
• Elements of Style: Copyright, Similarity, and Generative AI

Readings:

• Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits
• People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text
• Generative AI enhances individual creativity but reduces the collective diversity of novel content

Readings:

• TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
• WebArena: A Realistic Web Environment for Building Autonomous Agents

Readings:

• The Rise of AI Companions: How Human-Chatbot Relationships Influence Well-Being
• "My Boyfriend is AI": A Computational Analysis of Human-AI Companionship in Reddit's AI Community