Recent Preprints

  • Generative Experiences for Digital Mental Health Interventions: Evidence from a Randomized Study

    Ananya Bhattacharjee, Michael Liut, Matthew JΓΆrke, Diyi Yang, Emma Brunskill
  • Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforce

    Yijia Shao*, Humishka Zope*, Yucheng Jiang, Jiaxin Pei, David Nguyen, Erik Brynjolfsson, Diyi Yang
  • Optimizing Pretraining Data Mixtures with LLM-Estimated Utility

    William Held, Bhargavi Paranjape, Punit Singh Koura, Mike Lewis, Frank Zhang, Todor Mihaylov
  • Social Skill Training with Large Language Models

    Diyi Yang*, Caleb Ziems*, William Held*, Omar Shaikh*, Michael S. Bernstein, John Mitchell

2026

  • ProgramBench: Can Language Models Rebuild Programs From Scratch?

    John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, Diyi Yang, Ofir Press
  • Offloading Score: Measuring AI Reliance Through Counterfactual Workflows

    Vishakh Padmakumar, Lujain Ibrahim, Zora Zhiruo Wang, Jennifer Wang, Q. Vera Liao, Diyi Yang
  • Verbalizing LLMs' assumptions to explain and control sycophancy

    Myra Cheng, Isabel Sieh, Humishka Zope, Sunny Yu, Lujain Ibrahim, Aryaman Arora, Jared Moore, Desmond Ong, Dan Jurafsky, Diyi Yang
  • SWE-chat: Coding Agent Interactions From Real Users in the Wild

    Joachim Baumann, Vishakh Padmakumar, Xiang Li, John Yang, Diyi Yang, Sanmi Koyejo
  • Relative Scaling Laws for LLMs

    William Held, David Leo Wright Hall, Percy Liang, Diyi Yang
  • Learning Next Action Predictors from Human-Computer Interaction

    Omar Shaikh, Valentin Teutschbein, Kanishk Gandhi, Yikun Chi, Nick Haber, Thomas N. Robinson, Nilam Ram, Byron Reeves, Sherry Yang, Michael S. Bernstein, Diyi Yang
  • Do Language Models Consistently Encode the Current Year?

    Suze van Adrichem, Aditi Bhaskar, Diyi Yang, Christopher Potts, Jing Huang
  • CooperBench: Why Coding Agents Cannot be Your Teammates Yet

    Arpandeep Khatua*, Hao Zhu*, Peter Tran, Arya Prabhudesai, Frederic Sadrieh, Johann Kaspar Lieberwirth, Xinkai Yu, Yicheng Fu, Michael J Ryan, Jiaxin Pei, Diyi Yang
  • CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks

    Yijia Shao, Zora Zhiruo Wang, Neel Ahuja, Yicheng Wang, Bowen Liu, Diyi Yang
  • Whose Knowledge Counts? Co-Designing Community-Centered AI Auditing Tools with Educators in Hawaii

    Dora Zhao, Hannah Cha, Michael Ryan, Angelina Wang, Rachel Baker-Ramos, Evyn-Bree Helekahi-Kaiwi, Rebecca Diego, Josiah Hester, Diyi Yang
  • Towards Execution-Grounded Automated AI Research

    Chenglei Si, Zitong Yang, Yejin Choi, Emmanuel Candes, Diyi Yang, Tatsunori Hashimoto
  • The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas

    Chenglei Si, Tatsunori Hashimoto, Diyi Yang
  • Searching for Privacy Risks in LLM Agents via Simulation

    Yanzhe Zhang, Diyi Yang
  • Real-Time Reasoning Agents in Evolving Environments

    Yule Wen, Yixin Ye, Yanzhe Zhang, Diyi Yang, Hao Zhu
  • Mapping the Spiral of Silence: Surveying Unspoken Opinions in Online Communities

    Dora Zhao, Diyi Yang, Michael Bernstein
  • HumanLM: Simulating Users with State Alignment Beats Response Imitation

    Shirley Wu, Evelyn Choi, Arpandeep Khatua, Zhanghan Wang, Joy He-Yueya, Tharindu Cyril Weerasooriya, Wei Wei, Diyi Yang, Jure Leskovec, James Zou
  • How Dark Patterns Manipulate Web Agents

    Phil Cuvin, Hao Zhu, Diyi Yang
  • Contextualized Privacy Defense for LLM Agents

    Yule Wen, Yanzhe Zhang, Jianxun Lian, Xiaoyuan Yi, Xing Xie, Diyi Yang
  • CodeClash: Benchmarking Goal-Oriented Software Engineering

    John Yang, Kilian Lieret, Joyce Yang, Carlos E. Jimenez, Muhtasham Oblokulov, Aryan Siddiqui, Ofir Press, Ludwig Schmidt, Diyi Yang
  • AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators

    Michael Ryan, Yanzhe Zhang, Amol Salunkhe, Yi Chu, Di Xu, Diyi Yang
  • AutoLibra: Agent Metric Induction from Open-Ended Human Feedback

    Hao Zhu, Phil Cuvin, Xinkai Yu, Charlotte Ka Yee Yan, Jason Zhang, Diyi Yang

2025

  • Creating General User Models from Computer Use

    Omar Shaikh, Shardul Sapkota, Shan Rizvi, Eric Horvitz, Joon Sung Park, Diyi Yang, Michael S. Bernstein
  • Blackbox Model Provenance via Palimpsestic Membership Inference

    Rohith Kuditipudi, Jing Huang, Sally Zhu, Percy Liang, Christopher Potts, Diyi Yang
  • SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs

    Michael Ryan, Omar Shaikh, Aditri Bhagirath, Daniel Frees, William Held, Diyi Yang
  • SPHERE: An Evaluation Card for Human-AI Systems

    Dora Zhao*, Qianou Ma*, Xinran Zhao, Chenglei Si, Chenyang Yang, Ryan Louie, Ehud Reiter, Diyi Yang*, Tongshuang Wu*
  • Mind the Gap: Static and Interactive Evaluations of Large Audio Models

    Ella Li*, William Held*, Michael Ryan, Kunat Pipatanakul, Potsawee Manakul, Hao Zhu, Diyi Yang
  • EgoNormia: Benchmarking Physical Social Norm Understanding

    MohammadHossein Rezaei*, Yicheng Fu*, Phil Cuvin*, Caleb Ziems, Yanzhe Zhang, Hao Zhu, Diyi Yang
  • Distilling an End-to-End Voice Assistant Without Instruction Training Data

    William Held, Yanzhe Zhang, Weiyan Shi, Ella Li, Michael Ryan, Diyi Yang
  • Attacking Vision-Language Computer Agents via Pop-ups

    Yanzhe Zhang, Tao Yu, Diyi Yang
  • Can LLM-Simulated Practice and Feedback Upskill Human Counselors? A Randomized Study with 90+ Novice Counselors

    Ryan Louie, Ifdita Hasan Orney, Juan Pablo Pacheco, Raj Sanjay Shah, Emma Brunskill, Diyi Yang
  • Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors

    Jing Huang*, Junyi Tao*, Thomas Icard, Diyi Yang, Christopher Potts
  • SWE-smith: Scaling Data for Software Engineering Agents

    John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, Diyi Yang
  • Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration

    Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, Diyi Yang
  • SWE-bench Multimodal: Do Autonomous Programming Systems Generalize to New Software Domains

    John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, Diyi Yang, Sida Wang, Ofir Press
  • No Preference Left Behind: Group Distributional Preference Optimization

    Binwei Yao, Zefan Cai, Yun-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, Junjie Hu
  • Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers

    Chenglei Si, Diyi Yang, Tatsunori Hashimoto
  • Aligning Language Models with Demonstrated Feedback

    Omar Shaikh*, Michelle Lam*, Joey Hejna*, Yijia Shao, Hyundong Justin Cho, Michael Bernstein, Diyi Yang
  • Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping

    Ryan Li, Yanzhe Zhang, Diyi Yang
  • Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering

    Chenglei Si*, Yanzhe Zhang*, Ryan Li, Zhengyuan Yang, Ruibo Liu, Diyi Yang

2024

  • Semi-Truths: A Large-Scale Dataset for Testing Robustness of AI-Generated Image Detectors

    Anisha Pal, Julia Kruk, Mansi Phute, Manognya Bhattaram, Diyi Yang, Duen Horng Chau, Judy Hoffman
  • PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action

    Yijia Shao, Tianshi Li, Weiyan Shi, Yanchen Liu, Diyi Yang
  • DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph

    Zhehao Zhang, Jiaao Chen, Diyi Yang
  • Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles

    Ryan Louie, Ananjan Nandi, William Fang, Cheng Chang, Emma Brunskill, Diyi Yang
  • Modeling Gender and Dialect Bias in Automatic Speech Recognition

    Camille Harris, Chiji Mgbahurike, Neha Kumar, Diyi Yang
  • Demystifying Verbatim Memorization in Large Language Models

    Jing Huang, Diyi Yang, Christopher Potts
  • Decoding Susceptibility: Modeling Misbelief to Misinformation Through a Computational Approach

    Yanchen Liu, Mingyu Derek Ma, Wenna Qin, Azure Zhou, Jiaao Chen, Weiyan Shi, Wei Wang, Diyi Yang
  • CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies

    Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Chunhua Yu, Raya Horesh, RogΓ©rio Abreu de Paula, Diyi Yang
  • Benchmarking LLM-based Machine Translation on Cultural Awareness

    Binwei Yao, Ming Jiang, Diyi Yang, Junjie Hu
  • Are Large Language Models Consistent over Value-laden Questions?

    Jared Moore, Tanvi Deshpande, Diyi Yang
  • The Practice of Online Peer Counseling and the Potential for AI-Powered Support Tools

    Tony Wang, Amy Bruckman, Diyi Yang
  • Helping the Helper: Supporting Peer Counselors via AI-Empowered Practice and Feedback

    Shang-Ling Hsu, Raj Sanjay Shah, Prathik Senthil, Zahra Ashktorab, Casey Dugan, Werner Geyer, Diyi Yang
  • Dynamic LLM-Agent Network: An LLM-agent Collaboration Framework with Agent Team Optimization

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, Diyi Yang
  • Auditing Gender Presentation Differences in Text-to-Image Models

    Yanzhe Zhang, Lu Jiang, Greg Turk, Diyi Yang
  • Simulated Misinformation Susceptibility (SMISTS): Enhancing Misinformation Research with Large Language Model Simulations

    Weicheng Ma, Chunyuan Deng, Aram Moossavi, Lili Wang, Soroush Vosoughi, Diyi Yang
  • Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistles

    Julia Kruk, Michela Marchini, Rijul Ragu, Caleb Ziems, David Muchlinski, Diyi Yang
  • Perceptions of Language Technology Failures from South Asian English Speakers

    Faye Holt*, William Held*, Diyi Yang
  • Measuring and Addressing Indexical Bias in Information Retrieval

    Caleb Ziems, William Held, Jane Dwivedi-Yu, Diyi Yang
  • Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

    Matthias Gerstgrasser*, Rylan Schaeffer*, Apratim Dey*, Rafael Rafailov*, Dhruv Pai, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Andrey Gromov, Daniel A. Roberts, Diyi Yang, David L. Donoho, Sanmi Koyejo
  • Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselors

    Alicja Chaszczewicz, Raj Sanjay Shah, Ryan Louie, Bruce A Arnow, Robert Kraut, Diyi Yang
  • Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Future

    Minzhi Li, Weiyan Shi, Caleb Ziems, Diyi Yang
  • Unintended Impacts of LLM Alignment on Global Representation

    Michael Ryan, William Held, Diyi Yang
  • Rehearsal: Simulating Conflict to Teach Conflict Resolution

    Omar Shaikh, Valentino Chai, Michele J. Gelfand, Diyi Yang, Michael S. Bernstein
  • Training Socially Aligned Language Models on Simulated Social Interactions

    Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M. Dai, Diyi Yang, Soroush Vosoughi
  • DyVal: Graph-informed Dynamic Evaluation of Large Language Models

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, Xing Xie
  • What Makes Digital Support Effective? How Therapeutic Skills Affect Clinical Well-Being

    Anna Fang, Wenjie Yang, Raj Sanjay Shah, Yash Mathur, Diyi Yang, Haiyi Zhu, Robert Kraut
  • Anchor Points: Benchmarking Models with Much Fewer Examples

    Rajan Vivek, Kawin Ethayarajh, Diyi Yang, Douwe Kiela

2023

  • LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, Tong Sun
  • Using Large Language Models in Psychology

    Dorottya Demszky*, Diyi Yang*, David S. Yeager*, Christopher J. Bryan, Margarett Clapper, Susannah Chandhok, Johannes C. Eichstaedt, Cameron Hecht, Jeremy Jamieson, Meghann Johnson, Michaela Jones, Danielle Krettek-Cobb, Leslie Lai, Nirel JonesMitchell, Desmond C. Ong, Carol S. Dweck, James J. Gross, James W. Pennebaker
  • Unlearn What You Want to Forget: Efficient Unlearning for LLMs

    Jiaao Chen, Diyi Yang
  • Understanding Black Content Creator Experiences on TikTok

    Camille Harris, Amber Gayle Johnson, Sadie Palmer, Diyi Yang, Amy Bruckman
  • Task Agnostic Dialect Adapters for English

    William Held, Caleb Ziems, Diyi Yang
  • Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers

    William Held, Diyi Yang
  • Parameter-Efficient Fine-Tuning Design Spaces

    Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, Diyi Yang
  • On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning

    Omar Shaikh, Hongxin Zhang, William Held, Michael S. Bernstein, Diyi Yang
  • NormBank: A Knowledge Bank of Situational Social Norms

    Caleb Ziems, Jane Dwivedi-Yu, Yi-Chia Wang, Alon Halevy, Diyi Yang
  • Multi-VALUE: A Framework for Cross-Dialectal English NLP

    Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, Diyi Yang
  • Modeling Cross-Cultural Pragmatic Inference with Codenames Duet

    Omar Shaikh, Caleb Ziems, William Held, Aryan J. Pariani, Fred Morstatter, Diyi Yang
  • Impressions: Visual Semiotics and Aesthetic Impact Understanding

    Julia Kruk, Caleb Ziems, Diyi Yang
  • Human-in-the-loop Abstractive Dialogue Summarization

    Jiaao Chen, Mohan Dodda, Diyi Yang
  • Forgotten Knowledge: Examining the Citational Amnesia in NLP

    Janvijay Singh, Mukund Rungta, Diyi Yang, Saif M. Mohammad
  • DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules

    Yanchen Liu, William Held, Diyi Yang
  • CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations

    Myra Cheng, Tiziano Piccardi, Diyi Yang
  • CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

    Ella Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy F. Chen, Zhengyuan Liu, Diyi Yang
  • Can Large Language Models Transform Computational Social Science?

    Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, Diyi Yang

2022

  • Will AI Console Me when I Lose my Pet? Understanding Perceptions of AI-Mediated Email Writing

    Yihe Liu, Anushk Mittal, Diyi Yang, Amy Bruckman
  • VALUE: Understanding Dialect Disparity in NLU

    Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, Diyi Yang
  • TreeMix: Compositional Constituency-based Data Augmentation for Natural Language Understanding

    Le Zhang, Zichao Yang, Diyi Yang
  • Robustness of Demonstration-based Learning Under Limited Data Scenario

    Hongxin Zhang, Yanzhe Zhang, Ruiyi Zhang, Diyi Yang
  • Pretty Princess vs. Successful Leader: Gender Roles in Greeting Card Messages

    Jiao Sun, Tongshuang Wu, Yue Jiang, Ronil Awalegaonkar, Xi Victoria Lin, Diyi Yang
  • Modeling Motivational Interviewing Strategies On Online Peer-to-Peer Counseling Platforms

    Raj Sanjay Shah, Faye Holt, Shirley Anugrah Hayati, Aastha Agrawal, Yi-Chia Wang, Robert Kraut, Diyi Yang
  • Inducing Positive Perspectives with Text Reframing

    Caleb Ziems, Minzhi Li, Anthony Zhang, Diyi Yang
  • GNN is a Counter? Revisiting GNN for Question Answering

    Kuan Wang, Yuyu Zhang, Diyi Yang, Le Song, Tao Qin
  • Geographic Citation Gaps in NLP Research

    Mukund Rungta, Janvijay Singh, Saif M. Mohammad, Diyi Yang
  • Continual Sequence Generation with Adaptive Compositional Modules

    Yanzhe Zhang, Xuezhi Wang, Diyi Yang

2021

  • Understanding the Usage of Online Media for Parenting from Infancy to Preschool At Scale

    Yujia Gao, Jinu Jang, Diyi Yang
  • To Protect and To Serve? Analyzing Entity-Centric Framing of Police Violence

    Caleb Ziems, Diyi Yang
  • The Importance of Modeling Social Factors of Language: Theory and Practice

    Dirk Hovy, Diyi Yang
  • Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs

    Jiaao Chen, Diyi Yang
  • Simple Conversational Data Augmentation for Semi-supervised Abstractive Dialogue Summarization

    Jiaao Chen, Diyi Yang
  • RECAST: Enabling User Recourse and Interpretability of Toxicity Detection Models with Interactive Visualization

    Austin P Wright, Omar Shaikh, Haekyu Park, Will Epperson, Muhammed Ahmed, Stephane Pinel, Duen Horng (Polo) Chau, Diyi Yang
  • Putting Humans in the Natural Language Processing Loop: A Survey

    Zijie Wang, Dongjin Choi, Shenyu Xu, Diyi Yang
  • Latent Hatred: A Benchmark for Understanding Implicit Hate Speech

    Mai ElSherief*, Caleb Ziems*, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, Diyi Yang
  • Evaluating the Effectiveness of Deplatforming as a Moderation Strategy on Twitter

    Shagun Jhaver, Christian Boylston, Diyi Yang, Amy Bruckman
  • Continual Learning for Text Classification with Information Disentanglement Based Regularization

    Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang and Diyi Yang

2020

  • This is a Problem, Don’t You Agree? Framing and Bias in Human Evaluation for Natural Language Generation

    Stephanie Schoch, Diyi Yang, Yangfeng Ji
  • Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization

    Jiaao Chen, Diyi Yang
  • MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

    Jiaao Chen, Zichao Yang, Diyi Yang
  • Characterizing Collective Attention via Descriptor Context: A Case Study of Public Discussions of Crisis Events

    Ian Stewart, Diyi Yang, Jacob Eisenstein
  • Automatically Neutralizing Subjective Bias in Text

    Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang

2019 and Before

  • The Channel Matters: Self-disclosure, Reciprocity and Social Support in Online Cancer Support Groups

    Diyi Yang, Zheng Yao, Joseph Seering, Robert Kraut
  • Seekers, Providers, Welcomers, and Storytellers: Modeling Social Roles in Online Health Communities

    Diyi Yang, Robert Kraut, Tenbroeck Smith, Elijah Mayfield, Dan Jurafsky
  • Modeling Persuasive Strategies via Semi-Supervised Neural Nets on Crowdfunding Platforms

    Diyi Yang*, Jiaao Chen*, Zichao Yang, Dan Jurafsky, Eduard Hovy
  • Identifying Semantic Edit Intentions from Revisions in Wikipedia

    Diyi Yang, Aaron Halfaker, Robert Kraut, Eduard Hovy
  • Commitment of Newcomers and Old-timers to Online Health Support Communities

    Diyi Yang, Robert Kraut, John Levine
  • Who does What: Editor Role Identification in Wikipedia

    Diyi Yang, Aaron Halfaker, Robert Kraut, Eduard Hovy
  • Hierarchical Attention Networks for Document Classification

    Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, Eduard Hovy