Recent Preprints

• Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforce

  Yijia Shao*, Humishka Zope*, Yucheng Jiang, Jiaxin Pei, David Nguyen, Erik Brynjolfsson, Diyi Yang
  arXiv:2506.06576 [Paper] [Project Page]

• Can LLM-Simulated Practice and Feedback Upskill Human Counselors? A Randomized Study with 90+ Novice Counselors

  Ryan Louie, Ifdita Hasan Orney, Juan Pablo Pacheco, Raj Sanjay Shah, Emma Brunskill, Diyi Yang
  arXiv:2505.02428 [Paper]

• Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration

  Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, Diyi Yang
  arXiv:2412.15701 [Paper] [Code] [Project Page]

• Optimizing Pretraining Data Mixtures with LLM-Estimated Utility

  William Held, Bhargavi Paranjape, Punit Singh Koura, Mike Lewis, Frank Zhang, Todor Mihaylov
  arXiv:2501.11747 [Paper]

• Social Skill Training with Large Language Models

  Diyi Yang*, Caleb Ziems*, William Held*, Omar Shaikh*, Michael S. Bernstein, John Mitchell
  arXiv:2404.04204 [Paper]

2025

• Blackbox Model Provenance via Palimpsestic Membership Inference

  Rohith Kuditipudi, Jing Huang, Sally Zhu, Percy Liang, Christopher Potts, Diyi Yang
  NeurIPS 2025 🏆 Spotlight

• SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs

  Michael Ryan, Omar Shaikh, Aditri Bhagirath, Daniel Frees, William Held, Diyi Yang
  ACL 2025 [Paper] [Code] [Project Page]

• SPHERE: An Evaluation Card for Human-AI Systems

  Dora Zhao*, Qianou Ma*, Xinran Zhao, Chenglei Si, Chenyang Yang, Ryan Louie, Ehud Reiter, Diyi Yang*, Tongshuang Wu*
  ACL 2025 Findings [Paper]

• Mind the Gap: Static and Interactive Evaluations of Large Audio Models

  Ella Li*, William Held*, Michael Ryan, Kunat Pipatanakul, Potsawee Manakul, Hao Zhu, Diyi Yang
  ACL 2025 [Paper]

• EgoNormia: Benchmarking Physical Social Norm Understanding

  MohammadHossein Rezaei*, Yicheng Fu*, Phil Cuvin*, Caleb Ziems, Yanzhe Zhang, Hao Zhu, Diyi Yang
  ACL 2025 Findings [Paper]

• Distilling an End-to-End Voice Assistant Without Instruction Training Data

  William Held, Yanzhe Zhang, Weiyan Shi, Ella Li, Michael Ryan, Diyi Yang
  ACL 2025 [Paper] [Code] [Project Page]

• Attacking Vision-Language Computer Agents via Pop-ups

  Yanzhe Zhang, Tao Yu, Diyi Yang
  ACL 2025 [Paper] [Code]

• Estimating the Correctness of Language Model Predictions from Internal Causal Mechanisms

  Jing Huang*, Junyi Tao*, Thomas Icard, Diyi Yang, Christopher Potts
  ICML 2025 [Paper]

• SWE-smith: Scaling Data for Software Engineering Agents

  John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, Diyi Yang
  NeurIPS 2025 Dataset and Benchmarks Track [Paper] [Code] [Project Page] 🏆 Spotlight

• SWE-bench Multimodal: Do Autonomous Programming Systems Generalize to New Software Domains

  John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, Diyi Yang, Sida Wang, Ofir Press
  ICLR 2025 [Paper] [Project Page]

• No Preference Left Behind: Group Distributional Preference Optimization

  Binwei Yao, Zefan Cai, Yun-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, Junjie Hu
  ICLR 2025 [Paper] [Code]

• Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers

  Chenglei Si, Diyi Yang, Tatsunori Hashimoto
  ICLR 2025 [Paper] [Code]

• Aligning Language Models with Demonstrated Feedback

  Omar Shaikh*, Michelle Lam*, Joey Hejna*, Yijia Shao, Hyundong Justin Cho, Michael Bernstein, Diyi Yang
  ICLR 2025 [Paper] [Code]

• Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping

  Ryan Li, Yanzhe Zhang, Diyi Yang
  NAACL 2025 [Paper] [Code] [Project Page]

• Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering

  Chenglei Si*, Yanzhe Zhang*, Ryan Li, Zhengyuan Yang, Ruibo Liu, Diyi Yang
  NAACL 2025 [Paper] [Code] [Project Page]

2024

• Semi-Truths: A Large-Scale Dataset for Testing Robustness of AI-Generated Image Detectors

  Anisha Pal, Julia Kruk, Mansi Phute, Manognya Bhattaram, Diyi Yang, Duen Horng Chau, Judy Hoffman
  NeurIPS 2024 Dataset and Benchmarks Track

• PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action

  Yijia Shao, Tianshi Li, Weiyan Shi, Yanchen Liu, Diyi Yang
  NeurIPS 2024 Dataset and Benchmarks Track [Paper] [Code] [Project Page]

• DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph

  Zhehao Zhang, Jiaao Chen, Diyi Yang
  NeurIPS 2024 [Paper] [Code]

• Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles

  Ryan Louie, Ananjan Nandi, William Fang, Cheng Chang, Emma Brunskill, Diyi Yang
  EMNLP 2024 [Paper] [Project Page]

• Modeling Gender and Dialect Bias in Automatic Speech Recognition

  Camille Harris, Chiji Mgbahurike, Neha Kumar, Diyi Yang
  EMNLP 2024 (Findings)

• Demystifying Verbatim Memorization in Large Language Models

  Jing Huang, Diyi Yang, Christopher Potts
  EMNLP 2024 [Paper]

• Decoding Susceptibility: Modeling Misbelief to Misinformation Through a Computational Approach

  Yanchen Liu, Mingyu Derek Ma, Wenna Qin, Azure Zhou, Jiaao Chen, Weiyan Shi, Wei Wang, Diyi Yang
  EMNLP 2024 [Paper]

• CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies

  Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Chunhua Yu, Raya Horesh, Rogério Abreu de Paula, Diyi Yang
  EMNLP 2024 (Findings) [Paper] [Code] [Project Page]

• Benchmarking LLM-based Machine Translation on Cultural Awareness

  Binwei Yao, Ming Jiang, Diyi Yang, Junjie Hu
  EMNLP 2024 (Findings) [Paper]

• Are Large Language Models Consistent over Value-laden Questions?

  Jared Moore, Tanvi Deshpande, Diyi Yang
  EMNLP 2024 (Findings)

• The Practice of Online Peer Counseling and the Potential for AI-Powered Support Tools

  Tony Wang, Amy Bruckman, Diyi Yang
  CSCW 2024

• Helping the Helper: Supporting Peer Counselors via AI-Empowered Practice and Feedback

  Shang-Ling Hsu, Raj Sanjay Shah, Prathik Senthil, Zahra Ashktorab, Casey Dugan, Werner Geyer, Diyi Yang
  CSCW 2024 [Paper]

• Dynamic LLM-Agent Network: An LLM-agent Collaboration Framework with Agent Team Optimization

  Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, Diyi Yang
  COLM 2024 [Paper]

• Auditing Gender Presentation Differences in Text-to-Image Models

  Yanzhe Zhang, Lu Jiang, Greg Turk, Diyi Yang
  EAAMO 2024 [Paper] [Code] [Project Page]

• Simulated Misinformation Susceptibility (SMISTS): Enhancing Misinformation Research with Large Language Model Simulations

  Weicheng Ma, Chunyuan Deng, Aram Moossavi, Lili Wang, Soroush Vosoughi, Diyi Yang
  ACL 2024 (Findings)

• Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistles

  Julia Kruk, Michela Marchini, Rijul Ragu, Caleb Ziems, David Muchlinski, Diyi Yang
  ACL 2024 [Paper] [Dataset]

• Perceptions of Language Technology Failures from South Asian English Speakers

  Faye Holt*, William Held*, Diyi Yang
  ACL 2024 (Findings)

• Measuring and Addressing Indexical Bias in Information Retrieval

  Caleb Ziems, William Held, Jane Dwivedi-Yu, Diyi Yang
  ACL 2024 (Findings) [Paper] [Code]

• Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

  Matthias Gerstgrasser*, Rylan Schaeffer*, Apratim Dey*, Rafael Rafailov*, Dhruv Pai, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Andrey Gromov, Daniel A. Roberts, Diyi Yang, David L. Donoho, Sanmi Koyejo
  COLM 2024 [Paper]

• Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselors

  Alicja Chaszczewicz, Raj Sanjay Shah, Ryan Louie, Bruce A Arnow, Robert Kraut, Diyi Yang
  ACL 2024 [Paper]

• Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Future

  Minzhi Li, Weiyan Shi, Caleb Ziems, Diyi Yang
  ACL 2024 (Findings) [Paper]

• Unintended Impacts of LLM Alignment on Global Representation

  Michael Ryan, William Held, Diyi Yang
  ACL 2024 [Paper]

• Rehearsal: Simulating Conflict to Teach Conflict Resolution

  Omar Shaikh, Valentino Chai, Michele J. Gelfand, Diyi Yang, Michael S. Bernstein
  CHI 2024 [Paper]

• Training Socially Aligned Language Models on Simulated Social Interactions

  Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M. Dai, Diyi Yang, Soroush Vosoughi
  ICLR 2024 [Paper]

• DyVal: Graph-informed Dynamic Evaluation of Large Language Models

  Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, Xing Xie
  ICLR 2024 [Paper]

• What Makes Digital Support Effective? How Therapeutic Skills Affect Clinical Well-Being

  Anna Fang, Wenjie Yang, Raj Sanjay Shah, Yash Mathur, Diyi Yang, Haiyi Zhu, Robert Kraut
  CSCW 2024 [Paper]

• Anchor Points: Benchmarking Models with Much Fewer Examples

  Rajan Vivek, Kawin Ethayarajh, Diyi Yang, Douwe Kiela
  EACL 2024 [Paper]

2023

• LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

  Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, Tong Sun
  NeurIPS Instruction Workshop 2023 [Paper]

• Using Large Language Models in Psychology

  Dorottya Demszky*, Diyi Yang*, David S. Yeager*, Christopher J. Bryan, Margarett Clapper, Susannah Chandhok, Johannes C. Eichstaedt, Cameron Hecht, Jeremy Jamieson, Meghann Johnson, Michaela Jones, Danielle Krettek-Cobb, Leslie Lai, Nirel JonesMitchell, Desmond C. Ong, Carol S. Dweck, James J. Gross, James W. Pennebaker
  Nature Reviews Psychology 2023 [Paper]

• Unlearn What You Want to Forget: Efficient Unlearning for LLMs

  Jiaao Chen, Diyi Yang
  EMNLP 2023 [Paper]

• Understanding Black Content Creator Experiences on TikTok

  Camille Harris, Amber Gayle Johnson, Sadie Palmer, Diyi Yang, Amy Bruckman
  CSCW 2023 [Paper] 🏆 Recognition for Contribution to Diversity and Inclusion Award

• Task Agnostic Dialect Adapters for English

  William Held, Caleb Ziems, Diyi Yang
  ACL 2023 (Findings) [Paper]

• Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers

  William Held, Diyi Yang
  EACL 2023 [Paper]

• Parameter-Efficient Fine-Tuning Design Spaces

  Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, Diyi Yang
  ICLR 2023 [Paper]

• On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning

  Omar Shaikh, Hongxin Zhang, William Held, Michael S. Bernstein, Diyi Yang
  ACL 2023 [Paper]

• NormBank: A Knowledge Bank of Situational Social Norms

  Caleb Ziems, Jane Dwivedi-Yu, Yi-Chia Wang, Alon Halevy, Diyi Yang
  ACL 2023 [Paper]

• Multi-VALUE: A Framework for Cross-Dialectal English NLP

  Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, Diyi Yang
  ACL 2023 [Paper]

• Modeling Cross-Cultural Pragmatic Inference with Codenames Duet

  Omar Shaikh, Caleb Ziems, William Held, Aryan J. Pariani, Fred Morstatter, Diyi Yang
  ACL 2023 (Findings) [Paper]

• Impressions: Visual Semiotics and Aesthetic Impact Understanding

  Julia Kruk, Caleb Ziems, Diyi Yang
  EMNLP 2023 [Paper]

• Human-in-the-loop Abstractive Dialogue Summarization

  Jiaao Chen, Mohan Dodda, Diyi Yang
  ACL 2023 (Findings) [Paper]

• Forgotten Knowledge: Examining the Citational Amnesia in NLP

  Janvijay Singh, Mukund Rungta, Diyi Yang, Saif M. Mohammad
  ACL 2023 [Paper] 🏆 Best Paper Honorable Mention

• DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules

  Yanchen Liu, William Held, Diyi Yang
  EMNLP 2023 [Paper]

• CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations

  Myra Cheng, Tiziano Piccardi, Diyi Yang
  EMNLP 2023 [Paper]

• CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

  Ella Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy F. Chen, Zhengyuan Liu, Diyi Yang
  EMNLP 2023 [Paper]

• Can Large Language Models Transform Computational Social Science?

  Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, Diyi Yang
  Computational Linguistics 2023 [Paper]

2022

• Will AI Console Me when I Lose my Pet? Understanding Perceptions of AI-Mediated Email Writing

  Yihe Liu, Anushk Mittal, Diyi Yang, Amy Bruckman
  CHI 2022 [Paper]

• VALUE: Understanding Dialect Disparity in NLU

  Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, Diyi Yang
  ACL 2022 [Paper]

• TreeMix: Compositional Constituency-based Data Augmentation for Natural Language Understanding

  Le Zhang, Zichao Yang, Diyi Yang
  NAACL 2022 [Paper]

• Robustness of Demonstration-based Learning Under Limited Data Scenario

  Hongxin Zhang, Yanzhe Zhang, Ruiyi Zhang, Diyi Yang
  EMNLP 2022 [Paper]

• Pretty Princess vs. Successful Leader: Gender Roles in Greeting Card Messages

  Jiao Sun, Tongshuang Wu, Yue Jiang, Ronil Awalegaonkar, Xi Victoria Lin, Diyi Yang
  CHI 2022 [Paper] 🏆 Best Paper Honorable Mention

• Modeling Motivational Interviewing Strategies On Online Peer-to-Peer Counseling Platforms

  Raj Sanjay Shah, Faye Holt, Shirley Anugrah Hayati, Aastha Agrawal, Yi-Chia Wang, Robert Kraut, Diyi Yang
  CSCW 2022 [Paper]

• Inducing Positive Perspectives with Text Reframing

  Caleb Ziems, Minzhi Li, Anthony Zhang, Diyi Yang
  ACL 2022 [Paper] 🏆 Outstanding Paper Award

• GNN is a Counter? Revisiting GNN for Question Answering

  Kuan Wang, Yuyu Zhang, Diyi Yang, Le Song, Tao Qin
  ICLR 2022 [Paper]

• Geographic Citation Gaps in NLP Research

  Mukund Rungta, Janvijay Singh, Saif M. Mohammad, Diyi Yang
  EMNLP 2022 [Paper]

• Continual Sequence Generation with Adaptive Compositional Modules

  Yanzhe Zhang, Xuezhi Wang, Diyi Yang
  ACL 2022 [Paper]

2021

• Understanding the Usage of Online Media for Parenting from Infancy to Preschool At Scale

  Yujia Gao, Jinu Jang, Diyi Yang
  SIGCHI 2021 [Paper] 🏆 Best Paper Honorable Mention

• To Protect and To Serve? Analyzing Entity-Centric Framing of Police Violence

  Caleb Ziems, Diyi Yang
  EMNLP 2021 (Findings) [Paper]

• The Importance of Modeling Social Factors of Language: Theory and Practice

  Dirk Hovy, Diyi Yang
  NAACL 2021 [Paper]

• Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs

  Jiaao Chen, Diyi Yang
  NAACL 2021 [Paper]

• Simple Conversational Data Augmentation for Semi-supervised Abstractive Dialogue Summarization

  Jiaao Chen, Diyi Yang
  EMNLP 2021 [Paper]

• RECAST: Enabling User Recourse and Interpretability of Toxicity Detection Models with Interactive Visualization

  Austin P Wright, Omar Shaikh, Haekyu Park, Will Epperson, Muhammed Ahmed, Stephane Pinel, Duen Horng (Polo) Chau, Diyi Yang
  CSCW 2021 [Paper]

• Putting Humans in the Natural Language Processing Loop: A Survey

  Zijie Wang, Dongjin Choi, Shenyu Xu, Diyi Yang
  HCI+NLP workshop at EACL 2021 [Paper]

• Latent Hatred: A Benchmark for Understanding Implicit Hate Speech

  Mai ElSherief*, Caleb Ziems*, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, Diyi Yang
  EMNLP 2021 [Paper]

• Evaluating the Effectiveness of Deplatforming as a Moderation Strategy on Twitter

  Shagun Jhaver, Christian Boylston, Diyi Yang, Amy Bruckman
  CSCW 2021 [Paper] 🏆 Best Paper Honorable Mention

• Continual Learning for Text Classification with Information Disentanglement Based Regularization

  Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang and Diyi Yang
  NAACL 2021 [Paper]

2020

• This is a Problem, Don’t You Agree? Framing and Bias in Human Evaluation for Natural Language Generation

  Stephanie Schoch, Diyi Yang, Yangfeng Ji
  Workshop on Evaluating NLG Evaluation at INLG 2020 [Paper]

• Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization

  Jiaao Chen, Diyi Yang
  EMNLP 2020 [Paper]

• MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

  Jiaao Chen, Zichao Yang, Diyi Yang
  ACL 2020 [Paper]

• Characterizing Collective Attention via Descriptor Context: A Case Study of Public Discussions of Crisis Events

  Ian Stewart, Diyi Yang, Jacob Eisenstein
  ICWSM 2020 [Paper]

• Automatically Neutralizing Subjective Bias in Text

  Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang
  AAAI 2020 [Paper]

2019 and Before

• The Channel Matters: Self-disclosure, Reciprocity and Social Support in Online Cancer Support Groups

  Diyi Yang, Zheng Yao, Joseph Seering, Robert Kraut
  CHI 2019 [Paper] 🏆 Best Paper Honorable Mention

• Seekers, Providers, Welcomers, and Storytellers: Modeling Social Roles in Online Health Communities

  Diyi Yang, Robert Kraut, Tenbroeck Smith, Elijah Mayfield, Dan Jurafsky
  CHI 2019 [Paper] 🏆 Best Paper Honorable Mention

• Modeling Persuasive Strategies via Semi-Supervised Neural Nets on Crowdfunding Platforms

  Diyi Yang*, Jiaao Chen*, Zichao Yang, Dan Jurafsky, Eduard Hovy
  NAACL 2019 [Paper]

• Identifying Semantic Edit Intentions from Revisions in Wikipedia

  Diyi Yang, Aaron Halfaker, Robert Kraut, Eduard Hovy
  EMNLP 2017 [Paper]

• Commitment of Newcomers and Old-timers to Online Health Support Communities

  Diyi Yang, Robert Kraut, John Levine
  CHI [Paper]

• Who does What: Editor Role Identification in Wikipedia

  Diyi Yang, Aaron Halfaker, Robert Kraut, Eduard Hovy
  ICWSM 2016 [Paper] 🏆 Best Paper Honorable Mention

• Hierarchical Attention Networks for Document Classification

  Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, Eduard Hovy
  NAACL 2016 [Paper]