Specification design is a critical question in AI because specifications are prone to overspecification, underspecification, and misspecification. We will discuss how experts write reward functions for reinforcement learning (RL) and how non-expert provide preferences for reinforcement learning from human feedback (RLHF). I will show evidence that experts are bad at writing reward functions: even in a trivial setting, experts write specifications that are overfit to a particular RL algorithm, and they often write erroneous specifications for agents that fail to encode their true intent. Next, I will show that the common approach to learning a reward function from non-experts in RLHF uses an inductive bias that fails to encode how humans express preferences, and that our proposed bias better encodes human preferences both theoretically and empirically. I will discuss some of our early work in improving specification design. Lastly, I will discuss my work to govern AI systems through my policymaking experiences in the United States Senate.
Serena Booth is an incoming Assistant Professor in Computer Science at Brown University. Serena studies how people write specifications for AI systems and how people assess whether AI systems are successful in learning from their specifications, as well as how to govern these complex systems. Serena was previously a UC Berkeley Simons Institute for the Theory of Computing Law and Society Fellow (2025) and she worked in the U.S. Senate as a AAAS AI Policy Fellow (2023--2025), where she worked on AI policy for the Senate Banking, Housing, and Urban Affairs Committee. Serena received her PhD at MIT CSAIL in 2023. She is also a Rising Star in EECS, an HRI Pioneer, and a CIFAR Azrieli Global Scholar.