LURK Werewolf
LURK Werewolf tests whether LLM agents react to what is said or to how it is said. Agents commit to a structured decision before any text is written, so the same game can be replayed with the language switched on or off. Turning those decisions into speech changed 52.6% of what agents did next.
The Problem
Werewolf runs on talk. Agents accuse, defend, vote, rate each other's trustworthiness, and use hidden-role abilities. In most LLM frameworks a single free-form message carries both the decision and its phrasing, so when behavior shifts there is no way to tell whether an agent reacted to different content or to a different way of saying the same thing. Measuring linguistic bias needs those two separated first.
System Design
Every speech starts as a predicate from a 34-item vocabulary that covers public discussion, sheriff procedures, night actions, and system events. A complete game can be played in predicates alone, with no natural language at all. A second model running with deterministic decoding then turns each predicate into the same baseline sentence every time, and an optional rewrite stage varies a single linguistic feature such as formality while the underlying predicate stays fixed. Any behavioral difference between conditions traces back to wording, not content.
Three-stage pipeline
Predicate inventory
Evaluation Method
Two independently generated games drift apart after a few choices, so direct comparison fails. LURK saves a checkpoint at the start of each discussion phase and replays the identical game state under a different language condition. Four replay settings cover predicate-only play, natural language, and rewrites applied to one speaker or to everyone else. The framework then counts how many later skill targets, votes, and reliability ratings changed between the branches.
Results
In the standard seven-player setup, moving from predicate-only play to natural language changed 52.6% of all later decisions. Trust moved most. 49% of reliability judgments shifted, and average trust dropped once agents spoke in prose. The shift was uneven across roles. Werewolves became more trusted when they spoke, while Seer, Guard, and Villagers lost trust, which suggests speech makes deceptive statements more plausible. The effect held in every configuration from eight to twelve players.
Formality Analysis
The second experiment rewrote speeches to be formal or casual across 1,635 replayed phases. The style shift was measurable in the text itself, with longer sentences and fewer contractions, yet no stable behavioral effect appeared. That null keeps the claim precise. The jump happens when structure becomes language, and not every surface change in wording introduces bias.
Citation
@inproceedings{liu2026lurk,
title = {LURK Werewolf: Evaluating Linguistic Bias in Large Language Models},
author = {Liu, Lisa and Zou, Runqi and Kuang, Shuxuan and Li, Yuchen and Merino, Tim and Togelius, Julian},
booktitle = {Proceedings of the IEEE Conference on Games (CoG)},
year = {2026},
note = {To appear}
}