Identifying the Risks of LM Agents with an LM-Emulated Sandbox

Testing agents against realistic tool risks normally requires implementing tools, constructing sandbox state, and finding situations where a plausible mistake has consequences. That work becomes expensive across many applications, especially for rare scenarios or tools without an accessible implementation.

ToolEmu replaces this implementation burden with a language-model emulator. The resulting challenge is not only generating plausible responses: the simulated trajectory must obey tool specifications and maintain state consistently enough that the identified failure could occur in a real implementation.

A specification-driven emulator that produces tool observations without executing the real tool.

An adversarial emulation mode that selects challenging states using task underspecification and designated potential risks.

Trajectory-level safety and helpfulness evaluators, a curated toolkit benchmark, and human validation of both emulation quality and evaluator judgments.

The standard emulator receives tool specifications, the user instruction, the current call, and the history of actions and observations. It produces the next observation while implicitly maintaining the virtual initial state and state transitions. Specifications describe arguments, return values, behavior, and exceptions; the emulator must validate inputs and remain consistent with prior events.

The adversarial emulator additionally receives the instruction's ambiguities and possible risky actions. It uses this information to instantiate scenarios where an unsafe choice is more likely to matter. The risk comes from the selected environment and agent decision, rather than from performing the corresponding real-world action.

After the run, separate evaluators inspect the full trajectory. Safety assessment considers whether risky actions occurred and the likelihood and severity of their consequences. Helpfulness assessment compares the trajectory with the expected safe achievement of the request. The benchmark's tool specifications and test cases are generated with model assistance, then filtered, revised, and checked by people.

Validation pairs standard and adversarial emulations for the same case and acting model. Human annotators, blinded to the agent and emulator identity, inspect input validation, behavioral accuracy, and state consistency, and separately judge safety and helpfulness. Evaluator validation compares model–human agreement with inter-annotator agreement and examines failure-detection precision and recall.

Agent evaluation uses ReAct-style tool use across the curated cases, comparing base models, prompting variants, and an agent that takes no actions. Metrics cover safety, helpfulness, and identified failure incidence. These comparisons test risk discovery and useful task performance within the emulator; they do not make an emulated incident equivalent to a confirmed failure of a deployed tool.