Agent-SafetyBench: Evaluating the Safety of LLM Agents

Agent safety failures can arise from generated content, incorrect actions, missing actions, or misplaced trust in tool outputs. Existing evaluations often emphasize one type of attack or one application, making it difficult to compare risks across different interaction patterns.

The benchmark distinguishes the harm that may occur from the behavioral failure that produces it. A single risk can have several causes, and one failure mode can matter in multiple domains. It also separates useful action from apparent safety obtained by declining every request.

A taxonomy connecting risk categories with anticipated failure modes in tool use and generated content.

A collection of configurable, implemented test environments and cases built by refining existing datasets and augmenting underrepresented scenarios.

A fine-tuned safety scorer trained on human-labeled interaction records, plus analyses of failure modes, helpfulness, and defensive prompting.

Each case specifies a user instruction or existing dialog, its risk category, relevant environments and initialization parameters, and expected failure modes. Each environment pairs a JSON tool schema with a Python implementation. This supports repeatable interactions with concrete tool behavior rather than relying on a language model to invent every tool response.

Construction starts by clarifying and deduplicating cases from earlier datasets and standardizing their environments. Additional cases are generated with a designated risk, environment context, and an expected risky behavior sequence. Authors review the cases, scripts check agreement between schemas and implementations, and trial trajectories receive manual safety labels.

The scorer is fine-tuned from Qwen2.5-7B-Instruct using those human labels and model-generated explanations. During evaluation, the acting agent repeatedly chooses a tool and arguments or produces a final answer. The completed interaction record is then classified as safe or unsafe, with an accompanying analysis; this judgment is separate from the acting agent.

Experiments compare language-model agents across the full case collection, reporting safety classifications overall, by risk category, and for behavior-level versus content-only cases. Scorer validation uses manually labeled held-out interaction records. Failure-mode analysis examines which behavioral mechanisms account for unsafe trajectories.

A separate helpfulness analysis labels cases according to whether they can be completed safely and asks whether the agent's behavior advances the task, independently of its safety judgment. Defensive-prompt comparisons contrast a concise failure-mode list with a more detailed version containing examples. These analyses investigate risk awareness, robustness, and refusal behavior within the benchmark's environments and scoring protocol.