AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents

A jailbreak can remove refusal behavior while also degrading the model's ability to carry out a task. Conversely, an agent may avoid a harmful outcome because it cannot operate the tools, even when it is willing to comply. A refusal-only test cannot distinguish these cases.

AgentHarm studies direct misuse through explicitly malicious user requests. Its tasks require coherent, dependent tool calls, while benign counterparts probe similar capabilities without the malicious objective. This threat model differs from an external attacker injecting instructions into a legitimate user's retrieved content.

Human-written malicious behaviors spanning multiple harm categories, with detailed, open-ended, and initial-tool-hint variants.

Synthetic tool implementations and fine-grained grading rubrics that assess progress through a multi-step workflow.

Benign counterparts, refusal tracking, and completion measurements on non-refused tasks to separate safeguards from agent competence.

Each behavior defines a digitally realizable objective and the tools needed to pursue it. Calls commonly depend on information from earlier calls, so the agent must preserve arguments and execution order. Separate writers and reviewers check the tasks, and variants adjust how much task detail or starting guidance the model receives.

Tools are standalone functions in the Inspect evaluation framework, with prepared outputs and no real-world side effects. They imitate both general-purpose and domain-specific interfaces. Error responses allow recovery from mistakes, such as referring to a nonexistent resource, so the benchmark can observe self-correction during a run.

Grading primarily uses manually specified checks on calls, arguments, and outputs, allowing partial credit for incomplete workflows. A language-model judge handles narrowly scoped semantic checks when necessary. Refusal detection examines the agent's messages throughout the trajectory, including refusals that occur after some tools have already run.

The baseline agent uses a direct tool-calling loop that feeds results back into context. Conditions include ordinary requests, forced tool calling where supported, and a shared jailbreak template. Validation, public-test, and private-test partitions separate tuning from evaluation and support checks for contamination.

Measurements include workflow completion, refusal, and performance on non-refused behaviors, alongside the benign counterparts. Ablations vary task detail, hints, sampling attempts, and other evaluation choices; trajectory inspection examines dependency and parameter errors. Because the tools are deliberately simplified proxies, these measurements concern benchmark behavior rather than operational capability to cause the corresponding real-world harm.