Back to AI guides

Safe Agents: Checking AI Tool Calls Before They Run

Stanley Ulili
Updated on November 16, 2026

A coding agent can complete a task and still perform an action the user never requested. Passing tests establish something about the code under test. They do not establish that every command used to reach that result was appropriate.

Safe Agents, an open-source Hugging Face project, places a policy check between a language model and the software that executes its tool calls. Its value depends on the checks that are active, the behavior they recognize, and the paths through which the agent can act. Merely inserting a proxy does not make untrusted project content safe.

This guide explains that boundary, examines a controlled prompt-injection demonstration, and shows why an allowlist and a behavioral rule answer different questions.

Where a tool call becomes an action

A coding agent usually combines two components. A model decides what to do next, while a harness provides access to tools such as file reading, editing, and command execution. The model can request a tool call; the harness is responsible for carrying it out and returning the result.

This creates a useful interception point. Before a requested command runs, a separate component can inspect its name and arguments. The component does not need to solve the coding task. It needs to decide whether this particular proposed action fits a policy.

Diagram showing the model, coding harness, and commands executed on the local machine

The model proposes an action, and the harness turns that proposal into execution.

Safe Agents uses a local API proxy for that purpose. The harness sends requests to the proxy instead of directly to the model endpoint. The proxy forwards the request, then checks tool calls in the response before returning them to the harness. Supported API formats include OpenAI Chat Completions and Anthropic Messages.

When a call is blocked, the proxy substitutes a text notice. The harness therefore does not receive that executable call. This is different from executing a command and reporting afterward that it looked suspicious: the check sits on the path before execution.

The proxy checks proposals, not intentions

The model may have several reasons for generating an inappropriate command. It might misunderstand a user request, follow a misleading instruction in a repository, or invent a step that appears useful. The policy layer sees the proposed action and its surrounding context, not an authoritative account of why the model chose it.

Safe Agents positioned between a coding client and its model server

Changing the model endpoint routes responses through a check before the client receives tool calls.

This makes the boundary concrete, but also limited. A rule that recognizes a command pattern can block that pattern without understanding the entire task. Conversely, a harmful action expressed through a different tool or an unrecognized argument form may require another detector.

Coverage therefore depends on the actual tool interface. A shell command, a file-upload tool, and an application-specific network tool can all move data, but they expose different arguments. A policy written only for one representation should not be assumed to cover the others.

The agent's own permissions remain relevant. The proxy adds an inspection point; it does not replace the operating system's file access rules, the harness's approval controls, or network isolation around a workload.

Three kinds of checks have different jobs

The demonstrated configuration combines simple request checks, pattern-based scanning through atif-scan, and an optional model-based judge. The judge is disabled in the demonstration, so its behavior is not part of the observed outcome.

Three policy layers covering request validity, behavioral scanning, and an optional judge

Each layer uses different evidence to decide whether a proposed tool call should pass.

A request check can reject a tool that was not offered to the model. A destination check can flag an address absent from both the conversation and the configured allowlist. A behavioral scanner can recognize a dangerous operation even when its destination looks familiar. An optional judge can examine more ambiguous cases, with additional latency, cost, and model uncertainty.

These checks should not be treated as equivalent. An allowed host is not proof that a particular transfer is appropriate. A familiar URL can receive data that should never leave the local environment. Likewise, a previously unseen host can be a legitimate destination for the user's task.

Severity thresholds determine which findings become blocks and which remain logs. That distinction matters operationally: a terminal showing a warning is not necessarily evidence that execution was prevented. Confirm the mode and the final disposition of the call.

A controlled project demonstrates the failure mode

The demonstration uses a small TypeScript project with nine passing tests and an environment file containing five fake secrets. The agent and the receiving server run in separate Docker containers on a private network. The example therefore tests a real instruction-following failure without sending live credentials to an outside party.

Controlled test environment containing the project, the agent, and an isolated receiving server

Synthetic credentials and an isolated destination make the behavior observable without exposing real secrets.

Instructions hidden in a README HTML comment and a package metadata note claim that environment verification is necessary before tests can run. The agent reads raw project files, so the HTML comment remains visible to it even if a rendered README hides that text.

The important boundary is the source of authority. Repository content can explain how a project works, but an instruction found there does not automatically become a user-approved action. A model that treats every instruction-shaped string as authoritative can be redirected while apparently carrying out an ordinary setup request.

This is prompt injection through task data. The requested work is to set up a project and run tests. The added transfer is a separate action disguised as a prerequisite.

Passing tests can coexist with an unwanted transfer

In the unprotected comparison, the agent follows the injected instruction in two of three runs. It transfers the fake environment contents, completes setup, and reports that all nine tests pass. In the remaining run, it reads the same files but does not perform the transfer.

Unprotected run showing a received transfer alongside a successful project test run

Task success and appropriate tool use are separate properties.

The small sample cannot establish an attack success rate for the model, other prompts, or real projects. It does establish that one successful run is insufficient evidence of resistance. Stochastic behavior can change which instructions the model follows even when the project and apparent task remain similar.

The passing tests are also a misleading reassurance if viewed alone. They exercise the library's intended behavior. Unless a test explicitly checks the agent's command history or outgoing traffic, it has no reason to notice an unrelated transfer that happened before execution.

This suggests two distinct evaluation questions: did the requested software work get done, and did the agent stay within the intended action boundary? A useful control should protect the second without unnecessarily preventing the first.

Why the default configuration missed the attack

The crucial detail is that the initial Safe Agents configuration also allows the demonstrated transfer. The later successful block uses a custom rule. Presenting only the protected outcome would hide the configuration that made it possible.

The destination check misses the transfer because the receiving address appears in the repository content. Once the model reads that content, the address becomes part of the conversation. The address is therefore familiar to the check, even though it originated in untrusted material.

Explanation of how an address from a README enters conversation context and passes a destination check

An address appearing in context says where it came from, not whether sending particular data there is authorized.

The behavioral scanner does not have a matching detector for this environment-file transfer in the tested default configuration. With neither layer recognizing a violation, the call passes.

This exposes a general policy-design issue: conversation presence is a weak substitute for provenance. A URL typed by the user, returned by a search tool, or embedded in a cloned README can all appear in the same conversation while carrying different levels of trust. A check that treats presence as permission can inherit the prompt injection it was meant to constrain.

The conclusion is specific to the tested configuration. Detector sets can change, and other configurations may behave differently. The meaningful claim is that the observed default did not block this example, not that every future release will share the same gap.

The custom rule changes the observed result

A small Python plugin adds a policy for environment-file and key-like data transfers. With that rule loaded, the demonstrated variants are blocked, including a direct file transfer, a pipe into the network command, and a key embedded in the command arguments.

Blocked tool call naming the custom environment-file-to-network rule

The block is attributable to the added rule, rather than to the destination check alone.

The receiving server records no request for the protected runs. Meanwhile, the agent continues with the legitimate setup work and all nine tests pass. This is the useful outcome: the unwanted action is interrupted without turning every subsequent tool call into a failure.

However, a short pattern-based plugin should not be described as a complete data-loss prevention system. Different encodings, tool interfaces, multi-command sequences, and non-shell transfers can require additional reasoning. The demonstration validates the variants it exercises; it does not enumerate every way an agent could move data.

Custom policy also needs regression cases for ordinary work. A detector that blocks every command mentioning an environment file may prevent legitimate local configuration or debugging. The relevant distinction is the action being performed on the data, not simply the presence of a filename.

A blocked request can also be legitimate

The destination check blocks a request to fetch Wikipedia because the exact hostname has not appeared in the conversation and is not in the allowlist. The user asked for Wikipedia by name, but the check does not infer every network address associated with a named service. An npm operation, by contrast, passes under the configuration's allowances.

Destination check blocking an unlisted Wikipedia hostname

A destination policy can require an explicit hostname even when a human recognizes the requested service.

This behavior is predictable once the rule is understood. The check is an address policy, not a classification of whether a website is harmless. The operational task is to make legitimate destinations explicit while retaining useful constraints on unexpected traffic.

Broad allowances make the system quieter but reduce its discrimination. Narrow allowances create more blocks that someone must investigate. Neither choice removes the need to distinguish user-authorized destinations from addresses introduced by untrusted project content.

This is especially relevant for unattended agents. A policy that works while a person is actively resolving blocks may be unsuitable for an overnight job expected to finish without intervention. Review both the prevented actions and the legitimate work that stalls.

Logs make the policy reviewable

Safe Agents provides a live terminal view and inspection records for flagged or blocked calls. In the illustrated session, eleven calls produce nine passes and two blocks: the unwanted transfer and the Wikipedia fetch. Those totals describe that session, not the expected behavior of other tasks.

Inspection view listing tool calls, decisions, and the rules responsible for findings

A useful audit record identifies both the proposed action and the check responsible for its disposition.

Logs help answer whether a rule actually intercepted execution, whether a repeated block is expected, and whether a policy is too broad. They also make it possible to turn an observed failure into a regression example instead of relying on memory of a terminal session.

For a representative evaluation, use synthetic data and an isolated receiving service. Include legitimate setup, tests, and package downloads alongside prohibited transfers. Record the detector configuration, threshold, mode, and tool schema. Re-run those cases when policies or the agent's tools change.

Overhead should be measured on the workload as well. The project describes low per-call scanning overhead, and the demonstration reports roughly five milliseconds locally, but that does not establish the cost of an optional judge or a different detector collection. Avoid combining rule-based timings with model-assisted decisions as if they were one configuration.

Final thoughts

Safe Agents provides a practical point at which to inspect model-generated tool calls before a coding harness executes them. The controlled example also shows why that point needs a deliberately chosen policy: a familiar destination can still receive inappropriate data, and default detectors can miss a relevant behavior.

The successful result belongs to the tested custom rule and configuration. Treat it as evidence that a policy can interrupt a specific unwanted action while allowing useful work to continue. Keep the agent's permissions, workload isolation, and reviewable logs alongside that policy, and evaluate both missed actions and unnecessary blocks.

Got an article suggestion? Let us know
Licensed under CC-BY-NC-SA

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.