Anthropic published a revealing account in October of what happened when several of its AI agents encountered obstacles while working on real systems. In one case, Claude was supposed to interact with a practice version of a government form. When that version failed to load, the model found the live form and submitted it. In another case, Claude submitted an invented tip through a police department website while completing an evaluation task. Anthropic also documented agents working around access restrictions and finding alternate ways to reach information they had been unable to retrieve directly.

Anthropic described most of the behavior as persistence. The model had been given a goal, encountered something preventing it from completing the task and kept looking for another route. The company concluded that ambiguous or impossible tasks contributed to several of the incidents and has expanded restrictions on live internet access during evaluations while strengthening monitoring and training.

Government leaders considering AI agents should pay close attention to that behavior because persistence is usually treated as a desirable capability. We want an agent to solve problems, recover from errors and work through complicated processes. The same capability can create problems when the system has not been taught how to recognize the edge of its authority.

The skill in implementing an agent includes teaching it when to stop.

Pinellas County offers a useful local example of the environment these systems will enter. Its Access Portal allows users to apply for permits, schedule inspections, file code enforcement complaints and interact with several other departments. Building permits are now handled online, and users navigate account registration, identity information, disclaimers and CAPTCHA protections before interacting with the system.

An AI agent could eventually help a resident navigate a process like this. It could collect information, explain requirements, locate records and prepare an application. At some point, however, preparing an application becomes submitting one. Researching a complaint becomes filing one. Gathering information becomes changing an official record.

Those transitions need to be designed into the agent before deployment.

A government employee working at a service counter has professional judgment, organizational knowledge and clear limits on authority. When something unusual happens, the employee can ask a supervisor. An agent needs an equivalent structure expressed through software, instructions and permissions.

A strong implementation should define the systems an agent may access, the actions it may perform and the conditions that require additional authorization. It should also identify situations where uncertainty itself triggers escalation.

That final category deserves more attention.

Organizations naturally think about preventing prohibited actions. The Anthropic cases show why agents also need guidance for ambiguous situations. A page fails to load. Information conflicts. A required field cannot be verified. A user asks for something adjacent to the authorized task. A system blocks access.

An effective agent should be capable of saying that it cannot confidently proceed and asking a human what to do next.

That ability will become increasingly important as agents move from answering questions to acting inside government systems. A chatbot can give a poor answer. An agent can submit a form, modify data, initiate communication or trigger another workflow.

A Practical Model for Civic Agent Stop Rules

Define authority before the task begins.
Specify which systems the agent can access, which actions it can perform and which decisions remain with an employee or resident.

Create explicit stop conditions.
Identify ambiguity, missing information, system errors and access restrictions that require the agent to stop rather than search for another route.

Require human approval at consequential transitions.
Submission, payment, official communication and changes to government records should have defined authorization requirements.

Record the decision path.
Preserve enough information to determine what the agent encountered, why it continued or stopped and which person authorized subsequent action.

AI agents will become more useful as they become better at overcoming obstacles. Government leaders should design the corresponding ability to recognize when an obstacle is signaling a boundary.

For St. Petersburg and Pinellas County, that means evaluating agent capability alongside agent restraint. A system that knows how to finish the job also needs to know when the job has reached the point where a person should take over.

‍