AI agents are beginning to move beyond answering questions. They can navigate software, retrieve records, complete workflows and take actions on behalf of users.

That capability is increasingly relevant in Pinellas County. The County is currently seeking a consultant to develop an Artificial Intelligence Strategic Roadmap and Governance Framework. Its own budget documents have already identified potential uses for AI in citizen information, process automation, employee productivity and existing government applications.

As those systems become more capable, St. Petersburg and Pinellas County will need ways to determine whether an AI agent is ready before residents become part of the testing process.

The Latin American digital bank Nubank has deployed AI extensively in customer service and recently published research describing how it tests new AI agents through large-scale simulation before exposing them to actual customers.

Its researchers built a system called Snowglobe that generates synthetic customers with different circumstances, behaviors and objectives. Those simulated customers hold complete conversations with an AI agent.

The environment also simulates the software tools the agent would normally use. If an agent needs to look up an account, change information or perform another action, the request goes to a simulated system rather than Nubank's live infrastructure.

That allows the company to observe the entire workflow, including the decisions the agent makes along the way.

Nubank used the system while developing agents that handle credit card delivery and management issues. Researchers compared the simulated results with thousands of real customer interactions and found that simulation could help identify which agent configurations were likely to perform better in production.

The team ultimately evaluated 29 AI configurations through more than 16,000 simulated conversations. The approach also allowed developers to move through testing and deployment significantly faster than their previous process.

The civic application is easy to imagine in St. Petersburg.

Consider an AI agent designed to help residents navigate permitting.

Before deployment, developers could create thousands of simulated St. Petersburg residents, contractors, property owners and architects. Some would understand the permitting process. Others would have little idea where to begin.

They could give the agent incomplete addresses, incorrect parcel information or conflicting descriptions of a project. A simulated permitting database could contain missing records or inconsistent information. Some users could ask the agent to perform actions outside its authority.

The city could then measure what happens.

Did the agent provide accurate information? Did it follow city policy? Did it access the correct records? Did it protect information that should remain private? Did it recognize when a city employee needed to take over?

The same testing model could apply throughout Tampa Bay.

A nonprofit intake agent could encounter thousands of simulated clients before handling a real case. A school district agent could work through student and parent scenarios. A Pinellas County service agent could encounter residents trying to navigate benefits, records, transportation or emergency assistance.

Simulation will never reproduce every person or circumstance an AI system will encounter. Nubank's own researchers found differences between synthetic and real conversations. The value comes from finding predictable failures before deployment and comparing different versions of an agent under the same conditions.

That becomes especially important as AI gains permission to do more than generate text.

The procurement test

The most immediate use in Pinellas County is not a city-built agent. It is the roadmap the County is about to commission, and the vendor products that will follow it.

The County could require every vendor proposing an AI agent to run it against the same set of civic scenarios and show how it behaves when information is incomplete, systems fail or a user asks for something they should not be allowed to do. A vendor that cannot produce that evidence has not finished building the product. A vendor that can has given the County something a demo never will: a comparable record of failures, found before a resident found them.

A Practical Model for a Civic Agent Test Range

A civic AI system should have the opportunity to fail thousands of times in a controlled environment before its first failure involves a real resident. AICOE could establish a Civic Agent Test Range for AI systems intended for use in St. Petersburg and across Pinellas County.

Define the job.
Testing begins by stating exactly what an agent is supposed to do, which systems it can access and which actions it has permission to take.

Build the population.
A synthetic population is built around the service. For permitting, that includes homeowners, contractors, landlords and residents unfamiliar with city terminology.

Simulate the systems.
Simulated versions of government systems respond to the agent without exposing live databases or resident information.

Run it at volume.
Thousands of interactions run automatically, each evaluated for accuracy, policy compliance, privacy, successful task completion, appropriate human escalation and a complete audit trail.

Apply it to procurement.
Vendors run their systems against the same civic scenarios before the County buys, and the results travel with the contract.

Nubank developed this approach for financial services, where mistakes carry immediate consequences. St. Petersburg faces its own version of that challenge as AI moves into public-facing systems.