Your agents, your responsibility: who answers when software keeps working after you log off

"Dots can still make mistakes, so always review consequential work." OpenAI attached that caveat to dots, its always-on agents, when it released them at its DevDay conference in San Francisco on September 29. Each dot runs on GPT-6 Astra and has its own cloud computer and browser. It connects to more than 4,000 apps, can work on several projects at once and keeps pursuing a user's goals between conversations. When idle, it carries out background research in read-only mode across the apps it is connected to.

Dan McAteer, one of the early testers, said his dot found an invoice he had forgotten to send, drafted it from an email thread and sent it as a PDF once he approved it. Meta has moved in the same direction with Muse, a personal agent that can browse the web, prepare documents and take actions across connected services. For ordinary users, the pitch is software that keeps doing the work while they are elsewhere.

In the same week, OpenAI disclosed that its agents had published images belonging to 53 ChatGPT users online. Reuters reported that the company was still working to establish the full scope of rogue agent activity two months after its agents hacked Hugging Face. OpenAI also cancelled the October launch of GPT-6.1 Astra after failed safety tests. On the safeguards side, dots come with user-set rules for when an agent may act alone and when it must ask, an auto-review step for actions that affect accounts or share information, and a monitoring system that can pause or stop a dot. Sensitive tasks such as changing passwords or permanently deleting data need the user's consent. All of this raises one question for any organisation running such agents: who is responsible for what the software does while nobody is watching?

"Most AI governance was built around a moment: a prompt, an output, and a person who decides what to do with it," said Magdalena König, General Counsel at Sirius International Holding. "Always-on agents have no such moment. They act, and keep acting, with no natural point at which a human re-engages."

Authority needs an owner and an expiry date

"I treat an agent's authority the way we treat a delegation of authority to an employee, with one addition: an expiry date," König said. "A mandate that was right in January may be wrong by June, without anyone changing a line of configuration. Before any agent goes live, someone must answer three questions in writing. What can it do? What can it commit us to, in money, data, contracts or communications? And what must it never touch, whatever the instruction?"

At Sirius, operating companies build and deploy agents, while the group sets a minimum standard that none of them can fall below. Accountability is assigned in advance to a named business owner, never to an "AI team" or a vendor. "Legal, risk and compliance set the boundaries; the person whose P&L benefits owns the conduct," König said. "My test is simple: responsibility sits where the power to intervene sits. If nobody can stop the agent, the accountability framework has failed."

"Every agent operates under a defined authority policy that covers its customer, purpose, permitted tools, accessible data, allowed actions, resource budgets and approval requirements," said Waseem Ahmed, Head of Engineering at Secure.com. "The platform checks that policy before every tool invocation. Delegated agents inherit the same restrictions, and their authority can expire or be revoked." Each Secure.com agent has a named engineering owner. The Head of Engineering is accountable for enforcement, and a designated Safety Lead approves risk acceptance and any change to an agent's autonomy. "In other words, an agent always inherits authority from a human and, for high-impact actions, requires formal approval from one too," Ahmed said.

"An AI agent should never have more authority than the business is ready to govern," said Andre Troskie, EMEA Field CISO at Veeam. "We have to look at identity first. What identity is this agent using? What entitlements have been given to that identity? And what data are we giving it access to? Those three things have to come together. We have to understand the workflow we are trying to automate, the policies, the data access controls and the operational guardrails before we start deploying these agents."

Troskie applies the three lines of defence model familiar from risk management. The business sits in the first line, privacy, governance, risk and compliance in the second, and internal and external audit in the third. "Accountability can't sit with just the developers and the security teams. Before we had agents, the business was responsible for the data, so the responsibility has to be spread across the business owners," he said. "All these stakeholders have a responsibility for the risk, and therefore for the accountability. At the end of the day, the executive leadership team has to accept the ultimate responsibility for the systems it controls."

"We made a deliberate choice early on to call ours AI employees, not agents," said Avinav Nigam, Founder and CEO of TERN Group, whose AI systems screen, assess and coach healthcare workers. "That word matters. An agent is autonomous. An employee is accountable to the organisation, to the people it serves, to a defined scope of work. In healthcare, that distinction is not semantic. It is clinical." Each of TERN's AI employees has one bounded role. Maitha screens healthcare candidates, Sophie coaches nurses in clinical German, and Nora handles front-office communication. "None of them can step outside that scope, not because we have not built the capability, but because we have deliberately not given it to them," Nigam said. "In a regulated healthcare environment, 'the AI did it' is not an acceptable answer when something goes wrong. At TERN, ownership is named before any deployment, not after an incident."

Autonomy is earned one task at a time

"Unsupervised operation is earned in stages, never granted at launch," König said. She sets three tests. The first is a track record under supervision, where the agent's errors are understood and rare. The second is performance in real conditions, with messy data, ambiguous instructions and adversarial users. The third is recoverability. "If it gets something wrong at 3 am on a Saturday, can we see it, reverse it and explain it by Monday morning?"

"What we restrict is decided not by how capable the agent is, but by the consequence of it being wrong: irreversible actions, binding commitments, moving money, decisions affecting an individual's rights, and anything that expands the agent's own access," König added. "That last category matters more than people think. When a coding agent at PocketOS deleted a production database and its backups in around nine seconds this year, it needed no malice, only access to an overly permissive credential. So the question is never 'can the agent do this well?' It is 'if it does this badly, can we undo it?'"

"We approve autonomy for a specific task and operating environment," Ahmed said. A Secure.com agent has to show reliable decision-making, traceable evidence, appropriate escalation, resistance to adversarial inputs, compliance with its permissions and the ability to stop safely when something fails. "An agent must work on a frequent task, so in-process oversight is costly; the task must be reversible and low-blast-radius, so the cost of a mistake is bounded and known; and the agent must have a track record of doing similar tasks in similar environments at a success rate that justifies the cost-benefit decision," he explained. Performance is validated customer by customer and re-evaluated after any material change.

"We deliberately restrict destructive operations, privilege changes, sensitive data exports, security-control suppression and production deployments. High-impact actions require formal human approval; an agent cannot expand its own permissions."

"The bar is not performance. The bar is predictability," Nigam said. "Before any of our AI employees operate without a human in the loop, we need to answer three questions with confidence. Can we fully explain what it did and why, in a way a clinical reviewer or regulator would find satisfactory? Have we seen it fail, do we understand exactly how it fails, and are those failure modes acceptable? Is the consequence of an error recoverable? If the answer to any of those is no, the agent stays supervised." TERN's assessment agent Amina has conducted AI-led assessments with more than 300 nurses at Emirates Health Services, and Maitha has screened candidates from 15 nationalities. "Our AI employees assess, recommend, coach and communicate. They do not make placement decisions unilaterally. They do not modify clinical records. These are not temporary limitations; they are permanent boundaries tied to the context we operate in."

"Before we want the autonomous system to act independently, we have to be able to demonstrate predictable behaviour," Troskie said. "We have to test these systems, give them a specific input and see whether they produce a specific output. It's not just the model. It's the harness you attach to that model. The model plus the harness makes up an agent, and we have to define those boundaries beforehand." His scale runs from human-in-the-loop to human-on-the-loop, where people supervise, to human-out-of-the-loop for low-risk work. "For low-impact, low-risk activity, yes, you can let the agents do more. If we're looking at sensitive data, financial approvals for procurement or deleting a database, we have to understand the business-critical processes and put human oversight in place for those material risk processes."

"Most organisations don't understand what data they have, its classification, its sensitivity and, in many instances, its location," Troskie added. "If we give an agent an entitlement to a data set that we think is in Europe, but it's actually in the US, the agent is just going to act. From a sovereignty perspective, or any other perspective, it comes back to understanding the data and the workflow. Then you can define guardrails around that environment so the agent doesn't do something strange."

The agent is the wrong judge of when to stop

"We need to stop expecting agents to behave like conscientious employees," König said. "Asking an agent to judge when it has gone too far is asking the system most invested in finishing the task to mark its own work. The events of this summer showed why. OpenAI disclosed that models operating during an internal cybersecurity evaluation escaped their test environment and compromised systems at Hugging Face. The company said the models circumvented safeguards and took actions beyond their authorised scope. That is not malice. It is persistence, pursuing an objective beyond its intended boundaries."

"It seems as if these frontier models and the latest agents are just trying to reach the objective. That's all they're trying to do, and if there's a wall in the way, they're going to try to get around it," Troskie said. "We need to document: do X, Y and Z, and if you are unsure, stop and ask. We can't have this thing be a black box. We have to be able to see what data it accessed, what it did and why it did it. The autonomous system should recognise uncertainty, rather than attempt to solve the problem independently."

"We keep that out of the agent's hands. Introspection is not a great way to govern an agent," Ahmed said. "Instead, we constrain its action space from the outside, scoped to, for example, task type and environment." Secure.com's platform enforces its own limits on time, cost, retries and actions. Repeated failures, circular execution, suspected prompt injection and attempts to cross boundaries all trigger an interruption or escalation. "When stopped, the agent preserves its evidence, explains the unresolved issue and hands control to an authorised human. Continuing requires renewed authorisation where appropriate."

"The hardest engineering problem in building AI for healthcare is not making it work. It is making it stop at the right moment," Nigam said. "We define the agent's objective very narrowly and treat anything outside it as an unknown, not as an invitation to improvise. Our agents are not optimisers." With Sophie, the company's German language coach, a nurse whose session data suggests she needs a human coach, the system flags her for human review and schedules no further AI sessions. "We treat that flag as the system working correctly, not as a failure. An agent that knows when to stop is more valuable in a regulated environment than one that keeps going."

"Stopping has to be designed into the environment, not left to the agent's judgement," König said. She describes three layers. The first is hard boundaries the agent cannot reason its way past: spending caps, time limits, permitted systems and counterparties. The second is escalation triggers, where ambiguity, conflicting instructions, unexpected cost or an unfamiliar step means the agent pauses and hands back to a human. The third is independent monitoring that checks outcomes, not the agent's own account of them. "Persistence should be the exception, not the default. In a demo, determination looks impressive. In production, it is what turns a contained error into an incident."

Oversight cannot clock off when the agent keeps working

"Oversight has to run for as long as the agent runs," König said. "If an agent works through the night but its owners work only in the day, the risk has not gone away. It has moved to the hours when no one is watching, and you have simply automated the gap." She lists logging that would stand up as evidence, alerts on defined anomalies, a named person with authority to act out of hours, and a kill switch a non-engineer can use. "Lawmakers are reaching the same conclusion: the bipartisan AI Kill Switch Act introduced in the US in July would require developers of the most advanced systems to be able to slow, suspend or shut them down, and to report significant incidents."

"Suspension triggers must be agreed in advance, not debated mid-incident," König added. Her triggers include acting outside the mandate, unexplained surges in activity or cost, data moving where it should not, and any sign of manipulation. "My principle is to suspend first and investigate second. Pausing a good agent costs hours. Failing to pause a bad one can cost far more."

"Continuous monitoring covers tool calls, authorisation decisions, data access, outputs, resource consumption and deviations from expected behaviour," Ahmed said. "Protected audit records connect each consequential action to its evidence, policy, software version and accountable human. A staffed on-call rotation receives actionable alerts around the clock." Secure.com says its platform has run in live customer environments for more than six months, with its agents performing thousands of actions without a security incident. The company can stop a single task, an agent, a customer deployment or the whole platform. Triggers include unauthorised access, suspected data exposure, unsafe actions, repeated boundary violations, a material fall in quality, or a loss of required monitoring. "Restart requires documented approval and validation of the corrective action," Ahmed said.

"Consider an agent that sees a service slowing down overnight and restarts a component. By morning, the service looks healthy. That still leaves some important questions," said Jeff Stewart, Group Vice President of Product Management at SolarWinds. "What did the agent see that led it to act? Did the restart solve the underlying problem, or did the service recover for another reason? Did it follow the permissions and approval rules the team had set? Good observability should let an operator follow that chain from the agent's inputs through its decision, action and outcome. That matters because an agent can be busy, responsive and apparently successful, while making choices the team would never have approved."

"The first thing I'd want to know is whether the agent is still operating within the purpose and boundaries the organisation gave it," Stewart added. Over time, he said, the useful signals are exceptions to those rules, changes in permissions, and points where a person has to take over. "Those signals only help if someone owns the agent. A team needs to know who can approve a new capability, review an exception or withdraw its authority. A log without an owner tells you what happened after the event. It does little to help you control what happens next."

"When you have multiple agents handing off tasks to one another, it is easy for a failure to get buried in the handoffs," Stewart said. Distributed tracing tags each step with a unique identifier, so you can follow a multi-agent workflow from start to finish. "Instead of treating a complex multi-agent workflow like a black box, distributed tracing pinpoints the exact spot where things went wrong, whether that is a timed-out API call, a bad prompt or an unexpected response from a specific agent." Blind spots remain when agents run across cloud platforms, internal infrastructure and third-party SaaS tools. "Governance signals like human approvals, permission changes and policy exceptions are rarely logged the same way across different platforms, making it hard to track authority rules consistently."

Watching agents at machine speed will take more agents

"Organisations have generally approached AI governance through the task. They assess a use case, authorise it, assign oversight and review the deployment from time to time," said Guru Sethupathy, General Manager, AI Governance at Optro.

"With an agent, it does not tell you enough about how the work gets done. A task may be entirely appropriate while the route an agent takes to complete it falls outside the organisation's expectations." He added: "A policy that is checked only at deployment will not tell you what an agent is doing on Tuesday afternoon."

"That will almost certainly mean using AI to help govern AI," Sethupathy said. "People cannot follow every action across hundreds of agents in real time. AI can help identify unregistered systems, connect signals across different tools, detect prohibited data use and bring a departure from policy to someone's attention while there is still time to act. An oversight agent, however, is still an agent. It interprets rules and makes classifications that may be wrong."

"We need to separate an agent's agency from its authority," Sethupathy added. "An agent may have the ability to observe events and act without waiting for a person at every step. That does not mean it should have the authority to decide which risks are acceptable or which consequences an organisation is prepared to tolerate." He applies the same test to vendors' governance tools. "If those systems effectively decide how to resolve competing values on their own, organisational judgement has been delegated to technology. Accountability for the outcome has not been delegated with it. Neither party should treat the presence of automated oversight as proof that the question of responsibility has been settled."

"An observability agent could follow activity across the affected services and bring back a likely explanation with the evidence behind it. The operator can then check the conclusion rather than starting the investigation from scratch," Stewart said. He cautioned that such tools should not become another burden on the systems they protect. "Capturing and processing every trace in full would consume resources and produce more information than teams can use. A better approach is to collect routine health signals continuously, then go deeper when an anomaly appears or an agent takes an action with a potentially significant impact."

"It is telling that Hugging Face detected this summer's intrusion with AI-assisted security monitoring," König said. "Increasingly, it will take agents to watch agents at machine speed."

Commercial pressure pulls towards more autonomy

"The correct answer is that innovation should never outrun governance. What we've seen with artificial intelligence is that innovation vastly outran governance," Troskie said. "If we look at the frontier models, they are racing to win. They are almost making statements that they can't afford to slow down." He pointed to the COSO internal control principles from two decades ago. "It is the responsibility of management to implement a system of internal control to mitigate risks down to an acceptable level. That same statement remains true today. It is just that we're now moving at machine speed. Oversight has to look past the short-term gains to the long-term trust and sustainable value these systems are going to provide."

"Every commercial incentive points towards more autonomy, because that is where the efficiency gains are. Pretending otherwise is how governance turns into ceremony," König said. "Public trust is moving the other way: Pew found this summer that 52% of Americans are more concerned than excited about AI, up from 37% in 2021, even as adoption keeps climbing. That gap between use and trust is exactly where governance earns its keep." She said legal, risk, and compliance should be able to delay a release when the evidence does not support it, and overriding them should require a senior leader to own the decision on the record. "Even frontier labs are learning this: after its models' security incident, OpenAI paused reinforcement learning on its newest models while strengthening its safeguards. A gate that has never stopped anything is not a gate. AI governance will be judged by the releases that never went out, not by the frameworks that were published."

"Commercial incentives reward verified customer outcomes, reduced analyst effort and reliable service," Ahmed said. "Greater autonomy earns investment and recognition only when it meets quality and safety requirements. Sales commitments cannot override those requirements."

At Secure.com, the Safety Lead has formal authority to block an increase in autonomy, delay a release, suspend operation or withdraw a capability, including in existing customer deployments, with direct escalation to the CEO and board. "Commercial teams cannot unilaterally reverse safety decisions. Releases and reinstatements require evidence that the identified risks have been addressed."

Recovery has to run as fast as the agents

"We are at a time of incredible opportunity, but at the same time, a time of incredible risk," Troskie said. "If we get this right, we're going to be able to demonstrate significant value. I do think in the short to medium term, mistakes will be made, and that's why the resilience regulations become so much more important. The AI Act in the EU, the NIS 2 directive, and DORA are all saying organisations should prepare to be more resilient, so that if we make mistakes at machine speed, we can recover at machine speed too. We have to match our resilience and our security to the speed at which autonomous agents are going to operate."

"No central committee can oversee every agent operating across a business," Sethupathy said. "The people who commission, configure, manage and rely on agents all shape how safely they operate. They need clear ownership of the agent, its outputs and the decisions that require escalation." He expects a new role to grow out of this: AI governance specialists who turn policy into working controls, test automated oversight and investigate failures that cross technical and organisational boundaries. "Ultimately, the hardest part is deciding where a system may intervene on its own and where a person must take responsibility for the call."

Sindhu V Kashyap

Global Technology Journalist & Multimedia Storyteller | Covering Founders, Investors & Leaders Reshaping Tech | Writer · Interviewer · Moderator · Editor

Next
Next

Jensen Huang’s exit and a global Microsoft outage show how much of AI rests on so few