The boundary of autonomy
How to decide what AI may do on behalf of the business

- By
- Cam Myers
- Content type
- Point of View
- Reading time
- 6 min read
- Status
- Author’s perspective
- Topics
- Operating Models
In brief
- A recommendation and a binding customer commitment require different operating standards.
- Important limits should be enforced by the systems that execute the action.
- Verify authoritative outcomes and contain cumulative exposure, exceptions, and partial failures.
The argument
There is a major difference between asking software what price to charge and allowing it to send the customer a binding quote.
The recommendation can be reviewed. The quote changes the customer's expectations and may commit the company to an expensive promise. Even when both actions use the same model and the same information, they require different operating standards.
That distinction should guide how owners approach AI agents: software that can select and take actions using connected tools. The important decision is how much authority the business should delegate for a particular kind of work.
An agent that can read every record and change anything may be convenient to demonstrate. It is difficult to operate responsibly because its possible actions extend far beyond any single task.
The decision underneathWhich actions, under which conditions, within which limits, with what evidence of completion?
Start with the business action
Consider an illustrative midsized equipment rental company in the northeastern United States using AI to help with customer requests. Summarizing a conversation, checking equipment availability, issuing a quote, reserving a unit, and collecting payment are separate actions.
They can be combined in a workflow without being granted the same freedom. The company might allow automatic summaries, require current availability before a reservation, and prevent any quote outside approved pricing rules. An unusual refund could remain with a manager.
The boundary follows the consequence and the evidence available. Can the action be checked before it takes effect? Can it be undone? Would reversal actually repair the harm? Canceling an incorrect email does little after a customer has relied on it.
Repetition changes the exposure too. A small pricing error repeated across hundreds of quotes may matter more than a larger error on a single internal recommendation. Limits should account for the total activity, not only the amount of each transaction.
NIST's 2026 concept work on software agents explicitly raised identification, authorization, and auditing as practical issues.[1] For an owner, the underlying requirement is familiar: know which actor did the work, what it was allowed to do, and what record exists of the action.
Keep business limits enforceable
A written instruction asking an agent to respect a price floor is useful. A system rule that prevents a below-floor quote from being issued provides a different kind of assurance.
Where a limit is definite and important, enforce it through the application or service that carries out the action. The AI can interpret a customer's request or propose an answer while the surrounding system checks permissions, prices, inventory, and required approvals.
This separates judgment from constraints the business has already settled. The model does not need discretion over whether an employee's access rights apply or whether unavailable inventory can be promised.
It also protects against information being mistaken for authority. A customer email, supplier document, or web page can contain instructions. Those instructions should not acquire permission to change the company's rules merely because an agent reads them during its work.
Credentials should provide the access the task requires. An agent handling delivery questions usually has no reason to change payroll or export the full customer database. Narrow access makes both mistakes and misuse easier to contain.
Verify the outcome outside the explanation
A confident statement that a task succeeded is not sufficient evidence that it did.
If an agent says it booked a rental, the authoritative booking record should show the right equipment, dates, customer, and price. If it says payment was collected, the payment system should confirm the result. The business needs to check the state that matters rather than rely on the agent's description of its own performance.
Failures between systems need particular attention. A payment may succeed while the booking request times out. Repeating the entire workflow could charge the customer twice. The design should recognize an existing transaction and reconcile the incomplete steps before trying again.
Human review is only useful when the reviewer can assess the evidence. Asking someone to approve an AI-generated explanation without showing the source records can turn oversight into a formality. The reviewer needs time, context, and the ability to stop the action.
Research on AI-assisted consulting has documented how performance varies across tasks, including tasks of apparently similar difficulty.[2] That supports testing the particular work the company intends to delegate. It provides no basis for assuming that success on one workflow carries over to every nearby one.
Consider the way errors accumulate
Owners often ask how accurate an agent is. The average answer can hide the failures that matter most.
A quoting system might handle ordinary orders correctly and repeatedly misprice a less common rental duration. The overall success rate can remain high while every customer in that category receives the same wrong promise. These errors share a cause, so they can accumulate together.
Test important exceptions and ambiguous requests, including stale records, unavailable equipment, conflicting instructions, and interrupted transactions. Examine the financial exposure of a repeated mistake. A system handling low-value actions still needs limits if it can perform them at high volume.
The objective matters as much as accuracy. An agent told to maximize bookings may offer terms that increase sales while consuming scarce capacity at poor margins. The company must specify what acceptable performance means, including the costs and commitments the agent is allowed to create.
That is a management decision before it is a model-selection decision.
The total exposure should include related actions. An agent authorized to issue credits of up to $100 could still issue fifty credits to the same account unless the surrounding system also limits cumulative activity. An action can satisfy its individual rule while the combined sequence violates the owner's intent.
The company also needs an operating response when confidence falls. A missing confirmation should cause a defined reconciliation or exception, rather than an optimistic message to the customer. A growing pattern of pricing mistakes should trigger a pause in quoting authority while other safe capabilities continue.
Containment is more useful than a single all-or-nothing switch. A business may need to stop automated commitments without losing access to summaries, searches, or recommendations. Designing those permissions separately allows the service to degrade in a controlled way while the cause is investigated.
Source notes
Business examples and figures are illustrative and are not reported client results.
- NIST. New Concept Paper on Identity and Authority of Software Agents. February 5, 2026. The announcement concerns concept work, not a completed mandatory standard.
- Fabrizio Dell’Acqua and coauthors. Navigating the Jagged Technological Frontier. Organization Science, published online March 11, 2026. DOI 10.1287/orsc.2025.21838. The experiment studied GPT-4-assisted consultants; it does not establish the accuracy of current models or autonomous agents.
For the leadership team
In practice
- Grant access and authority separately for distinct business actions.
- Test exceptions and repeated mistakes, not only average task success.
- Expand execution rights with evidence and preserve safe capabilities when others are paused.
Questions worth asking
- What commitments can the agent create, and what limits can it not override?
- Which authoritative record confirms completion when part of a workflow fails?
Weighing this decision now? Tell us what you are considering and we will say plainly whether we can help.
Discuss a software capability

