How to design AI agents people can trust

1 The bottom line

Start agents with narrow tasks, tight access and a human in charge, then widen both as reliability is proven. Predictable behavior is what lets people hand over real work.

2 Key takeaways

• Start with low autonomy and a human in control. Every bit of autonomy you hand over costs some control.
  • In customer support, begin with AI suggesting answers to human agents, then show answers to customers, then add refunds Aishwarya Naresh Reganti and Kiriti Badam, OpenAI15:22.
  • About 74 to 75% of enterprises named reliability as their biggest problem Aishwarya Naresh Reganti and Kiriti Badam, OpenAI10:00.
• Give the agent safe access first and loosen it over time. Safety is the starting point, and productivity is what you iterate toward.
  • WiseSam began on a pre-configured virtual machine with its own email address and an allow list of senders Patrick and Pablo, Social (Superwall)38:50.
  • OpenAI runs specialist dots that touch production systems on their own hardware with extra monitoring Tibo Sottiaux, OpenAI22:02.
• Make behavior predictable and show the work. People trust what they can foresee and verify.
  • A settings menu or popup no longer excuses surprising behavior from a semi-autonomous agent Tara Seshan and Nan Yu, OpenAI22:40.
  • In knowledge work, show in-progress work, citations and reasoning, not only the final output Tara Seshan, OpenAI1:05:38.
• Group agents into a few bundles under one lead. People cannot manage dozens of agents any better than dozens of direct reports.
  • Users asked how they manage many agents usually answer with a chief of staff agent Tara Seshan and Nan Yu, OpenAI9:05.
  • A supervisor with sub-agents works, and peer-to-peer designs do not Aishwarya Naresh Reganti and Kiriti Badam, OpenAI1:02:00.
• Always have a fallback that finishes the task. A product that fails at 99% feels worse than one that never started.
  • Computer use can finish the job when plugins or MCPs are missing, though slower and with more tokens Tara Seshan and Nan Yu, OpenAI15:59.

we want the agent to start from being very safe and then we iterate towards you know being productive

Patrick and Pablo, Social (Superwall)

2.1 Start narrow with a human in charge

Begin with one agent and one small task where a person can undo mistakes.

• Start with low agency and high human control Aishwarya Naresh Reganti and Kiriti Badam, OpenAI15:22.
  • Hand over more decisions only as the system earns trust.
  • Starting with a full agent makes it impossible to control the complexity or tabulate the many ways it fails.
  • A coding assistant goes from inline completions to larger blocks for review to autonomous PRs.
  • Marketing goes from drafted copy to multi-step campaigns to auto-optimized campaigns.
• Pick a first task that humans can undo Aishwarya Naresh Reganti and Kiriti Badam, OpenAI51:10.
  • In a support build, version one only classified and routed tickets, because humans can undo wrong routing.
  • Even routing is hard at enterprises. Shoes, women's shoes and men's shoes can sit at the same level, and a dead node may have been last updated in 2019.
  • Early stages surface these data problems and the context engineering needed before you add drafting and end-to-end resolution.
• Launch a single agent first and learn from real use Tibo Sottiaux, OpenAI10:55.
  • Dots launched with one primary dot that learns preferences most deeply.
  • Launch to Pro users first, see what they do, and tune with the community before adding a second, third or fourth dot.
  • Tibo uses one dot only to monitor Twitter, and that is enough work for an entire dot.
• Choose the right tool for each workflow step Aishwarya Naresh Reganti and Kiriti Badam, OpenAI29:35.
  • Understand the workflow, then decide which parts suit AI and which need a human in the loop.
  • Often an ML model does one part and deterministic code does another.

2.2 Give safe access first

Limit what the agent can touch at launch, watch it closely, and widen access in steps.

• Start from very safe and iterate toward productive Patrick and Pablo, Social (Superwall)38:50.
  • WiseSam runs on a pre-configured virtual machine with its own email address and an allow list of senders.
  • The team admits this was a bit too conservative and plans to add read access to the inbox over the next weeks.
• Give the agent its own computer and keep the harness off your machine Roman Ugarte, Grok Bot (SpaceXAI)32:05 and Tibo Sottiaux, OpenAI22:02.
  • A bot with its own computer avoids sharing credentials and tripping over the user.
  • Run the harness separate from the machine, so one agent can connect to many devices.
  • Give specialist agents that touch production extra guardrails and monitoring, on their own hardware.
  • Some of OpenAI's specialist dots run on Mac minis, and Tibo's dot flagged a ChatGPT outage 5 minutes before a live demo.
• Run a second system that watches the first Tibo Sottiaux, OpenAI34:33.
  • The monitor stops high-risk actions or interrupts the agent if it may be getting prompt injected.
  • The majority of OpenAI's API stack investment goes into the safety stack, and the model 6.1 Astra was not released.
• Guide new users with preset first tasks Patrick and Pablo, Social (Superwall)34:25.
  • Open the chat with buttons like helpful tasks, calendar and mailbox that send a prepared prompt.
  • Nail the first three to five steps and build trust before sensitive tasks like reading mail.
  • The team is identifying about 10 use cases that anyone feels value from quickly, such as birthdays or price checks.

2.3 Make behavior predictable and visible

Users should foresee what the agent will do, check how it got there, and control their data.

• Let users predict what the agent will probably do Tara Seshan and Nan Yu, OpenAI22:40.
  • At any point, a user should be able to predict what a feature or primitive will likely do.
  • Users also need to understand privacy, how their data is used, and the norms of the product.
  • With semi-autonomous agents, you can no longer excuse surprising behavior with a settings menu or popup.
• Show process, inputs and citations, not just output Tara Seshan, OpenAI1:05:38.
  • Expose in-progress work, citations and reasoning so users can verify the result.
  • Make the model a collaborator the user goes on a journey with.
  • Coding can be verified by tests on the output. Knowledge work cannot be trusted from a final deck number alone.
• Give users per-item control over what the AI learns Casey Winters, SuperMe22:12.
  • Do not let the AI alone decide what is public.
  • For each synced item, the owner picks private, company only, network only, or public.
  • This makes people willing to connect sensitive sources like meeting notes and Slack.
• Watch for over-moderating Connor Zwick, Speak19:50.
  • Speak added heavy moderation to avoid a viral inappropriate screenshot, then dialed it back.
  • Speak tested GPT-4 with 30 power users, and one used it about 8 hours a day to simulate a boyfriend.
  • Over time the bigger risk was users being offended by censorship in their specific scenarios.

2.4 Make sure the task always finishes

Layer capabilities and add human help so a stuck agent never leaves the user with a broken promise.

• Finish the last mile Tara Seshan and Nan Yu, OpenAI15:59.
  • A product that gets 99% there and then fails is an unfulfilled promise, and feels worse than one that never started.
  • Use computer use as the fallback when plugins or MCPs are missing. It may be slower and cost more tokens.
  • Nan notes computer use will 100% do the job while the ecosystem catches up with hooks and MCPs.
• Structure capabilities as layers Tara Seshan and Nan Yu, OpenAI14:35.
  • Order them as native features, first and third party plugins, then computer use as a fallback.
  • Expose hooks so developers can build cohesive, composable experiences.
  • A meetings plugin pulls meeting data, and computer use covers the case where the plugin interface is missing.
• Build a fallback experience for when safeguards block a model Dianne Penn, Anthropic37:00.
  • Improve safety systems and user experience together.
  • Before the Fable models Anthropic had no strong fallback UX, so it built systems that route users to a great Opus 4 response.
• When the agent is stuck, have it call a human and learn Anish Acharya, a16z19:30.
  • Treat every failure as a knowledge gap or data gap to fill, so the agent does not need to call next time.
  • Kavak runs an agent per customer for online used car sales. A human coaches it through and the agent captures the traces.

2.5 Group agents and give them work

Keep the number of things a person manages small, with one lead agent over the rest.

• Bundle agents so people can hold them in their heads Tara Seshan and Nan Yu, OpenAI9:05.
  • People struggle to manage 40 direct reports, and they do the same with agents.
  • Map agent design to human nature and group activity into bundles.
  • Users often create a chief of staff agent, so "you're not managing seven you're managing one."
• Use a supervisor with sub-agents, not peer-to-peer agents Aishwarya Naresh Reganti and Kiriti Badam, OpenAI1:02:00.
  • Agents talking to each other in a gossip-style protocol do not work with current capabilities.
  • In customer support, peer-to-peer agents make it hard to control which agent replies and force guardrails everywhere.
• Let users promote a bot to chief of staff, but keep it a two-way door Roman Ugarte, Grok Bot (SpaceXAI)13:20.
  • The team did not lead early users there, saw many reach it themselves, then encouraged it slightly without making it mandatory.
  • The pattern emerged around the end of week two of internal rollout.
• Settle single versus multiple agents by working through edge cases Tara Seshan and Nan Yu, OpenAI9:33.
  • Decide how memory is segmented, for example per private Slack channel.
  • Decide how the agent talks to each person, and whether it uses the user's credentials or a universal service account.
  • Make that behavior easy for the user to understand.

2.6 Widen autonomy as surprise falls

Hand over more only when the system has stopped teaching you anything new.

• Make the agent earn autonomy Aishwarya Naresh Reganti and Kiriti Badam, OpenAI10:00.
  • Constrain autonomy by number of actions, or by topic and risk.
  • In insurance pre-authorization, AI can auto-approve MRIs and blood tests while invasive surgery stays with humans.
• Move up when you get little new information Aishwarya Naresh Reganti and Kiriti Badam, OpenAI58:20.
  • If you calibrate every day or two and see no new data distribution patterns and consistent user behavior, you can move up.
  • Recalibrate when events change the system, or when user behavior evolves.
  • Swapping a deprecated model like GPT-4o for GPT-5 is one such event.
  • A bank underwriting assistant worked well for about three months. Then users asked deep questions about what previous underwriters did, which required rebuilding parts of the system.
• Budget months and distrust one-click agents Aishwarya Naresh Reganti and Kiriti Badam, OpenAI31:00.
  • Enterprise data is messy, so an agent needs time to learn the system.
  • Prefer a team that builds a pipeline that learns over one promising instant gains.
  • Meaningful ROI easily takes four to six months, even with the best data and infrastructure layers.

2.7 Where they disagree

Some tips pull against each other, and the right call depends on your product and its risks.

• Tight structure versus open tools.
  • Boris Cherny, Claude Code (Anthropic)1:04:06 says to give the model tools and a goal, not a fixed step-by-step workflow.
  • Aishwarya Naresh Reganti and Kiriti Badam, OpenAI29:35 advise splitting the workflow so deterministic code and humans handle some parts.
  • Scaffolding may improve performance 10 to 20%, but the next model often wipes out those gains Boris Cherny, Claude Code (Anthropic)1:04:06.
• How many agents to start with.
  • Tibo Sottiaux, OpenAI10:55 launched with one primary dot, then added a virtual team with specific roles.
  • Tara Seshan and Nan Yu, OpenAI9:33 say the answer depends on use case details like memory, permissions and credentials.
• Heavy safety versus over-moderation.
  • Tibo Sottiaux, OpenAI34:33 keeps investing ahead on safety and monitoring.
  • Connor Zwick, Speak19:50 found heavy moderation became the bigger risk over time.
  • WiseSam's team also called its safe start a bit too conservative Patrick and Pablo, Social (Superwall)38:50.
• Computer use as the main path or as a fallback.
  • Roman Ugarte, Grok Bot (SpaceXAI)32:05 gives each bot its own computer, because humans do their jobs by clicking on pixels, not through MCPs and APIs.
  • Tara Seshan and Nan Yu, OpenAI14:35 place computer use last, behind native features and plugins.