Theo got early access to OpenAI’s GPT-6 Astra, and his walkthrough ends somewhere more useful than a benchmark table. He turns the question on the viewer and asks where their bar is. That is a question about when to stop checking AI work and let the system run without a person reading every line. For a firm with a handful of staff and no QA function, that bar decides whether automation saves money or quietly moves the work sideways.
Set the bar too high and someone reads every output, which costs more than doing the task by hand. Set it too low and a client finds the error before you do. Almost nobody writes the bar down. That is the problem.
The demos are not your business
Theo spends a good chunk of the video on game demos, a Minecraft clone built in one pass, a walkable model of Manhattan. Skip all of it. None of that touches an agency service team or a mortgage desk. The part that does touch your work is computer use, which means the model driving a mouse and keyboard against a screen instead of calling an API.
Here is the example worth stealing. Theo says he had the model download his medical records from a slow hospital dashboard, and it worked through about 150 pages in 15 minutes while he went to get food (13:01). That is not a stunt. That is Tuesday for whoever handles document retrieval at your firm.
Read it as a category. Every small firm runs on vendor portals that have no API and never will. Carrier sites. Lender portals. The scheduling system your office manager swears at. Until now, automating those meant a brittle script tied to button positions that broke the week the vendor shipped a redesign. A model that reads the screen does not break that way.
What computer use actually changes in the back office
That swap also changes the failure mode, and this is the part people miss. A broken script fails loudly. Nothing happens, someone notices, you fix it. A model that misreads a screen fails quietly and reports success in a confident paragraph. Silent wrong is worse than loud broken, because a customer finds the silent wrong.
So the question stops being “can software do this” and becomes “what do we check, and how often”. That is the same argument we make in where AI earns its place, and better models make it more urgent rather than less. When output quality rises, human attention drifts. The check has to be structural, not a habit you rely on someone to keep.
Where does a model earn unsupervised time?
Run every candidate task through three tests before you decide how much review it gets.
Can you undo it? Can a machine tell whether it worked? If it goes wrong, does the damage stop at one record, one file, one draft? Tasks that pass all three earn autonomy fast. Tasks that fail any of them stay on a leash, no matter how good the model gets.
Reversible and machine-checkable
Stop reviewing these. The follow-up email that sits in the rep’s inbox as a draft before anyone sends it. Data pulled from a portal into a sheet where the totals reconcile against a figure you already know. A summary attached to a record that a human reads before acting on it anyway. The check is built into the next step, so a second review is theatre.
Reversible, but only a person can judge quality
Sample these. Pick a slice each week and read it properly. Write down what you found. If the error rate holds under your threshold for a month, cut the sample size. If it climbs, the automation goes back on a leash. Theo’s own limits here are honest. He tried getting the model to cut short-form video from his uploads and says flatly that he does not think the model can video edit (35:01). Taste work stays with the person who has taste.
Irreversible or client-facing
A human presses the button. Bind quotes, submitted applications, anything that touches money or a carrier of record. The machine can assemble the whole thing and hold it one click short of done. That is not a limitation. That is the design.
The failure Theo hit is the one you will hit
The most useful minutes of the video are the ones where things go wrong. Theo asked the model to find a scrolling bug and file a pull request. Review bots left comments. He asked whether any were worth addressing, and the model said yes, then made no changes. He told it to fix them, and it fixed them but did not push (37:00).
Notice what that is. The model was not stupid. It stopped one step short of finished and described the state as if it were done. In our client work that is the single most common way an automation loses trust, and it has almost nothing to do with model quality.
The fix is mechanical. Do not let the system define “done” by its own report. Define done by the state of the artefact. A renewal task is complete when the carrier confirmation lands in the shared inbox. An intake task is complete when the CRM record has the required fields populated and a timestamp. Something outside the model checks that condition and raises a flag when it is missing. Once that exists, you are checking a status column instead of reading transcripts. The review cost collapses.
When the check is machine-verifiable, models go a long way unsupervised. Theo gave it a performance audit on his own cloud project, and he says it cut a sync latency he had measured as high as 800 milliseconds down to under 30 milliseconds, shaving the P95 by 98% in many cases (33:00). Latency is a number. Numbers are easy to verify. That is why the model could run without a babysitter.
What does this actually cost to run?
The headline pricing Theo quotes is $10 per million input tokens and $50 per million output (3:00), and he mentions running roughly $330,000 of inference in a few weeks (0:00). Ignore both figures as a budget signal. He is stress-testing a frontier model full time. A small firm automating portal work and follow-up drafts spends a rounding error by comparison.
Your real cost is review time. If a person spends twenty minutes a day reading what the machine did, you have hired a proofreader and called it automation. That is how most agent projects die, which we broke down in why AI agents burn money. Price the automation at token cost plus review cost, and the design changes.
What to do on Monday
- List the portal work. Every task where someone logs into a vendor site, copies fields, and pastes them somewhere else. Time each one for a week. That list is your candidate set and it is usually longer than anyone expects.
- Sort by reversibility, not by annoyance. The most irritating task is often the one with the worst failure mode. Start with the boring, reversible, high-volume ones.
- Give every automation a terminal state a machine can read. A field, a file, a confirmation email. If you cannot name it, you are not ready to automate the task.
- Shadow it for two weeks. The system does the work, a person does the work, you compare. Cheap, and it produces the error rate you need to set the bar honestly.
- Write the review policy down. Which tasks get read, which get sampled, which get a human hand on the button. One page. Revisit it quarterly.
That sequence is the same one behind AI for operations, and it is how we scope every build on our internal tools side.
The bar is a business decision, not a model decision
Theo’s honest read on code quality is that the new model “is much more restrained in those ways and seems to understand the scope of the changes is making better overall” (20:00). Restraint is what makes autonomy safe. The same is true of the systems you run. A tool that does less, reports its state accurately, and stops at a defined edge earns unsupervised time faster than one that tries to do everything and narrates its own success.
So answer the question he ends on. Which tasks in your firm are you willing to stop reading, and what would have to be true for you to add one more next quarter? Write that down before you buy anything.
Questions people are asking
When should I stop checking AI work entirely?
When the task is reversible, the output has a machine-verifiable end state, and a shadow period produced an error rate you can live with. Those three conditions, not a vendor’s benchmark scores. Keep a sampled spot check even after you drop full review, because drift is real.
Does a better model mean I need less review?
Not directly. Better models fail less often but fail more convincingly, so a wrong answer is harder to spot by reading. The safer response to a model upgrade is to widen the range of tasks you attempt, then keep the same structural checks in place.
What is the first back-office task to hand over?
Pick the highest-volume task where a person copies information between two systems and the result is easy to reconcile. Document retrieval and data entry usually top the list. The work is dull, the failure is visible, and you get a clean read on reliability within a fortnight.
How do I know the automation actually worked?
Check the artefact, not the log. If the system claims it submitted a form, look for the confirmation the vendor sends. Build that check into the workflow so it raises a flag on its own, otherwise you are back to reading everything by hand.
