What Do You Do While Your AI Agents Are Working?

Recently, I spent two days in a hackathon with a teammate. We had four independent product workstreams running at the same time.

Neither of us wrote code.

We were still busy for two full days.

A year ago, I would have struggled to explain what we were doing. If nobody is coding, is it still a software hackathon?

We prepared tasks, reviewed plans, opened completed candidates, used the actual product, dictated feedback, and decided what the agents should do next. While one agent was implementing, we were reviewing the result of another.

This only became a real problem once my AI workflows started running for days instead of hours. In I Used to Run AI Agents for an Hour. Now They Run for Days, I wrote about what made that possible. I had not answered the obvious follow-up: what happens to my job while they are running?

Four workstreams are enough for now

My first response was to parallelize. If one agent needs six hours, I can start another independent mission instead of waiting.

That works, but concurrency is not free. Tokens burn quickly. The bigger limit is my attention. Agents can submit plans and product candidates faster than I can review them responsibly.

Launching twenty missions would create an expensive queue of work waiting for judgment. During this hackathon we ran four. My current hypothesis is that three or four active missions are a reasonable range for one operator, depending on their risk and review cost. I am testing that range, not proposing a universal number.

AI-generated editorial illustration: four agent workstreams can still create one human review bottleneck.

The first thing I check is whether an active mission needs a decision from me. A missing product choice or unclear boundary can leave an agent waiting, so I handle the questions that unlock more work. I do not need to follow every update. I need to notice the handoffs where the next step genuinely depends on me or my teammate.

Once the active missions can continue, I prepare what should run next.

Better agents have made detailed product work more important in my workflow. A vague prompt can send an agent through hours of plausible implementation in the wrong direction. Clear acceptance criteria, known edge cases, product boundaries, and verification instructions reduce that back-and-forth.

Preparing a PRD is no longer something I do before the real work begins. It is part of the work I do while another mission is running.

The loop I am developing is simple: launch a bounded mission, answer the decisions that block it, prepare the next task, and review the candidates that return. The review then improves the next specification.

The review moved into the product

The localization workstream gave me the clearest example.

An agent implemented translation in the frontend and submitted the candidate for review. I opened the actual product, selected another language, and started using it. As I moved through the interface, I dictated feedback for the next iteration.

The implementation looked plausible, but the experience was wrong. Parts of the product had loading problems, and native-language users could briefly see English content while the translated interface was loading.

We did not take over the implementation. We asked the agent to move translation to the backend so the frontend could reuse the loading states already used elsewhere. We also made the product requirement more precise: native-language users should not see English content during loading or transitions.

That next iteration is still in progress. I am not using the story as proof that one feedback cycle solved the problem. What changed for me was the place where human judgment entered the process.

The agent produced a plausible frontend translation candidate. We used the real product, noticed a failure that mattered to the experience, and turned it into an architectural change and a better acceptance criterion.

This is outcome feedback, not implementation intervention.

AI-generated editorial illustration: the reviewer uses the product while the next localization architecture remains in progress.

Product review asks whether the result creates the experience we intended. Code and architecture review ask whether consequential changes are safe and whether I understand what they do. I still inspect the implementation, especially when a change affects security, data, money, or a difficult-to-reverse boundary. But I do not approach every generated line as if my job is to prove I could have written it better.

Implementation checks alone did not tell me whether this loading experience felt wrong. I had to open the candidate and use it.

I still need to understand what I approved

Direct product review catches one class of problem. It does not keep me informed about every decision that became part of the architecture.

I am experimenting with a second review after the urgent work is handled. I ask the agent why it chose a library, why it placed a responsibility in the frontend or backend, which alternatives it considered, and what trade-offs it accepted.

I am not trying to confirm that every choice is optimal. Usually I will not have a better alternative ready. I want enough of a mental model to understand the product I am responsible for and make the next decision without starting from zero.

Implementation can advance faster than my understanding. If I only ask whether the visible feature works, more decisions harden into the system while I know less about why it behaves that way. That knowledge gap makes the next PRD worse and turns future review into guesswork.

The order matters. I first handle decisions that keep active missions moving. Then I review submitted candidates and prepare future tasks. Once those are under control, I can ask why consequential technical decisions were made.

This is also where my earlier idea for an educational agent came from. I built a small experiment called Operator Gym that tried to turn live implementation work into questions and architecture exercises. The first versions created more work and confusion instead of understanding. I still suspect there is room for tools that help an operator learn from delegated work, but I do not need that product to follow the simpler rule today: use the real candidate, ask about consequential decisions, and preserve the answers for the next mission.

The work moved

By the end of the hackathon, I had not written implementation code. I had moved repeatedly between active missions, product candidates, feedback, and the specifications for what should happen next.

That did not feel like waiting. It also did not feel much like the software development I learned years ago.

The temptation is to fill every recovered hour by launching another agent. I already know where that leads: more token usage, more completed artifacts, and a review queue I cannot honestly understand. The other mistake is easier to hide. I can accept completion reports, watch features accumulate, and slowly lose the ability to explain the product I am approving.

For now, I am keeping the portfolio small. I run a few independent missions, answer the decisions that unlock them, prepare detailed work for the next run, and spend most of my attention on the candidates I can actually use and judge.

Neither of us wrote code during those two days. We were still doing the work. I am only beginning to understand what that work has become.