no obvious trends. The most important thing we’ve seen is giving the agent the ability to fully drive the box, whatever the box may be. For terminal apps that’s driving through tmux. For webapps, that’s playwright. For slack apps, we straight up created a dummy slack app in a dummy slack env that it can play with. It’s challenging but works well when done right
The Ci/Cv framing puts words on something I've been living for 18 months as a non-programmer running an agent-built trading system: my Ci is near zero by construction (I don't write the code), so the entire discipline lives in Cv. It works — but this week taught me a corollary your framework hints at without stating:
Verification debt can accumulate inside the verification layer itself.
I run new components in shadow mode before granting them any authority: the candidate computes decisions, persists them, and a comparator checks them against the live path. First full diagnostic cycle: 25 agents, 17 mismatches — every one exactly 2x. Expected 0.500 vs candidate 1.000, expected 0.425 vs 0.850, and so on.
My instinct said the candidate was broken. The evidence said otherwise. Because the comparator persists which rules composed each value (not just the final number), one query showed the expected-value path was applying a legacy 50% reduction that the approved architecture had reassigned to another subsystem. The oracle carried the bug. The candidate was right. Fourteen minutes from divergence to fix, and the failed cycle stays in the database as evidence.
Which makes me want to add a fourth item to your solutions list: verification tooling needs the same skepticism as implementation — especially now that agents make it cheap to build. A test oracle isn't trustworthy because it occupies the evaluator's chair. If Cv tooling is built with the same speed and laxity as the features it checks, the debt doesn't disappear; it migrates to the one layer nobody is auditing.
Looking forward to the agent legibility piece — persisting the composition of each decision (the why, not just the what) is what made this diagnosable in minutes instead of days, and it sounds adjacent to where you're heading.
This is what we're doing as well with our agentic coding workflows. We have worked to automate more and more of the verification.
Also, a lot of my time when creating tickets is figuring out the verification conditions to ensure that the agent implemented the features correctly.
One thing Ive noticed is that coding skills stay similar between projects but verification skills and tooling change between projects.
What trends have you observed with verification?
no obvious trends. The most important thing we’ve seen is giving the agent the ability to fully drive the box, whatever the box may be. For terminal apps that’s driving through tmux. For webapps, that’s playwright. For slack apps, we straight up created a dummy slack app in a dummy slack env that it can play with. It’s challenging but works well when done right
The Ci/Cv framing puts words on something I've been living for 18 months as a non-programmer running an agent-built trading system: my Ci is near zero by construction (I don't write the code), so the entire discipline lives in Cv. It works — but this week taught me a corollary your framework hints at without stating:
Verification debt can accumulate inside the verification layer itself.
I run new components in shadow mode before granting them any authority: the candidate computes decisions, persists them, and a comparator checks them against the live path. First full diagnostic cycle: 25 agents, 17 mismatches — every one exactly 2x. Expected 0.500 vs candidate 1.000, expected 0.425 vs 0.850, and so on.
My instinct said the candidate was broken. The evidence said otherwise. Because the comparator persists which rules composed each value (not just the final number), one query showed the expected-value path was applying a legacy 50% reduction that the approved architecture had reassigned to another subsystem. The oracle carried the bug. The candidate was right. Fourteen minutes from divergence to fix, and the failed cycle stays in the database as evidence.
Which makes me want to add a fourth item to your solutions list: verification tooling needs the same skepticism as implementation — especially now that agents make it cheap to build. A test oracle isn't trustworthy because it occupies the evaluator's chair. If Cv tooling is built with the same speed and laxity as the features it checks, the debt doesn't disappear; it migrates to the one layer nobody is auditing.
Looking forward to the agent legibility piece — persisting the composition of each decision (the why, not just the what) is what made this diagnosable in minutes instead of days, and it sounds adjacent to where you're heading.