TL;DR: Google’s The New SDLC With Vibe Coding — Day 1 (Updated in 2026) of the Kaggle 5-Day AI Agents Intensive — gets the central insight right: an AI coding agent is roughly 90% harness, 10% model, and that reframing alone will stop CTOs from buying the wrong thing. But the paper ends with “generation is solved” while burying the three constraints that actually bound production delivery — security (independent testing finds ~45% of AI-generated code carries exploitable flaws), maintainability (copy-pasted code rose from 8.3% to 12.3% of commits while refactoring collapsed), and a measured 19% productivity slowdown for experienced engineers that the paper cites but never reconciles. The kicker: the paper preaches “verification is the new craft” and then fails its own verification — a five-parts-that-become-four-parts error, a headline 90/10 number with no citation, and key statistics laundered through SEO blogs. Generation isn’t solved. Verification, security, and maintainability are the new craft — and none of them demo well.
Google’s Cloud team — Addy Osmani, Shubham Saboo, and Sokratis Kartakis — published a sharp, genuinely useful paper. As a firm that ships agentic AI infrastructure into regulated trading environments every day, we agree with most of its architecture. We’re writing this critique precisely because the paper is good enough to be dangerous: its blind spots map exactly onto the places where an engineering leader will burn six months and a security budget.
What does Google’s vibe coding paper actually get right?
Three ideas in this paper are correct and underappreciated, so we’ll bank them before we attack.
First: “Agent = Model + Harness,” and the harness is ~90% of the outcome. This is the most important idea in the document. The harness — instruction files (AGENTS.md, CLAUDE.md, GEMINI.md), tool and MCP wiring, sandboxes, orchestration, guardrails, observability — determines results far more than which frontier model you pick. The paper supports it with real benchmark movement: a Terminal-Bench team climbing into the top tier by changing only the harness, and LangChain lifting an agent’s score through prompt, tools, and middleware alone (LangChain).
Second: the spectrum framing. The paper resists the lazy binary of “vibe coding good / vibe coding bad” and instead lays out a continuum — vibe coding → structured AI-assisted development → agentic engineering — each appropriate to a different risk tier. That maps to how disciplined teams actually work.
Third: context engineering as a discipline. Six context types (Instructions, Knowledge, Memory, Examples, Tools, Guardrails) plus Agent Skills delivered via progressive disclosure is a genuine contribution. It is the correct mental model for feeding an agent.
So far, so good. Now the problem.
What is missing from Google’s vibe coding paper?
The paper’s structural flaw is an incentive-shaped omission — our first named mental model, incentives drive behaviour. Google sells the generation layer: models, Cloud compute, and the IDE surfaces where tokens are produced. A paper from a generation vendor will, predictably, frame generation as the solved problem and treat the rest as “craft” to be figured out later. That isn’t dishonesty; it’s gravity. The paper even steers explicitly toward Google’s own stack — ADK, A2A, Jules, Gemini Code Assist, Gemini CLI — which is fine, as long as you read it knowing where it’s standing.
Gap 1 — Security is one passing mention, and it’s the whole ballgame
The paper touches security exactly once with substance — a single “Security Remediation” bullet — and a one-clause aside telling you to “check imports for real packages” (its only nod to slopsquatting). In production, security is the constraint. The data is brutal:
- Veracode’s 2025 analysis ran 80 standardized tasks across more than 100 models and found 45% of AI-generated code introduced a known vulnerability; cross-site-scripting defenses (CWE-80) failed in 86% of cases and log-injection (CWE-117) in 88% (Veracode). Java fared worst at over 70% failure.
- Veracode’s Spring 2026 follow-up is the part that should terrify a CTO: syntactic correctness climbed above 95%, but the security pass rate stayed flat at ~55% (Veracode). Models got better at writing code that runs and no better at writing code that’s safe.
- The Cloud Security Alliance’s 2026 research note, drawing on Apiiro data, found AI-assisted developers commit 3–4x faster but introduce security findings at roughly 10x the rate, with privilege-escalation paths up 322% and architectural design flaws up 153% (CSA).
This is the map-vs-territory model — our second named mental model. Benchmarks and demos are the map; production is the territory. “It passed the tests” is map. “It’s not exploitable” is territory. The paper optimizes for the map.
Example: In a regulated trading system, an LLM happily generated a position-sizing endpoint that passed every unit test and leaked the entire order book through an unparameterised query string. Functionally correct. Catastrophically insecure. No demo would ever catch it — because the demo is the map.
The paper’s own framing makes this worse. It celebrates the “keep prompting until it works” loop, but iterative prompting optimises toward passing tests, not toward being safe — and the two diverge precisely on the inputs no test enumerates.
Gap 2 — Maintainability and the silent debt explosion
Generation speed creates a downstream tax the paper never books. GitClear’s analysis of 211 million changed lines (2020–2024) found copy/pasted code rose from 8.3% to 12.3% of commits while moved/refactored code — the signature of healthy engineering — collapsed from ~25% to under 10%; code churn (lines reverted within two weeks) climbed from 3.3% to 5.7%, and 2024 was the first year copy/pasted code exceeded moved code (GitClear). DORA’s 2024 research found a 7.2% decrease in delivery stability for every 25% increase in AI adoption.
Translation: agents write code that works today and rots faster. They duplicate instead of abstract, because duplication is locally cheaper for a stateless generator that doesn’t carry the cost of the eventual refactor. The paper’s economics section celebrates a “high CapEx / low OpEx” profile for agentic engineering but never prices in the maintenance OpEx of churn debt — which is the OpEx that actually kills you.
Gap 3 — The 19% slowdown the paper cites but won’t reconcile
The paper cites METR, then walks past it — and how it cites METR is itself a tell. The famous “19% slower” finding comes from METR’s July 2025 randomized controlled trial: 16 experienced open-source developers, 246 real tasks, mature repositories (averaging 22k+ GitHub stars and over a million lines of code), using then-current frontier tooling. Result: developers were 19% slower with AI — having predicted they’d be 24% faster, and still believing afterward they’d been ~20% faster (METR). That is a ~40-point gap between measured and perceived productivity.
But the paper’s endnote points instead to METR’s February 2026 “Uplift Update.” Read that update and it does not cleanly reverse the result: the original developer subset still showed an 18% slowdown (confidence interval −38% to +9%), and a newly recruited cohort showed a 4% slowdown (METR). Citing the later, softer-sounding post for a claim that originates in the harder RCT is exactly the kind of date-drift a verification-first culture is supposed to catch. A paper announcing a “new SDLC” owes its readers a reconciliation of a measured 19% slowdown against its own “25–39% productivity improvement” claims. It never attempts one.
The honest synthesis: AI accelerates greenfield and boilerplate, and decelerates expert work in large, high-context, correctness-critical codebases — which is exactly where institutional money is made and lost.
Gap 4 — The paper fails its own verification standard
Here is the edge nobody is naming, and it’s the most damning thing in this critique. The paper’s thesis is that generation is easy and verification is the new craft. Yet the paper itself is a verification failure on multiple counts — it is, in effect, vibe-written:
- Five parts that silently become four. The paper states every agent “is built from five parts,” lists Model, Tools, Memory, Orchestration, and Deployment — then in the very next breath says “these four parts work together in a continuous loop.” Five, then four. A reviewer reading for correctness catches this in ten seconds; the paper shipped it.
- The headline 90/10 number has no citation. The “Model ~10% / Harness ~90%” split — the single most quoted claim in the document — appears in a figure with no source, no methodology, and no defined denominator. Ninety percent of what? Outcome variance? Engineering effort? Token cost? The number is doing enormous rhetorical work on a foundation of zero.
- An economics chart with no axes. The figure asserting that vibe coding “costs 3–10x more per feature” before an agentic-engineering crossover is drawn as a curve with no numeric axes — the labels are literally “Cumulative Total Cost of Ownership” and “Time / Features Shipped” — and undefined “12%” boxes. There is no TCO math behind the TCO chart.
- Laundered statistics. The eye-catching adoption figures (85% / 51% / 41%) trace back through the endnotes not to primary surveys but to SEO marketing blogs. Several other endnotes resolve to the lead author’s own website. For a document whose whole point is “verify your sources,” that’s the inversion writing itself.
This isn’t pedantry. It’s the inversion model in action: the fastest way to evaluate a claim that “verification is the new craft” is to verify the claim itself. It doesn’t survive its own test. The paper proves its thesis by failing to follow it.
Gap 5 — “Where to start” without a reproducible harness playbook
The paper correctly says the harness is 90% of the result, then offers no concrete, reproducible recipe for building one. That’s like telling a fund “execution is everything” and naming zero execution venues. The paper is also honest about the “80% problem” — agents nail roughly 80% of a task, and the last 20% (edge cases, integration, correctness) needs humans, with errors that “look right” and “may even pass basic tests.” Good. But it stops at naming the problem. So here’s the chapter it left out.
How do you actually build the harness Google describes? (Our production playbook)
When we deployed agentic coding across RocketEdge’s trading-infrastructure stack, seven practices moved the needle more than any model upgrade. This is the missing chapter.
1. The harness effect is real and brutally model-agnostic — prove it to yourself. The same model behaves like a different model depending on the harness wrapping it. In our experience, Claude Opus 4.8 is markedly weaker inside GitHub Copilot in Visual Studio Enterprise than the identical model in VS Code — and the dominant cause is the different system prompts each surface injects into the context window. Same weights, same API, different scaffold, materially different code. The research agrees: same-model / different-scaffold swings of 17–22 points are larger than the gap between most frontier models (benchmarks landscape). The practical punchline for a CTO: your IDE choice can cost you more capability than a model downgrade. Stop arguing about models. Audit your harness.
2. Put the spec in the repo, not in the prompt. We add a docs/ directory containing a detailed codebase spec, the database schema, implementation justifications, architectural diagrams, and — critically — decision records explaining why we chose X over Y. This is durable, version-controlled context the agent reads on every run, not ephemeral context that dies with the chat session. The paper’s “Knowledge” context type stays abstract; this is the concrete instantiation. Architecture Decision Records stop the agent from “fixing” a deliberate choice it doesn’t understand.
3. Document each major method and class — for the LLM, not just the human. In an agentic codebase, the docstring is no longer courtesy; it’s a context-injection mechanism. A well-documented method tells the agent the contract, the invariants, and the failure modes before it touches a line — which directly attacks the 80% problem by front-loading the 20% the agent would otherwise guess at.
4. Encode the test cases manually, first — including security invariants as tests. Before the agent generates implementation, we hand-write the tests that must pass. The paper gets this half-right in one bullet (“write the tests and evals before generating the code… they are the contract with the AI”). We go further: we encode security invariants as tests, so the failure mode in Gap 1 is inverted — instead of “generate, then hope it’s secure,” correctness and safety become the executable specification the agent is forced to satisfy.
# repo/docs/test_first/test_position_sizing.py
# Encode the contract MANUALLY before the agent writes a line of implementation.
def test_position_size_never_exceeds_risk_budget():
book = OrderBook(nav=2_000_000, max_risk_pct=0.02)
size = position_size(signal=0.83, price=101.25, book=book)
assert size * 101.25 <= book.nav * book.max_risk_pct # hard risk cap
assert size >= 0 # no negative sizing
assert not leaks_order_book(size) # security invariant AS a test5. Keep Agent Skills high-level, not prescriptive. Counterintuitively, the most useful Skills describe what good looks like and the constraints, not the detailed solution. The model almost always knows a better concrete implementation than the one we’d hand-write; over-specifying a Skill caps the agent at our cleverness instead of the frontier’s. Give it the goal and the guardrails, not the algorithm.
6. Wire two MCPs the paper never names. Use Perplexity MCP for research during design, planning, and bug-fixing — pulling current, cited domain knowledge into the agent’s context (MCP, the Model Context Protocol, is the now-industry-standard open protocol from Anthropic — Anthropic). Use Context7 MCP to inject version-specific, up-to-date library documentation, which kills the single most common agentic failure: confidently hallucinated or outdated API calls (Context7).
7. Write the plan with one model, red-team it with two others. We have one model (say, Claude Opus) draft plan.md, then route that plan to a different lineage — GPT-5.5 and Gemini Pro — with a single instruction: find the gaps, omissions, and boundary cases. Models share blind spots within a family; cross-lineage review is the cheapest verification layer in the entire stack. This is how you operationalise “verification is the new craft” instead of merely asserting it — and it’s the security-review step the paper omits entirely. Generated code goes through the same adversarial pass: a second model reviews the first model’s output specifically for injection, secrets handling, authZ, and dependency provenance before a human signs off on the last 20%.
Alternative Perspectives
The contrarian read: “Generation is solved” is a category error. The consensus — and the paper’s final line — treats generation as finished and verification as the frontier. We’d argue the two are not separable: code that’s ~45% insecure isn’t “generated and awaiting verification,” it’s not generated correctly. Calling generation solved while half the output is exploitable is like calling a backtest “solved” because it produced a number — ignoring whether the number survives out-of-sample. Generation is solved the way a backtest is profitable: only until reality runs.
The emerging counter-view: the slowdown is temporary and harness-shaped. A fair challenge to our own critique: METR tested 2025-era tooling and developer habits. As engineers learn agent-native workflows and harnesses mature, the slowdown may invert — and METR’s 2026 update, with its smaller measured drag, is weak evidence in that direction. We partly agree, and that agreement is the argument: the slowdown is a harness-maturity problem, not a model-capability problem. Which is exactly why leaders should pour investment into the harness, not the next model release.
Why is this happening now? The bigger picture
Two forces, one secular and one cyclical, are colliding. The secular force is real and permanent: the marginal cost of producing a line of code is collapsing toward zero. That won’t reverse. The cyclical force is a classic hype overshoot — the recurring gap between demo-grade generation and production-grade systems that the market forgets and painfully relearns. We saw it with no-code, with offshoring, and with the low-latency arms race.
The historical precedent is the CASE tools wave of the late 1980s and early 1990s. Computer-Aided Software Engineering promised to generate code from diagrams and largely failed — not because generation didn’t work, but because the generated systems were unmaintainable and integration plus verification swamped the savings. The analogy has limits: today’s models are vastly more capable, generalise across domains, and improve monthly, so this is not a clean rerun. But the shape of the failure — generation cheap, verification and maintenance expensive, total cost underestimated — is identical. The bottleneck moved; it didn’t disappear.
That’s our third named mental model: the theory of constraints. Speeding up generation when verification is the binding constraint doesn’t increase throughput — it piles up work-in-progress inventory (unreviewed, possibly-insecure code) faster than the system can safely ship it. Optimising a non-bottleneck is, by definition, an illusion of progress.
What should engineering leaders actually do?
The stated problem is “how do we adopt AI coding.” The actual problem — the intent diagnosis — is “how do we adopt AI coding without trading a visible velocity gain for an invisible security-and-maintenance liability that never shows up on the velocity dashboard.” Three moves:
- Invest in the harness, not the model. Standardise
AGENTS.mdand ADRs, wire MCP tooling (Perplexity for research, Context7 for live API docs), and measure harness-driven score swings before debating model vendors. Remember: a worse IDE can cost you more than a worse model. - Make security and correctness executable. Test-first, security invariants encoded as tests, cross-lineage plan review, and mandatory human sign-off on the last 20% — the edge cases the paper itself admits agents fail.
- Instrument the debt. Track churn, duplication, and delivery stability (DORA metrics), not just throughput. Velocity that quietly raises churn is borrowing against the codebase, not earning.
Two questions I’d put to any leader rolling this out — because strategy lives or dies on them. First: if your only adoption metric is throughput, how would you ever detect the security-and-churn liability before it detonates in production? Second: are you investing in the model (the 10%) because it’s measurable and procurable, while neglecting the harness (the 90%) because it’s organisational and hard?
FAQ
Is Google’s vibe coding paper worth reading?
Yes. Its “Agent = Model + Harness” framing and context-engineering taxonomy are the best concise treatment available, and they’ll stop you over-indexing on model selection. Read it for the architecture; treat “generation is solved” as a vendor’s optimism, not a finding.
Is AI-generated code actually secure enough for production?
Not without a verification harness. Independent testing shows roughly 45% of AI-generated code carries exploitable flaws, and a 2026 follow-up found the security pass rate stuck near 55% even as syntactic correctness passed 95% (Veracode). Treat every agent output as untrusted input until it passes security tests, cross-model review, and human sign-off.
Does AI coding make experienced developers faster?
Not reliably in large, mature codebases. METR’s RCT measured a 19% slowdown for experienced developers on real tasks, despite those developers believing they were faster (METR). AI clearly accelerates greenfield and boilerplate; expert work on complex, correctness-critical systems is the exception, not the rule.
What is the “harness effect” in one sentence?
The same model produces materially different results depending on the system prompts, tools, and scaffolding wrapping it — swings larger than the gap between frontier models — which is why an agent is best understood as ~90% harness and ~10% model, and why your IDE choice can cost you more capability than a model downgrade.
Why use Perplexity MCP and Context7 MCP specifically?
Perplexity MCP injects current, cited research into design and debugging; Context7 MCP injects version-specific library docs that eliminate hallucinated or outdated API calls — the two most common context failures in agentic coding, and two tools the Google paper never names.
About RocketEdge. RocketEdge builds AI-powered trading infrastructure for institutional and professional traders — the MultiEdge AI Signal Fabric (signal-as-a-feature API for quant models and trading agents), the MultiEdge Agentic Research Platform (five specialised agents that cut investment-research time 10–15x), and the AI Trade Idea Generator (autonomous, CPCV-validated strategy discovery). We don’t just write about agentic systems; we run them in production under regulatory constraint. More at rocketedge.com.
If you’re adopting agentic coding across a quant or trading-infrastructure team and want to build the verification harness before the velocity bites you, book a 30-minute strategy call.
Disclaimer: This article discusses software-engineering practices and references third-party benchmarks and studies. Productivity figures, security statistics, and code-quality metrics are drawn from the cited public research and reflect those studies’ specific conditions. Observations about specific tools reflect our own production experience and are stated as opinion. Nothing here is investment advice or a guarantee of any engineering, security, or trading outcome. Backtested or illustrative figures do not predict future results.