Essay · 9 min read

What 'governance' means when AI writes your software — 1729 AI Labs

September 22, 2025

1729 AI Labs

Every CIO evaluating AI development asks some version of the same question: “How do we know the software is right?”

This is the correct question. But it is almost always aimed at the wrong target.

The question gets framed as an AI problem — concerns about hallucinated logic, untested edge cases, code that no human reviewed before it reached production. These concerns are not without merit. But they describe a failure mode that exists with or without AI. Organizations were shipping ungoverned, under-validated software long before large language models could write a for-loop.

The governance failure in most enterprise software development isn’t new. AI didn’t create it. It did, however, make it harder to ignore.

What governance looks like in practice

Ask any architect at a mid-market company to show you the governance process for their current backlog. What you will typically find:

Requirements documents that nobody reads. A Word document or Confluence page that was accurate on the day it was written, that nobody has touched since, and that the engineers who implemented the feature have never opened. The implementation reflects what the engineer understood in a ticket and two Slack threads, not what the document says.

Architecture decisions in ephemeral channels. The reasoning behind a critical design choice — why this database, why this API pattern, why this service boundary — lives in a Slack conversation that is already archived and unsearchable. The decision is invisible to anyone who joins the project six months later.

Testing as a final event. Validation happens at the end, after the code is written and the gap between intent and delivery has had the entire development cycle to compound. By the time QA finds a mismatch between what the business wanted and what was built, fixing it means unraveling months of downstream work.

Sign-off as theater. Change advisory boards, compliance reviews, formal sign-offs — these processes exist, but they typically review what was built, not whether it matches what was intended. They catch the symptoms of governance failure, not the causes.

This is the baseline. This is what “governed software development” looks like in most organizations today, AI or not.

What AI actually changes

AI coding tools accelerate the delivery of what was specified. This is useful. It is also clarifying.

When code generation was slow, the gap between specification and delivery was obscured by calendar time. A requirements failure that took twelve months to manifest felt like a project management problem, a scope problem, a resourcing problem. The cause was legible only in retrospect.

When code generation is fast, the gap between specification and delivery manifests faster. The same requirements failure that used to take a year to become visible can now surface in a week. AI doesn’t create the failure mode — it compresses the feedback loop.

This is why the CIO’s concern, while pointed at AI, is actually pointing at something real: the absence of structured requirements management. The risk isn’t “AI wrote ungoverned code.” The risk is “the specification was never precise enough to govern against in the first place.”

Governance by construction

There is a meaningful distinction between governance as a review activity and governance as a structural property of the process.

Governance as review: An external step that examines outputs after they’ve been produced, looking for deviations from standards. This is how most enterprise governance works — code review, security review, compliance review, CAB approval. It catches some failures. It misses the ones that were specified incorrectly to begin with.

Governance by construction: A process in which the properties you care about — traceability, auditability, requirement-to-delivery correspondence — are structural outputs of the process, not results of external review. You don’t check for traceability at the end; the process cannot produce a deliverable without it.

A software factory built around structured requirements management produces governance as a structural property. Every requirement is captured in machine-readable form before code is written. Every delivered component is validated against the requirement it implements. Every decision point — when a requirement changes, when a conflict is resolved, when a design choice is made — is logged. The audit trail isn’t assembled after the fact; it’s a byproduct of the delivery process itself.

This is not a claim about AI. It is a claim about process architecture.

The right questions for the briefing

When evaluating an AI development offering for enterprise use, the governance conversation should start with these questions:

Where are requirements formalized? In a document someone wrote once, or in a structured representation that drives the build process? If it’s a document, the first risk is already present: the document will be interpreted, not executed.

When is validation against requirements performed? At the end of the project, or continuously throughout the build? Late validation catches the symptoms of specification failure; continuous validation catches the cause.

What does the audit trail cover? Is there a record that links every piece of delivered code to the requirement it implements? Is there a record of every requirements change, when it was made, and what was rebuilt as a result? Without this record, the audit is reconstructive — you’re working backward from outputs to try to infer intent, which is exactly what you want to avoid in a regulated environment.

Who can see the specification? If the requirements are formalized in a way that both technical and non-technical stakeholders can review, disagreements surface early and resolve cheaply. If the requirements live only in technical artifacts, the business doesn’t discover the gap until it’s expensive to close.

These questions apply to any development process. They are not AI-specific. But they are the right filter for evaluating whether an AI-assisted process can meet enterprise governance standards — or whether it’s trading speed for accountability.

The governance gap is already there

The CIO who is skeptical of AI-built software is right to be skeptical. But the skepticism should be directed at the process, not the technology.

A traditional SDLC with imprecise requirements, invisible architecture decisions, late validation, and nominal sign-off processes does not produce governed software. It produces software with a governance story — a set of artifacts and approvals that can be presented to auditors, but that don’t reliably correspond to what the business intended.

A factory model that formalizes requirements before code is written, traces every component to its source requirement, validates continuously against specification, and produces a complete audit trail as a structural output of the process — that model produces genuinely governed software. It happens to use AI to generate the code. But the governance properties come from the process architecture, not from the code generator.

The question for the board isn’t “did a human write this code?” It’s “can you show me that this system does what the business intended it to do, and how you verified that?”

If the answer is yes — if you can produce the requirement, the implementation, the validation record, and the audit trail — then the governance question is answered. If the answer is no, then the software is ungoverned regardless of how it was written.

What to ask before the next AI development engagement

Before signing a contract with any software development firm — AI-assisted or traditional — ask for three things:

Show me how a business requirement becomes a formalized specification before code is written.
Show me how a delivered component is validated against its source requirement.
Show me the audit trail for a completed engagement.

Any firm that can answer all three questions with evidence, not narrative, has addressed the governance concern. Any firm that responds with process diagrams and compliance certifications but can’t show you the actual artifact chain has not.

The governance concern is correct. It just isn’t about AI. It’s about requirements — and whether the process you’re buying treats them as a first-class engineering problem or an administrative formality.

1729 AI Labs builds software through a factory process in which every requirement is traced from first statement to shipped code. If you’re evaluating AI-assisted development for an enterprise engagement, request a factory briefing.