A generated change can compile and pass its supplied tests yet fail the user’s actual requirement, a boundary condition, a permission check, or an operational constraint. DORA’s 2024 research associated greater AI adoption with modest improvements in reported code quality but lower delivery stability, reinforcing that local code indicators do not replace end-to-end outcome measures. DORA’s 2025 analysis found a positive association with throughput and product performance, while the negative association with stability persisted. These observational findings do not establish causation.

When acceptance criteria, business invariants, error behaviour, data constraints, or non-functional expectations remain implicit, both the generated implementation and its tests can agree with one another while still disagreeing with the product’s needs.

Agents can create tests as well as implementations, so both may share the same assumptions. A green suite is weak evidence when it only confirms the behaviour the agent already encoded. Security is one visible dimension of this gap: Veracode’s July 2025 controlled security benchmark found that 45% of generated samples introduced a known OWASP Top 10 vulnerability when no security-specific guidance was provided. The result is task-level evidence, not a production defect rate.

Pre-release tests cannot reproduce every combination of real workloads, third-party dependencies, data distributions, and user behaviour. If monitoring stops at infrastructure availability, a service may appear healthy while the business journey it supports is failing.

This creates a visibility gap: an API can return a successful status while a purchase, submission, or reconciliation is wrong. Without signals tied to user and business outcomes, teams may discover regressions through complaints rather than timely operational evidence.

Review becomes less reliable when an autonomous agent returns a large, difficult-to-understand patch or when approval is treated as a formality. The reviewer may lack the time or domain context to identify subtle effects across business logic, permissions, data and failure paths.

Maintainability is also part of quality. Duplicate implementations, unnecessary complexity, unclear error paths, and code without a clear owner increase the cost and risk of later changes. GitClear’s observational analysis of 211 million changed lines found copy-pasted lines rising from 8.4% in 2021 to 12.3% in 2024 while refactoring-associated moved lines fell; the study cannot isolate AI as the sole cause, but it supports measuring maintainability explicitly.

Lines generated, suggestions accepted, commits, or merged pull requests are easy to count but poor standalone measures of value. Higher output can coexist with more rework, escaped defects, review burden, and customer-impacting failures. DORA recommends choosing measures around the decision being made rather than optimizing a single number.

Measurements need context. A rise in reported bugs may reflect stronger detection; a shorter cycle time is valuable only if delivery stability and customer outcomes remain acceptable. Results should be segmented by task risk and codebase conditions, not used to rank individual developers or attribute causation to AI without supporting evidence.

Unlike a completion tool that suggests a small block of code, an agent may edit many files, run commands, update test fixtures, or continue over several steps. That creates specific failure modes: weakened tests can make a build green without fixing the defect; large diffs obscure review; stale context can carry forward an outdated requirement; and broad credentials can turn a development mistake into a production action. These risks call for controls over what the agent can change and execute, not just checks on its final code.

Identify the user journeys whose failure would most damage customers, revenue, safety, compliance, or trust. Map each to clear acceptance criteria and a layered test portfolio: unit tests for local logic, component and integration tests for interfaces, and end-to-end tests for critical journeys. Use exploratory, accessibility, performance, and resilience testing where automation cannot fully represent real conditions.

When an agent creates tests, verify that they independently encode the requirement, include negative and boundary cases, and fail when a plausible defect is introduced. Mutation testing or deliberately seeded defects can reveal tests that pass without protecting the intended behaviour. Treat test quality as evidence to inspect, not merely a count to increase.

For each critical journey, define service-level indicators (SLIs), such as latency, errors, saturation, and availability, with agreed service-level objectives (SLOs). Also define product indicators such as completed purchases, successful submissions, or task abandonment. Correlate logs, metrics, and traces so teams can connect a customer symptom to the responsible service and change. Protect sensitive data in telemetry and maintain actionable diagnostic runbooks.

Define release gates by change risk. Every change should pass agreed build, test, and automated analysis checks. Changes affecting payments, identity, permissions, sensitive data, or core business rules should receive deeper domain review and targeted integration or migration checks. Use feature flags and staged or canary rollout for consequential changes, with explicit success thresholds and stop conditions tied to both technical and business signals. Keep rollback procedures documented, authorised, and tested. Code rollback may not undo data changes or external transactions. Test backup restoration, use backward-compatible migrations where practical, and define how to reconcile or compensate for external actions.

Track change lead time and deployment frequency alongside escaped defects, change-failure rate, rework, review effort, recovery time, and successful completion of critical user journeys. Compare AI-assisted and non-assisted work only when task risk, codebase condition, and measurement coverage make the comparison meaningful.

Interpret measures as a portfolio. A short-lived increase in reported defects may reflect better detection, while a lower test-failure rate could mean fewer defects—or weaker tests. Segment by application risk and change size, and avoid using individual output metrics as proxies for engineering value.

Every agent-assisted change should have a named engineer who owns the result and a release owner for production decisions. Give reviewers a clear summary of intent, the complete diff, test evidence, residual risks, and recovery plan. Run agents in an isolated execution environment, using a dedicated branch and least-privilege credentials; do not give development agents direct write access to production systems or secrets. For multi-step tasks, require agents to re-check current requirements and repository state, work in small reviewable increments, and stop for approval before irreversible actions.

Treat tests, assertions, fixtures, snapshots, continuous integration (CI) workflows, coverage thresholds, and release configuration as protected evidence-producing controls. Flag removed or skipped tests, weakened assertions, broad mocks, and rewritten snapshots for separate human approval; where risk warrants it, use an independent verifier or trusted baseline tests. Constrain diff size and file scope so reviewers can understand the change. Maintainability checks should also examine duplication, unnecessary complexity, readability, reuse, dependency growth, and whether a team is willing to own the code long term.

Treat repository content, issue text, retrieved pages, and tool outputs as untrusted inputs that may contain malicious instructions. Enforce permissions and approval requirements outside the agent-editable workspace, so the agent cannot rewrite the controls that govern it.

Use approved AI providers and confirm their data-retention and model-training terms before sending code or business information. Limit transmitted data to what the task requires; exclude secrets and unnecessary personal data, and apply access and retention controls to prompts, logs, and outputs.

Worked example: an AI-assisted leave approval change — Hypothetical example: an agent changes a leave workflow that requires manager review followed by final HR approval. These controls illustrate VISTA rather than an implemented Astrik system.

Validate outcomes: independently test manager-to-HR routing, rejected requests, repeat submissions, and attempts to bypass either approval. Instrument for observability: record approval transitions without unnecessary personal details; alert on skipped steps, stuck requests, and duplicate actions. Set risk-based release gates: require domain and permissions review, test in an isolated environment, then stage the release with a stop condition for incorrect routing. Track quality and delivery together: compare completion time, routing errors, rework, and recovery results against the baseline. Assign human accountability: name the workflow owner and release approver, and test restoration and reconciliation of affected approval records.