Automation Assessment

The Automation Assessment measures how much of your delivery path runs without a human in the loop, and how much signal that path gives you when something breaks. The questions cover orchestration, automated code review, testing, version control and branching, deployment strategy, feature toggles, observability, environment provisioning, and all four DORA metrics. Below is a breakdown of each scored question: what it measures, why it matters, and concrete steps to improve if your team scored low.

Score Interpretation

How to read your total maturity score (0 to 100).

1 to 20: Struggling. Delivery depends on people remembering steps. Builds and deployments are triggered by hand, testing is largely manual, and production problems are usually reported by users first. Start with version control hygiene and one automated build that runs on every commit.

21 to 40: Developing. A pipeline exists but it is partial and optional. Some checks run, some environments are automated, and the last mile to production is still manual. The fastest gains are making tests run on every commit and automating one complete path to production.

41 to 60: Norming. Automation is real and reliable for the common case. Tests and code review checks run automatically and most releases go through a pipeline. The next gains come from deployment strategy, rollback, and post-deployment validation, so a bad release stops being an event.

61 to 80: Performing. The pipeline is trusted and the team deploys on its own. Focus shifts to reducing batch size, tightening lead time, broadening test types, and putting security scanning and environment provisioning on the same automated footing as the rest of the pipeline.

81 to 100: Thriving. Delivery is a capability rather than a project. Changes flow to production daily with automated validation, fast rollback, and observability that detects issues before customers do. Protect it by watching change failure rate as you increase speed.

Orchestration Visibility

What level of visibility and control do you have over your orchestration processes?

Why it matters: When pipeline state lives in scattered logs and manual status checks, nobody can answer basic questions quickly: which build is deploying right now, what changed since the last good run, who approved the promotion. Teams in that position debug delivery by asking around in chat, and incident timelines get reconstructed from memory. Visibility is also what makes automation safe to trust, because an operator who cannot see a pipeline will eventually route around it.

To improve your score: Stand up one place where every pipeline run is visible with its commit, artifact version, target environment, duration, and outcome. Emit structured events from each stage rather than relying on console output, and keep an audit trail of who triggered and who approved each promotion. Add alerting on failed and stuck runs so the pipeline tells you it is broken instead of a developer noticing an hour later.

Orchestration Platform

Does your team use an orchestration platform like Microsoft DevOps or Jenkins to orchestrate your CI/CD pipeline?

Why it matters: Homegrown shell scripts on a build box work until the person who wrote them leaves or the box dies. A real orchestration platform gives you versioned pipeline definitions, credential management, agent pools, retries, and a shared execution history that survives staff turnover. Without one, delivery capability is concentrated in a few people and cannot be reproduced when they are unavailable.

To improve your score: Pick one platform and move your highest traffic service onto it first, so the pattern gets proven under real load. Define pipelines as code in the repository they build rather than as configuration clicked into a user interface, so pipeline changes are reviewed like any other change. Publish a shared template other teams extend, then retire the improvised scripts instead of leaving both paths alive.

Cross Environment Orchestration

To what extent are your CI/CD workflows orchestrated across environments such as dev, test, and prod?

Why it matters: Manual promotion between environments is where drift enters. A change validated in test reaches production with a slightly different configuration, a missed migration, or a rebuilt artifact, and the failure looks like a code problem when it was a process problem. Manual transitions also make lead time unpredictable, because promotion waits on whoever happens to be available.

To improve your score: Promote the same immutable artifact through every environment and inject configuration at deploy time rather than rebuilding per environment. Model the full path from commit to production as one pipeline with explicit stages, so each environment is a step rather than a separate job someone starts by hand. Where a stage genuinely needs a human decision, make it an approval gate inside the pipeline instead of a handoff outside it.

Automated Code Review Integration

How integrated is automated code review, such as linting, static analysis, and policy checks, into your CI/CD pipeline?

Why it matters: Human reviewers are expensive, and they are weakest at exactly what machines do well: formatting, unused variables, null handling, known anti patterns, and license or policy violations. When those checks are optional or manual they get skipped under deadline, and reviewers spend their attention on style instead of design. Inconsistent enforcement also means code quality depends on who happened to review.

To improve your score: Run linting and static analysis automatically on every pull request and publish results as annotations on the diff so authors see them in context. Start in warning mode for two weeks to clear the existing backlog of findings, then flip the highest value rules to blocking. Keep the check fast, ideally a couple of minutes, because a slow gate is the one people learn to bypass.

Code Review Feedback and Enforcement

What level of feedback and enforcement does your automated code review system provide?

Why it matters: Feedback that only reports style violations trains people to ignore the tool. Feedback that is actionable, points at the exact line, explains the risk, and suggests a fix gets acted on. Enforcement closes the loop, because a finding with no gate behind it is a suggestion, and suggestions lose to delivery pressure every time.

To improve your score: Tune the rule set so signal beats noise. Delete rules that fire constantly without changing behavior and keep the ones tied to real defects and security risk. Enable automatic fixes for mechanical issues like formatting and import order so those never reach a reviewer. Then set quality gates on what matters, such as coverage and critical severity findings, and gate on new code rather than the whole repository so the team is not blocked by legacy debt.

Unit Test Coverage

What percentage of your codebase is covered by automated unit tests?

Why it matters: Coverage is not quality, but low coverage is a reliable proxy for fear. Below 25 percent, engineers cannot refactor safely, so the design ossifies and every change carries manual regression cost. Coverage also determines how much your pipeline can tell you. A green build on a thinly tested service is a statement about compilation, not about correctness.

To improve your score: Stop chasing a repository wide number and set a coverage floor on new and changed code instead, which most quality gates support directly. Backfill tests around the modules that generate the most production incidents and the ones you plan to change next, not the easiest ones. Make writing a failing test the first step of every bug fix, which raises coverage exactly where defects actually live.

Test Execution in the Pipeline

How are unit tests integrated into your CI/CD pipeline?

Why it matters: Tests that run only when someone remembers are not a safety net. The value of automated tests comes from running them on every commit, so a break is attributed to one small change instead of discovered a week later inside a merge of twenty. Optional test execution also erodes the tests themselves, because nobody notices when they rot.

To improve your score: Run the unit suite on every push and every pull request, and make a failing suite block the merge. Keep that fast suite under ten minutes by running it in parallel and pushing slower checks to a later stage. Alert on failures in the main branch and treat a red main branch as the team’s top priority until it is green, because a tolerated red build removes the signal entirely.

Test Maintenance

How frequently are your automated tests reviewed and updated to reflect code changes?

Why it matters: Test suites decay. Flaky tests get retried, then muted, then forgotten, and each one that goes quiet removes coverage nobody notices losing. Teams that only touch tests at release time end up with a suite that fails for reasons unrelated to the change in front of them, which is how engineers learn to ignore red.

To improve your score: Treat test changes as part of the definition of done for every story so tests move with the code rather than trailing it. Track flaky tests explicitly with a quarantine list, a named owner, and a deadline, instead of quietly adding retries. Review skipped and muted tests on a fixed cadence and either fix them or delete them, since a permanently skipped test is worse than no test because it implies coverage that does not exist.

Test Result Visibility

Are test results automated, visible, and accessible to the entire team?

Why it matters: When results live in one engineer’s terminal or behind a login nobody has, quality becomes a private matter. Shared visibility is what lets a product owner understand why a release is held, lets a new engineer see which areas are fragile, and lets the team notice that the same test has failed intermittently for a month.

To improve your score: Publish test results as a pipeline artifact and surface pass rate, duration, and failure trend on a dashboard the whole team can reach without hunting for credentials. Post build outcomes into the team channel with a direct link to the failing test, not just a red icon. Show coverage and test status on the pull request itself so the information arrives where the decision is made.

Branching Strategy

How consistently does your team use branching strategies, such as Git Flow or trunk-based development, to manage code changes?

Why it matters: Branching strategy sets your integration frequency, and integration frequency sets your merge pain. Improvised branching produces long lived branches that diverge for weeks, and the merge at the end is where defects and delays cluster. It also makes the pipeline ambiguous, because nobody can say with confidence which branch is deployable.

To improve your score: Agree on one strategy and write it down, including branch naming, who merges, and how long a branch may live. Cap branch lifetime at a few days and slice work so that is realistic, using feature toggles to keep incomplete work merged but dark. Enforce it with branch protection rules requiring passing checks and review before merge, so the standard is a property of the repository rather than a habit people have to remember.

Version Control Integration

How is version control integrated into your development and deployment workflows?

Why it matters: Version control used only for backup wastes its real value, which is traceability. When commits link to work items and pipelines link to commits, you can answer what shipped, when, why, and what it touched, in seconds. Without those links, release notes are assembled by hand, audits become archaeology, and rollback decisions get made without knowing what is actually in the build.

To improve your score: Trigger builds from repository events rather than from schedules or manual clicks. Require a work item reference on every commit or pull request so changes trace back to intent, and let the pipeline generate release notes from that history. Tag every production release in the repository with the artifact version so any deployed build can be traced back to an exact commit.

Automated Deployment Pipelines

How frequently does your team use automated deployment pipelines for production releases?

Why it matters: Manual production releases are slow, unrepeatable, and stressful, which is why they get batched into large risky events. Every manual step is a place where a tired person late at night does something slightly different from last time. Automation is what makes a deploy boring, and boring deploys are what let a team release small changes often.

To improve your score: Automate the highest frequency path first and keep the manual runbook only as an emergency fallback. Write down every step of the current manual release, then convert them one at a time until the runbook is a single pipeline trigger. Set a standard that every production release goes through the pipeline, and track exceptions publicly so the manual path does not quietly persist.

Deployment Strategies

Which deployment strategies are actively used in your production environment?

Why it matters: Direct deployment makes every release all or nothing, and the blast radius is every user. Rolling updates reduce downtime but still expose everyone to a bad build eventually. Blue green and canary change the economics of risk, because you can expose a change to a small slice of traffic, watch real signals, and then decide with evidence rather than hope.

To improve your score: Start with canary on one service where you already have good metrics, because a canary without observability is just a slower outage. Define the promotion criteria before you start: error rate, a latency percentile, and one business metric, each with a threshold and a watch window. Bake the strategy into the pipeline so it is the default path rather than a special procedure reserved for scary releases.

Rollback Handling

How is rollback handled in your deployment strategy?

Why it matters: Recovery speed is mostly a function of how quickly you can return to a known good state. Manual rollback with downtime means every incident includes an argument about whether to roll back or fix forward, and that argument usually costs more minutes than the rollback would. Teams without a rehearsed rollback tend to fix forward under pressure, which is how a small incident becomes a long one.

To improve your score: Make rollback a single command or a traffic switch, not a redeploy from an older branch. Keep the previous artifact and its configuration ready, and make database changes backward compatible using expand and contract migrations so reverting code does not require reverting schema. Rehearse it in a lower environment and time it, then wire automatic rollback to your canary thresholds so the system reverts before a human joins the call.

Feature Toggle Usage

How frequently does your team use feature toggles to control feature rollout in production?

Why it matters: Toggles decouple deploy from release, which is what lets a team merge continuously without shipping half finished behavior to customers. Without them, incomplete work sits on a branch, integration is deferred, and the release date becomes the risk date. With them, code reaches production early and dark, and the business decides when it turns on.

To improve your score: Add a toggle to the next feature that would otherwise need a long lived branch, and enable it for internal users first. Keep the toggle check at one entry point rather than scattered through the code, so removing it later is a small change. Agree that a release toggle is temporary by default and gets deleted within a defined window after full rollout.

Feature Toggle Management

How are feature toggles managed and monitored across environments?

Why it matters: Toggles nobody tracks become permanent hidden configuration. Each stale toggle doubles the code paths under test, and a handful of forgotten ones make production behavior genuinely unpredictable, especially when a flag is on in one environment and off in another. The failure mode is a bug that reproduces for only some users and nobody can explain why.

To improve your score: Move toggles into a central service with per environment values and an owner, a created date, and an expiry date on every flag. Log and expose which flags are on in production, review the list on a regular cadence, and delete anything past its expiry. Separate short lived release toggles from long lived operational switches and permission flags, because they have different lifecycles and different removal rules.

Post Deployment Validation

How is post-deployment validation performed after a release to production?

Why it matters: A successful deploy and a working system are two different claims. Pipelines routinely report green after shipping a build that cannot reach its database or is serving errors to a subset of users. Without validation, the gap between shipping and knowing gets filled by customers, and your time to detect is measured in support tickets.

To improve your score: Run an automated smoke suite against production immediately after deploy that exercises the critical paths end-to-end, including authentication and one real transaction. Compare error rate and latency against the baseline from before the deploy for a defined watch window, and fail the release if it degrades. Wire that check to your rollback trigger so validation decides something rather than merely recording it.

Issue Detection and Response

What mechanisms are in place to detect and respond to issues after deployment?

Why it matters: Relying on user reports means your detection time equals your customers’ patience, and it means you learn about problems from the people you least want to hear it from. Logs alone help only if someone is looking. Detection capability is the single biggest lever on time to restore service, because you cannot fix what you have not noticed.

To improve your score: Define service level objectives for your most important user journeys and alert on symptoms users feel, such as error rate and latency, rather than on CPU. Add distributed tracing so a failing request can be followed across services instead of correlated by timestamp. Route alerts to a clear on call owner with a runbook link attached, and review every alert that fired without action so the signal stays trustworthy.

Deployment Independence

How independently can your team deploy code to production without relying on a separate deployment or operations team?

Why it matters: Every handoff to another team adds queue time you do not control, and queue time is usually the largest share of lead time. Handoffs also separate responsibility from consequence, since the team that wrote the change is not the team that watches it land. Teams that cannot deploy on their own batch changes to reduce the number of requests they have to make, which makes each release larger and riskier.

To improve your score: Move toward a platform model where the operations group provides paved paths, guardrails, and the pipeline, and the product team owns the trigger. Encode approvals as automated policy checks inside the pipeline rather than as tickets, keeping human approval only where a regulation genuinely requires it. Give the team production access matched to their responsibility, including the ability to roll back, and pair on the first few releases to build confidence.

Deployment Frequency

How frequently does your team deploy code to production?

Why it matters: Deployment frequency is one of the four DORA metrics and it is the clearest read on batch size. Monthly releases mean large batches, and large batches mean a failure is hard to attribute and expensive to unwind. Frequent deployment is not speed for its own sake. It is what keeps each change small enough that risk and diagnosis stay manageable.

To improve your score: Attack whatever forces batching, which is usually a manual test window, a change advisory board that meets weekly, or a shared environment only one team can use at a time. Slice work into smaller vertical increments and use toggles so partial work can ship safely. Set a visible target, such as moving from monthly to weekly, and remove one blocking step per iteration rather than trying to fix the whole pipeline at once.

Lead Time for Changes

How long does it typically take for a code change to go from commit to production?

Why it matters: Lead time is the second DORA metric and it measures the responsiveness of your delivery system. Long lead time means feedback arrives late, so you learn a feature missed the mark after you have built three more on top of it. It also raises the cost of urgent work, because a security patch moves at exactly the same speed as everything else.

To improve your score: Map the path from commit to production and record wait time separately from work time. The wait almost always dominates, sitting in code review queues, waiting for a test environment, or waiting for a scheduled approval. Attack the largest wait first with review response expectations, environments available on demand, or automated policy checks replacing scheduled approvals, then measure again to confirm the change worked.

Time to Restore Service

How long does it typically take to recover from a failure in production?

Why it matters: Time to restore is the third DORA metric and it is the one users experience directly. It is dominated by two things: how fast you detect, and how fast you can revert. Teams that measure it in days almost always have a detection gap or a deployment process that only runs in one direction, which forces them to fix forward through the full pipeline during an incident.

To improve your score: Instrument detection first, since minutes spent unaware are the cheapest minutes to remove. Make revert the default first response and rehearse it so the team is not debating it mid incident. Keep a current runbook per service with the rollback command, the dashboard link, and the escalation path, then run a game day to find out where your recovery assumptions are wrong before a real incident does.

Change Failure Rate

What percentage of deployments to production result in a failure requiring remediation, such as a hotfix, rollback, or patch?

Why it matters: Change failure rate is the fourth DORA metric and it is the counterweight to the other three. Any team can deploy faster by skipping checks, and this number is what exposes the cost of doing so. A high rate usually means changes are too large to reason about, or that the pipeline is missing a gate that would have caught the class of defect that keeps recurring.

To improve your score: First, actually measure it. Tag every hotfix, rollback, and emergency patch so the rate is calculated rather than estimated. Then categorize the failures from the last quarter and find the dominant class, which is often configuration differences between environments, missing integration coverage, or database migrations. Add the specific automated check that would have caught that class, and shrink batch size so each deployment carries less that can go wrong.

Test Type Breadth

What types of automated tests does your team run as part of the CI/CD pipeline?

Why it matters: Unit tests confirm that pieces work in isolation, which leaves the most common production failures untouched: a contract mismatch between services, a broken user journey, or a system that behaves fine until real load arrives. Breadth of test types is what lets a pipeline speak to production readiness rather than only to code correctness.

To improve your score: Follow the test pyramid rather than adding more of what you already have. Add contract or integration tests at the service boundaries you own, then a small set of end-to-end tests covering only your critical user journeys, since large end-to-end suites are slow and flaky. Add a performance test on your highest traffic endpoint with a threshold that fails the build, and run the slower tiers on merge to the main branch so the fast feedback loop stays fast.

Security Scanning Integration

How integrated is automated security scanning into your CI/CD pipeline?

Why it matters: Security findings discovered just before a major release arrive at the worst possible moment, when the change is large, the date is fixed, and the pressure is to accept the risk. Scanning that runs on every change keeps findings small and attributable to the commit that introduced them. Dependency vulnerabilities in particular appear without anyone touching the code, so a point in time scan is stale almost immediately.

To improve your score: Add three scans to the pull request pipeline: dependency scanning against a vulnerability database, static application security testing on the source, and secret detection so credentials never reach the repository. Gate on new critical and high severity findings rather than on the entire existing backlog. Turn on automated dependency update pull requests so patching becomes routine, and route findings to the owning team rather than into a central security queue.

Environment Provisioning

How are your deployment environments provisioned?

Why it matters: Hand configured environments drift, and drift is the origin of the phrase it works on my machine. When test and production differ in ways nobody has documented, your pipeline is validating a system that does not exist. Manual provisioning also creates scarcity, and scarce environments force teams to queue and to batch, which shows up directly in lead time and deployment frequency.

To improve your score: Define environments as code with a tool such as Terraform, Bicep, or Pulumi, keep those definitions in version control, and apply them through the same pipeline that ships your application. Rebuild a lower environment from scratch on a regular cadence to prove the definition is complete rather than a partial record of manual work. Make ephemeral environments available per pull request so testing stops competing for one shared box.