At Harness, we are here to champion Continuous Delivery for all. As a member of the Continuous Delivery Foundation [CDF], we’re happy to see that the industry has taken a collective step forward to better each other’s capabilities. We are pretty fortunate to have the opportunity to talk to many people along their Continuous Delivery journey. Part of what we do at Harness when engaging with a customer or prospect is help run a Continuous Delivery Capability Assessment (CDCA) to catalog and measure maturity.
Over the past year, we have analyzed and aggregated the capability assessments that we have performed. We uncovered common challenges organizations faced in the past 12 months. In Continuous Delivery Insights 2020, we identified the time, effort, cost, and velocity associated with their current Continuous Delivery process. Trending data that we have around key metrics is that velocity is up but also complexity and cost are on the rise.
Key Findings
We observed the following Continuous Delivery performance metrics across the sample of over 100 firms. Organizations that are looking towards strengthening or furthering their Continuous Delivery goals we noticed the following with median [middle] or average values. As sophisticated as organizations are, there is still a lot of effort to get features/fixes into production.
In terms of deployment frequency, we define deployment frequency as the number of times a build is deployed to production. In terms of a microservices architecture, deployment frequency is usually increased as the number of services typically have a one to one relationship with the build. For the sample set we interviewed, the median deployment frequency is ten days which shows bi-monthly deployments are becoming more the norm.
These bi-monthly deployments might be on demand but the lead times can start to add up. Lead time is the amount of time needed to validate a deployment once the process has started. Through the sample, organizations typically require an average of eight hours; e.g eight hours of advance notice to allow validation and sign off of a deployment.
If during those eight hours of lead time during the validation steps, if a decision is made to roll back, we saw that organizations in the sample averaged 60 mins of time to restore a service e.g roll back or roll forward. An hour might not seem too long to some but for engineers, every second can feel stressful as you race to restore your SLAs.
Adding up all the effort from different team members during a deployment, getting an artifact into production represented an average of 25 human hours of work. Certainly, different team members will have varying levels of involvement throughout the build, deploy, and validation cycles but represents more than ½ a week of a full-time employee in total burden.
Software development is full of unknowns; core to innovation we are trying and developing approaches and features for the first time. Expectation is there for iteration and learning from failures. We certainly have gotten better at deployment and testing methodologies and one way to measure is with change failure rate or the percentage of deployments that fail. Through the sample set, there was on average 11% of deployments failed. Not all doom and gloom, the Continuous Delivery Foundation is here to move the needle forward.
Looking Forward
The goal of the Continuous Delivery Foundation and Harness is to collectively raise the bar around software delivery. For organizations that are members of the Continuous Delivery Foundation, deploying via a canary deployment seems like second nature for safer deployments. If you are unfamiliar with a canary deployment, basically a safe approach to releasing where you send in a canary [release candidate] and incrementally replace the stable version until the canary has taken over. As simple as this concept is to grasp, in practice can be difficult. In Continuous Delivery Insights 2020, only about four percent of organizations were taking a canary based approach somewhere in their organization.
We are excited to start to track metrics on the overall challenges with Continuous Delivery year to year and work towards improving the metrics and adoption of Continous Delivery approaches for all. For greater insights, approaches, and breakdowns, feel free to grab your digital copy of Continuous Delivery Insights 2020 today!
