Contributed by Eric Minick, Harness (CDF Member)
Over the past ten years, DevOps practices have become a cornerstone of efficient and effective delivery. Two recent reports, the “State of Developer Experience 2024” and the “State of CI/CD Report 2024,” offer valuable insights into the current state of DevOps adoption, challenges, and performance trends. While these reports largely agree on many aspects, they also present some intriguing contradictions that shed light on the complexities of implementing DevOps at scale.
Widespread Adoption of DevOps
Both reports underscore the pervasive adoption of DevOps practices across the software development industry. According to the State of CI/CD Report, a staggering 83% of developers are involved in DevOps-related activities.
The Complexity of Tooling
One of the most significant challenges highlighted by both reports is the complexity introduced by the multitude of tools used in DevOps processes. The Developer Experience report reveals that developers manage an average of 14 different tools, leading to frequent context switching and potential productivity losses. In fact, 97% of developers report context switching due to using tools from multiple vendors.
Interestingly, the State of CI/CD Report presents a slightly different perspective on tool usage. It notes a recent decrease in the average number of DevOps technologies used by practitioners, from more than 4 before Q1 2023 to 3.4 in Q1 2024. This apparent contradiction might suggest a growing awareness of the drawbacks of tool sprawl and efforts to consolidate and streamline toolchains.
Performance Metrics and Trends
Both reports align on the use of key performance indicators, largely following the DORA metrics framework, which includes lead time for changes, deployment frequency, and time to restore service. However, the reports differ in their assessment of performance trends.
In contrast, the CI/CD observes that the proportion of top performers on key metrics has remained relatively stable or slightly decreased over time. More concerningly, it notes an increase in the proportion of low performers across these metrics. This discrepancy highlights the challenges in translating DevOps adoption into consistent performance improvements across the industry.
The Developer Experience report takes an optimistic view, suggesting that implementing tools consolidation could lead to significant productivity improvements. It estimates that organizations could realize benefits of $7.9M annually for every 1000 developers through such initiatives.
In some cases, partial adoption of “best practices” may lead to conflict. Early State of DevOps reports from 2013 and 2014, as well as Accelerate by Forsgren et al, emphasized the value of peer review for changes. With the rise of Git at a similar time, many teams adopted pull-request-centric commit practices, nominally lining up with best practices. However, the State of Developer Experience report found that two-thirds of developers experienced code review delays of over a week. These delays slow time to market and mute the benefits of continuous integration tooling, as code changes are integrated far from continuously.
Conscious measurement of flow through the lifecycle can help teams identify these bottlenecks and realize the benefits of DevOps.
The Impact of Experience
One area where both reports find common ground is the impact of developer experience on DevOps performance. The Developer Experience report notes that 71% of executive buyers said onboarding new developers takes at least 2 months, highlighting the steep learning curve associated with modern DevOps practices.
The CI/CD report provides more detailed insights, showing a clear correlation between years of experience and performance across all delivery metrics. This suggests that while DevOps tools and practices are widely adopted, mastering them to achieve top performance remains a significant challenge, especially for newer developers.
A controversial path may be to limit developers’ time with DevOps tools beyond issue tracking and code repositories. If they have less responsibility for building and maintaining pipelines, they will not have to invest in learning these tools. A move towards platform engineering in many organizations is attempting to realize this benefit.
The Tool Dilemma
Perhaps the most intriguing contradiction between the reports lies in their findings on optimal tool usage.
The two reports seem to agree that having too many tools can be a burden. The Developer Experience report found that 97% of developers say that they experience context-switching overhead because their tools are from multiple vendors. The outcome may be visible in the State of CI/CD Report, which found worse performance in lead time, deployment frequency, and time to restore as the number of CI/CD tools used grew beyond three. This was especially pronounced for self-hosted CI/CD tools. For example, the percentage of people reporting a lead time greater than a month for users of one or two tools was in the 25-30% range. Once the number of CI/CD tools grew to four or five, over 40% reported these slow releases.
At the same time, the State of CI/CD Report does observe some modest benefits to using a blend of self-hosted and managed platforms.
This contradiction underscores the delicate balance organizations must strike in their DevOps tooling strategies. While a diverse toolset can provide flexibility and cover a wide range of needs, it also introduces complexity that can hinder performance if not managed carefully.
Conclusion
The insights provided by these reports paint a picture of an industry that has widely embraced DevOps practices but is still grappling with how to optimize their implementation. While DevOps adoption is high, achieving consistently high performance remains challenging, particularly for less experienced developers and organizations dealing with complex toolchains.
As the industry moves forward, the key to success may lie in finding the right balance – leveraging the benefits of diverse tools while avoiding the pitfalls of overcomplexity. Organizations will need to focus on streamlining their DevOps processes, invest in developer experience and training, and carefully consider their tooling strategies to maximize performance and efficiency in software delivery.
The contradictions between these reports remind us of the complex and nuanced nature of DevOps implementation. They also highlight the need for ongoing research and analysis to better understand the factors that contribute to DevOps success in various contexts.
Acknowledgments
- The State of CI/CD Report 2024 is published by the Continuous Delivery Foundation. StashData led the research.
- The State of Developer Experience, 2024 report was published by Harness, CD Foundation Member.
