By Tracy Ragan, CEO DeployHub, CD Foundation Board Member
The New Mexico CI/CD CDF Meetup, hosted by DeployHub, enjoyed an amazing presentation by Louis Vernon, Site Reliability Engineer at Descartes Labs. Louis showed in detail how Descartes Labs improved service levels to customers, dumped a waterfall release approach and simplified their GKE releases using Spinnaker, Istio and StackDriver.
Louis’ presentation covers how the team at Descartes Lab implemented Spinnaker to push continuous deployments including the integration of Istio to route updates between Dev, Beta, Pre-Release, and Release, all running in the same cluster.
The use of both Istio and Spinnaker at Descartes Labs is a mature example of what can be done to build out a modern Kubernetes Pipeline.
While Descartes Labs still implements different ‘states’ of the pipeline, the release process uses a single cluster with Istio properly routing using virtual service names.
Louis explains how the team at Descartes Labs got to the point where they understood that a shift of this magnitude was essential for creating a stable environment for all users.
In my humble opinion, moving away from separate Dev, Test and Prod clusters is the direction we will all be moving.
The full recorded demo is here:
Louis also presented at the Spinnaker Summit 2019. You can download his presentation at:
The Continuous Delivery Foundation has recruited its first incubation project since its birth just a year ago, in the shape of container focused build service Screwdriver.
Screwdriver was originally spawned at Yahoo as “simplified interfacing” for Jenkins, before it was open sourced in 2016 and “completely rebuilt to handle deployments at scale along with CI/CD goals.”
According to the CDF, the project “ties directly into DevOps teams’ daily habits. It tests pull requests, builds merged commits, and deploys to any environment. It also defines load tests, canary deployments, and multi-environment deployment pipelines with ease.”
The Continuous Delivery Foundation (CDF), a vendor-neutral home for many of the fastest-growing projects for continuous delivery, is announcing Screwdriver as its newest incubation project. Screwdriver is a self-contained, pluggable service to help developers build, test, and continuously deliver software using the latest containerization technologies. Screwdriver was originally developed by Yahoo, now Verizon Media, as simplified interfacing for Jenkins. It was open sourced in 2016 and completely rebuilt to handle deployments at scale along with CI/CD goals.
Screwdriver ties directly into DevOps teams’ daily habits. It tests pull requests, builds merged commits, and deploys to any environment. It also defines load tests, canary deployments, and multi-environment deployment pipelines with ease.
Begin contributing to Screwdriver today. Pull requests are always welcome. Start by browsing the Screwdriver contributing guide.
“The CD Foundation welcomes Screwdriver. We believe Screwdriver is off to an excellent start, and we’re excited to be working together. By joining the CD Foundation, Screwdriver will be able to scale more quickly, taking greater strides forward in development and deployment,” said Dan Lopez, CDF program manager. “With so many supported integrations, Screwdriver provides the openness and flexibility that DevOps teams require.”
“The Screwdriver team and platform are heroes at Yahoo and Verizon Media for helping us run our massive software engineering operations at scale. Together we can make your CI/CD team heroes at your company too. We invite you to work with us in this neutral home for open source excellence.” Gil Yehuda, Sr. Director of Open Source, Verizon Media/Yahoo.
“It’s great to see Screwdriver joining the CDF. I know the people behind the project are passionate about the same thing we are, and together we can make a bigger impact faster. Open source has a proven unique ability of achieving that across project boundaries,” said Kohsuke Kawaguchi, Co-CEO at Launchable, Inc.
The CD Foundation provides a wide range of services to projects, and the first step is starting as an Incubation Project. Full details on bringing an open source CI/CD project to the CDF are available here.
“Our team is thrilled to join the CDF. Together with our partners from Yahoo! Japan and all our external contributors we’ll continue to rapidly deliver solutions which support developer workflows and interoperability with various Continuous Delivery solutions.” Jithin Emmanuel, Sr. Engineering Manager, Verizon Media/Yahoo and Product Owner for Screwdriver.
For more information on getting involved with Screwdriver, please visit:
Do you have your ticket for the CD summit yet? Grab it soon, because we have limited space and it looks like we’re gonna sell out!
We (Rosalind Benoit, Armory, and Christie Wilson, Google) are thrilled to be collaborating with the CDF as co-chairs of the upcoming Continuous Delivery Summit, happening March 30th, 2020, and co-located with KubeCon EU in Amsterdam.
What’s our vision for this event? And more importantly, what’s the story of Continuous Delivery (CD) that the fledgling CDF has committed to sharing with the world? Why does it matter, and what do we hope to gain from telling it?
If you create software, you know how much power your software delivery lifecycle (SDLC) has: it can feel slow and oppressive, or it can accelerate you and give you the freedom you need to try all your cool ideas!
Software is increasingly more and more important to our culture and our economy: enterprises need great software to offer competitive products and services, and humans need great software to automate tasks, do less with more, and improve the lives of their families and communities.
And we need CD to make the software that makes this all possible!
At the CDF, we believe in CD, we believe in CI, and we believe that we make better solutions when we have more perspectives. The CDF aims to bring together the people building and using CI/CD projects so that we can take CI/CD forward into the future together as a community. At the CD summit we want to unpack these goals, dig into delivery platforms and strategies, and give voice to the frustrations and successes you’ve run into with your own SDLC.
This blog post has been written by the owners of the different projects, and in particular, huge thanks to Christie Wilson, Andrea Frittoli, Adam Roberts and Vincent Demeester!
At the end of last year Dan wrote the blog post: A Year of Tekton. It was a great retrospective on what happened since the bootstrap of the project; a highly recommended read! Now that we’re getting into the swing of 2020, let’s reflect again back on 2019 and look forward to what we can expect for Tekton this year!
Tekton in 2019
We can safely say 2019 (more or less the project’s first year!) was a great year for Tekton. Just like a toddler we tried things, sometimes failed and learned a lot; we are growing fast!
The year 2019 saw 9 releases of Tekton Pipelines, from the first one (0.1.0) to the latest (0.9.2). We shared the work of creating the releases as much as possible, though many contributors are behind the work in each!!
0.1.x (Jason Hall) First Tekton Pipelines release!
0.2.x (Christie Wilson) Tekton Pipelines with graphs; without init containers!
0.3.x Chartreux C-3PO (Vincent Demeester) Released using Tekton itself!
0.4.x Aegean Brackenridge (Dan Lorenc) Exposes digests of built images
If you are curious about the naming of the release starting from 0.3.x, we decide to spice things up a bit and name our release with a composition of a cat breed and a robot name (in reference to our amazing logo, a robot cat).
Chip, our 0.5.0 mascot!
Aside from the initial project (tektoncd/pipeline), we bootstrapped a bunch of new projects:
tektoncd/cli: This project aims to provide an easy to use command line interface to interact with the tekton components. As Tekton objects are Kubernetes components you can always interact with them via the Kubernetes CLI — kubectl, but the kubectl experience can be very ‘raw’ and not very focused. The `tkn` CLI has the ambition to provide an easy to use user experience without having to know anything about kubectl (or Kubernetes for that matter).
tektoncd/dashboard: Alongside the CLI project, the Tekton Dashboard provides a user interface for the Tekton components, in a browser. It allows users to manage and view Tekton PipelineRuns and TaskRuns and the resources involved in their creation, execution, and completion.
tektoncd/catalog: Tekton pipeline is designed to provide highly shareable objects (Task, Pipeline, Condition, …), so creating a repo to store a catalog of shared Tasks and Pipelines came naturally!
tektoncd/experimental: With growing interest in Tekton came a growing number of “feature requests”. In order to be careful about how we expand the scope of Tekton pipeline while still allowing contributors to experiment, we created this repository to allow experiments to happen more easily. Experiments can graduate with enough traction. The biggest project so far is the webhooks extension which combines using the Dashboard project and Triggers to allow users to create webhooks for Git that trigger PipelineRuns.
tektoncd/operator: This project aims to provide an operator to manage installation, updation and uninstallation of tektoncd projects (pipeline, …). It has yet to be published in the community OperatorHub.
tektoncd/triggers: And speaking of the experimental repo, we have Triggers which started its life there! This project provides lightweight event triggering for Pipelines.
Looking forward into 2020 🔮
We’ve come a long way but we’ve got more to do! Though we can’t predict what will happen for sure, here is a preview of what we’d like to make happen in 2020!
Beta API, GA
As you’ve seen, we’ve made a lot of changes! Going forward we want to make sure folks who are using and building on top of Tekton can have more stability guarantees. With that in mind, we are pushing for Tekton Pipelines to have a beta release early in 2020. If you are interested in following along with your progress, please join the beta working group! Or keep an eye on our Twitter for the big announcement.
Once we’ve announced beta, users should be able to expect increased stability as we’ll be taking our lead from kubernetes and mirroring their deprecation policy, for example any breaking changes will need to be rolled out across 9 months or 3 releases (whichever is longer).
And once we get to beta, why stop there? We’d love to be able to offer users GA stability as soon as we possibly can. After we get to beta, we’ll be looking to progress the types that we didn’t promote to beta (e.g. Conditions), add any important features we don’t yet have (we’re looking at you on failure handling and “pause and resume” aka “the feature that enables manual approval”!), and then we should be ready to announce GA!
Task Interfaces and PipelineResources
Speaking of types that won’t be going beta right away: PipelineResources! PipelineResources are a type in Tekton that is meant to encapsulate and type data as it moves through your Pipelines, e.g. an image you are building and deploying, or a git commit you’re checking out and building from.
This concept was introduced early on in Tekton and bares a close resemblance to Concourse resources. However as we started trying to add more features to them, we started discovering some interesting edges to the way we had implemented them that caused us to step back and give them a re-think. Plus, some folks in our community asked the classic question “why PipelineResources” and we found our answer wasn’t as clear as we’d like!
As we started down the path of re-designing, and re-re-designing again, we started to get some clarity on what exactly it was we were trying to create: the interface between Tasks in a Pipeline! And thanks to a revolutionary request to improve our support for volumes, we finally feel we are on the right path! The next steps along this path are to add a few key features, namely the concept of workspaces (i.e. files a Task operates on) and allow Tasks to output values (aka “results”).
Once we have these in place we’ll revisit our designs and our re-designs.
tekton.dev
Hand in hand with our beta plans, we’re revamping our website! Soon at tekton.dev you’ll be able to find introductory material, tutorials, and versioned docs.
The Tekton Catalog
Besides making it easy for folks to implement cloud native CI/CD, one of the most important goals of Tekton is for folks to be able to share and reuse the components that make up your Pipeline. For example, say you want to update Slack with the results of a Task – wouldn’t it be great if there were one battle tested way to do that, with a clean interface?
But there’s so much more we want to do! We want to offer versioning and test guarantees that can make it painless for folks to depend on Tasks in the Catalog – and for companies to create Catalogs of their own.
Plus, the Catalog is a great place for us to build better interoperability even between the Tekton projects, for example with the Task that runs tkn (the Tekton CLI).
Shout outs 😻
A community is nothing without its users, contributors, adopters and friends, so we want to explicitly shout outs to our community for their tremendous effort and support in 2019 and hopefully even more in 2020.
We welcome friend requests! Please submit a PR to https://github.com/tektoncd/friends, this repository acts as a place that allows members of the ecosystem (known as “Tekton Friends”) to self-report in a way that is beneficial to everyone. We’d love to have you as a friend if your company is using Tekton and/or contributing to it 😀
Projects
Adoption of Tekton has grown and became a part of both free and commercial offerings by various companies, demonstrating that Tekton’s valuable and ready for anything
In mid-2019, Puppet launched a new cloud-native CD service called Project Nebula that’s built on Tekton Pipelines. It provides a friendly YAML workflow syntax and niceties like secrets management and a spiffy GUI on top of Tekton instance running in GCP. To coincide with the public beta of Nebula, Scott Seaward keynoted at the Puppetize PDX user conference to talk about how Tekton works under the hood. Since then, the Nebula team has contributed several PRs to the Pipelines repo and are looking forward to working on step interoperability, triggers, and other awesome upstream features in 2020.
It has been such a privilege to see more and more people get excited about Tekton and share it with the world! Here are some (but not all!!) of the great talks and tweets we saw about Tekton in 2020, not to mention our Tekton contributor summit!
Tara Hernandez recommends standards over specifications at DevOps World
Tekton birds of a feather at Devops Days Bangalore, India
Join us!
If you are interested in contributing to Tekton, we’d love to have you join us! Every tektoncd project has a CONTRIBUTING.md that can point you in the right direction, and our community contains helpful links and guidelines. Feel free to open issues, join slack, or pop into one of our working groups! Hope to see you soon 😀
It’s 2020, so what’s our New Year’s Resolution? #1, Make Spinnaker the Perfect Continuous Delivery Platform.
🤦 Voltaire said “perfect is the enemy of good,” and we’ve seen some resolution-minded ads lately reviving that adage (I’m thinking of Michael Phelps reminding me that Progress IS perfection : ) Striving for perfection in software development can lead to obsolete products. So, we hack. We listen to our users and iterate. When we do that as a community, we can progress towards something truly brilliant. Spinnaker’s progress was perfection in 2019, and by all accounts will exceed that trajectory in 2020.
Enterprise Adoption Crescendo (of Production workloads)
Spinnaker saw promising early adoption from large companies like Target and Adobe, and this year has been no exception. While literally everyone books stays on the site, oblivious to digital transformation, Airbnb is using Spinnaker to migrate from monolith to service-oriented architecture, and from brittle deployments to continuous delivery. SAP joyfully leverages Spinnaker on its mission to run the world better, and Pinterest uses it to boost productivity as it pioneers visual discovery. Transunion stays ahead of the fintech curve, providing total credit protection through applications it now deploys with Spinnaker, a more full-featured fit for ephemeral infrastructure than its previous Ansible solution. Companies like Comcast, going all-in on Kubernetes as a software-defined datacenter, have added Spinnaker to manage deployment pipelines. Meanwhile, Salesforce has adopted Spinnaker to bake images based on both Helm charts for Kubernetes and Packer templates for VMs, to support its complex software delivery requirements.
In 2019, we proudly welcomed engineers from new enterprises, including JPMorgan Chase and Home Depot, into the Spinnaker community. Now more than 175 companies have contributed to the project, with over 200 new ICs just last year, and many more companies have become key stakeholders, using and extending Spinnaker. These demonstrate Spinnaker as the mature CD solution, proven to handle production workloads flexibly and scalably.
Organic Growth Through Governance
As adoption continues to rise, and our community grows, it becomes crucial to create a growth-adapted project space. A transparent structure for building and maintaining the project invites new companies and users to take an active role in shaping Spinnaker’s future. To that end, 2019 saw the governance process and entities created in 2018 codified in GitHub, along with a process for modifying it via PRs and votes from members of the TOC and Steering Committee. Spinnaker governance also blossomed into an active space encompassing 8 community-initiated SIGs which organize contributors around feature growth and maintenance in areas of interest. SIGs welcome anyone to join, and we saw growing attendance from end-user companies in H2 2019. As the TOC experiments with public Open Office Hours, Spinnaker Slack is always open, and welcomes nearly 8500 participants to troubleshoot, chat with a SIG team, or reach out to a community member any time. Coupled with the donation of the project to the CDF, these growth factors signal the founding of Spinnaker as a neutral, democratized project space. Our goal? Fuel rapid innovation as we work to empower humanity to deliver their innovations, faster.
What came out of this investment in 2019? Where to begin…! An OSS ecosystem thrives with modular components that allow operators to optimize for business goals and maintain compliance. As our user base grows, the problem set expands, use cases vary, and we innovate across a richer toolchain. This allows us to create a smarter, more automated experience. Case(s) in point:
New data sources added for SignalFX and New Relic, to inform Automated Canary Analysis decisions that let app owners sleep instead of being paged.
A new Gremlin integration allowing chaos experimentation in Spinnaker pipelines will expand in 2020 to provide results useful for automated decision support.
Integrations with artifact repositories Nexus and JFrog’s Artifactory have added new native triggers for Spinnaker pipelines.
New end-to-end secrets management dynamically decrypts Spinnaker secrets as needed for validation and deployment from a backing store of your choosing, such as Vault or S3.
Since interoperability is crucial to Spinnaker, implementing a reliable plugin system was a key 2019 milestone. As our community leverages Spinnaker to solve problems, we must remove friction from the dev’s experience in contributing those extensions to the project. A plugin framework provides libraries and application context to devs, and defines clear extension points to start from when integrating something new. In 2019, we adopted PF4J as our backend plugin-loader framework. In 2020, we’ll implement plugin loading in the Spinnaker UI, and foster community around building and sharing plugins, to enrich our ecosystem.
Cloud Providers – Raining Champions : P
Spinnaker depends on cloud provider investment in extending the project for deployment to the ever-growing variety of ephemeral infra solutions. In 2019, engineers at Google developed a blueprint for a production-ready Spinnaker instance on GKE, integrated with Google Cloud services such as Cloud Build. Amazon Engineers have extended cloud providers for AWS services, ensuring that we can deploy with Spinnaker to any attribute available in Fargate or ECS (Elastic Container Service). As of this year, that includes any task definition attribute. AWS also added full support for deployments to serverless applications using AWS Lambda, including the ability to use Lambda functions as ALB targets.
Migrating to the cloud alleviates headaches, while bringing new operational challenges. Spinnaker evolves to capture and solve for these new challenges as we encounter them. The extendable Swabbie service, created in 2019, tackles the tedium (and potential nightmare at scale!) of reaping unused resources programmatically, to help optimize cloud spend. With Swabbie, an operator can set rules for cleanup candidates via YAML, and clean resources according to a configurable schedule. Deployments to highly automated cloud environments prompted enablement of dynamic account creation, discovery, and configuration for Cloud Foundry and Kubernetes accounts in the cloud.
Upleveling Functionality = Perfect Progress
The Kubernetes V2 Provider for Spinnaker also came into its own last year, offering the ability to deploy, delete, scale, and roll back K8S manifests as artifacts managed as code. The Kubernetes SIG iterated fast to improve the V2 user experience by surfacing more kubectl commands in the Spinnaker UI, and improving management of rollout strategies. They also enhanced traffic management to enable more deployment patterns with the provider, such as blue/green (AKA red/black) and dark canary. In 2020, simplifying the Kubernetes developer experience is an important roadmap element, and the community will tackle it by visualizing more K8S resources in the Spinnaker UI, and improving terminology, error, and workflow management.
Under the hood, 2019 saw lots of effort to provide operators the option to back Spinnaker with a MySQL database instead of Redis. Stateful data in Spinnaker enables event routing and orchestration for pipelines, integrated CI and SCM events, and Swabbie cleanup notifications. The choice of whether to use a relational DB or in-memory store to manage that data gives operators the freedom to optimize performance for their workloads and infrastructure. This makes all that effort, which required updating several microservices, including Echo, the eventing service, and Orca, the orchestration engine, well worth it. Likewise, updates to the Authorization model have allowed even more granular permissions to be durably API-driven across the platform.
A Bright Future Won’t Blind Us (to Your Story)
One high-level 2020 goal aims to better incorporate user stories and enterprise use cases into Spinnaker’s trajectory. The steering committee has committed to building a roadmap that tells high-level stories about using Spinnaker to solve problems. Tool chain interoperability, notably with Kubernetes, cloud providers, and monitoring systems figure large in the H1 2020 Roadmap. Managed Delivery, an exciting Spinnaker CD initiative incubating at Netflix, uniquely responds to a common narrative around software delivery. It uses declarative automation to alleviate the operational knowledge and maintenance burden that comes with ownership of modern, continuously delivered applications.
Users can help us tell the best Spinnaker stories by submitting comments and issues describing usage and business context. Please visit Spinnaker.io (which the Docs SIG will overhaul in 2020) and check out our Success Stories page. Join us on Spinnaker Slack or in the Spinnaker org and tell your tale!
Recognition of the importance of interoperability and identifying it as one of the strategic goals is a very important step for CDF to take for users. Users and organizations employ various CI/CD tools and technologies depending on their needs and where they are in their CI/CD transformation. Organizations often employ more than one tool in various stages of their CI/CD pipelines due to different capabilities provided by the tools and this is perhaps one of the biggest benefits users get by using open technologies for their CI/CD needs. For example, CDF member Salesforce has over 20 different CI/CD tools internally thanks to acquisitions and different requirements in teams.
However, one of the challenges users face is the lack of interoperability across the CI/CD tools and technologies, resulting in various issues while constructing and running pipelines such as passing metadata and artifacts between the tools or achieving traceability from commit to deployment. Often users end up building their “own glue code” to address what is a common problem, further complicating moving from one tool to another and adopting new technologies and methodologies.
These “glue code solutions” are generally specific to users’ needs and tools rather than being loosely coupled and agnostic to tooling and technology. Additionally these solutions are not visible to other users and the communities, making them vulnerable to the risk of outage in their CI/CD pipelines due to potential changes (i.e. non-backward changes to the APIs, changes in data models) that happen to the tools in respective projects.
Therefore, focusing on tool interoperability is critical.
There has been significant collaboration going on in this area. Linux Foundation Networking (LFN), OpenStack Foundation (OSF), and Cloud Native Computing Foundation (CNCF) projects have done a lot to raise awareness of CI/CD interoperability challenges. In addition to these communities, Spinnaker, Jenkins, Tekton, and Jenkins X, CDF founding projects, have been collaborating and sharing ideas. However, there are many more users, projects and communities, either looking for answers to similar interoperability challenges, on their way to developing solutions, or simply trying to find like minded people to work with together.
We believe the work should happen in a neutral forum where users come together with maintainers of open source CI/CD projects and have a dialog about the challenges we need to address.
Which is why the CDF Interoperability SIG was launched, led by Fatih Degirmenci of Ericsson and with support from representatives from Netflix, Google, China Mobile, CloudBees and others.
We, the CDF Interoperability SIG, aim to provide such a forum and enable a dialog around interoperability in order to:
clarify what interoperability means for the CI/CD ecosystem
promote the need to collaborate on interoperability challenges in a neutral forum
highlight and promote the needs of the users who face challenges constructing complex end-to-end CI/CD flows and pipelines by employing different tools and technologies
explore synergies between, and enable collaboration across, the CI/CD projects with regards to interoperability
pursue solutions which are loosely coupled, scalable, flexible, and tool and technology agnostic
reduce the need for users to implement in-house solutions by promoting native interoperability between tools
attract and assist projects that work on interoperability
Membership to the Interoperability SIG is open to the public. We invite users and contributors to open source CI/CD projects to join us to share ideas, use cases, challenges, and solutions with each other.
Here are some of the ways you can take part in the Interoperability SIG and start collaborating:
CDF SIG Meets every even week on Thursdays at 15:00UTC on Zoom and the meeting agenda and minutes are available here. Our first meeting will be on January 23, 2020.
Finally, we would like to thank everyone who has listened to our ideas, shared their thoughts, taken part in crafting the proposal, and most importantly, encouraged us with their +1s!
“Looking at the latest ‘State of Agile Report’, continuous integration was the third highest cited agile practice (with 53% of organisations indicated they had implemented some for CI), while continuous delivery was fifth (with 40%). Meanwhile, 73% of respondents stated that they either currently have, or are planning to have, a DevOps initiative in their organisation.”
