Dec 12

What’s New in Tekton 0.9

By cdfoundation Blog, Project

By Dan Lorenc
dlorenc@google.com
twitter.com/lorenc_dan

Another 6 weeks, another Tekton release. It’s mostly common knowledge that Tekton’s logo is a robot cat, but it’s lesser known that the releases are named after robots and cats! Each Tekton Pipelines release is given a codename of a type of cat followed by a famous robot.

On Monday, December 2nd, Andrea Frittoli of IBM cut the v0.9.0 release, dubbed “Bengal Bender”. This release contained commits from 21 different individuals. I wanted to take the time to highlight some of the new features and API changes, as well as to point out what much of the “under the covers” work is laying the groundwork for.

So let’s jump in!

New Features and Bug Fixes

“Bengal Bender” includes a solid set of features, bug fixes and performance improvements. Apologies if I missed anything here, this list is simply what I find most exciting.

Script mode!

https://github.com/tektoncd/pipeline/pull/1432

If you were at Kubecon San Diego you might have come away with the impression that Go is the language of the cloud. And while that is true to some extent, good old bash and yaml also play a huge part, especially when it comes to “glue” systems like delivery pipelines.

If you’ve spent much time working with containers, you’ve probably seen a yaml file with something like this in it:

- name: hello
  image: ubuntu
  command: ['bash']
  args:
  - -c
  - |
      set -ex
      echo "hello"

This is a lot of complicated boilerplate just to run a simple bash script inside a container. And if you’re not deeply familiar with how bash, shells, entrypoints and shebangs all work at the system level, this is a bit opaque. It’s also prone to subtle and confusing bugs. Even if you are familiar with these things, you’ve probably wasted time debugging issues when the shell in your container is set to something you’re not used to, or the entrypoint is overridden, or “bash -c” doesn’t do what you expect.

This makes delivery pipelines harder to write, understand and maintain.

Enter script mode! Inspired by some ideas from Ahmet Alp Balkan, Jason Hall put together a proposal to make it much easier to define tasks that just need to run a simple bash script. Here’s what it looks like:

- name: hello
  image: ubuntu
  script: |
    #!/bin/bash
    echo "hello"

You’ll notice that there are far fewer lines of boilerplate. No more need to specify args, an entrypoint or remember the tricky “-c” part. Just specify the interpreter you want to use and the commands to run. This has already let us simplify dozens of test cases and examples!

Performance

https://github.com/tektoncd/pipeline/pull/1545

Tekton has long suffered from poor performance around starting PipelineRuns. Christie Wilson and I did a debugging/coding session last spring to try to improve this, and identified PVC mounting as a major contributor. Unfortunately, our attempted fix didn’t work and needed to be rolled back.

A more general longer term fix is being handled by Scott Seaward and the re-resources effort, but I decided to take another stab at a short term fix. Fingers crossed that it works, but initial testing shows improvements of anywhere between 5 and 20 seconds per PipelineRun!

API Changes

There have been a few changes to the API as we start to firm things up for a beta release. We’re hoping to get the majority of these breaking changes in over the next few releases so users can start to build production systems on top of stable versions.

The breaking changes in v0.9.0 include:

Standardization of output paths for image digests

Tekton currently provides a mechanism to store the digests of container images built by Tasks. This mechanism predated the PipelineResource subsystem, and required Task authors to write these digests to a specific location at /builder/image-outputs. This change moves that to the standard path for output resources, at /workspace/output/<resource-name>.

Simplification of the cluster resource

Cluster PipelineResources make deploying and working with Kubernetes clusters from within Tasks simple. They provide mechanisms for users to declare where a cluster endpoint is and how to authenticate with it. Then, during Task execution, they automatically configure a .kubeconfig file so Kubernetes tooling can find that cluster. This release contained a few changes to make these cluster resources easier to work with.

Previously, users had to specify a name parameter twice: once in the resource name and once as a parameter to the resource. The second parameter has been removed.

Ground Work

Following the age-old programming advice above, most of the work contained in each Tekton release is in service of features that won’t be exposed until a later release. Digging into the commit log shows a detailed picture of what the community is working toward.

Here’s my editorialized version of what’s coming soon 🙂

Revamped PipelineResources

A lot of work went into cleaning up the existing PipelineResource subsystem, including the interface exposed to PipelineResource types and the types themselves. Getting these both solid will form the basis for the larger Re-resources effort, which is currently underway. This project will make the resource types extensible, allowing anyone to add and use their own types. It will also hopefully leave us with some embeddable components, so that other systems can make use of Tekton PipelineResources and the coming catalog.

API Versioning

One of the most important steps in shipping a stable API is figuring out how to make changes in a backwards compatible way. No API is perfect, so the ability to upgrade one is paramount. Vincent Demeester and his colleagues at Red Hat have been hard at work designing and implementing an API versioning system that will allow users to upgrade Tekton API versions without breaking existing workloads. This will be key to the coming beta release.

Wrapping Up

The Tekton project has been amazing to watch grow. This post only details the changes in the Tekton Pipelines release, but there has also been some awesome work done in the Triggers, CLI and Dashboard projects. Triggers now support out-of-the-box Github and Gitlab validation. The CLI has improved support for creating PipelineResources and starting Tasks interactively. Visualization is coming soon to the Dashboard! I’d like to thank everyone that has made Tekton what it is today.

The Tekton community has been hard at work shipping the right APIs and components to build cloud-native software delivery systems. If you’re using Tekton, or interested in learning more about Tekton, we’d love to hear from you. Please consider joining the community, becoming a Tekton Friend or contributing directly.

Nov 15

Love0

A Year of Tekton

By cdfoundation Blog, Project

Author: Dan Lorenc (dlorenc@google.com, twitter.com/lorenc_dan, github.com/dlorenc)

Last year at Kubecon Seattle, Tekton was just an idea in the heads of a few people, and a bit of code inside knative/build-pipeline. Fast forward to today, and we have a thriving community, independent governance inside a great foundation, and we’re quickly approaching our beta release! This year has flown by, so I wanted to highlight some of the original goals of the Tekton project, and some of the milestones we’ve hit toward them.

The Start

When Christie, Jason and I started sketching out the original Tekton APIs on a whiteboard in San Francisco, we had a straightforward goal in mind: make the hard parts of building CI/CD systems easy. There are already dozens of CI/CD systems designed to run on kubernetes, and for the most part they all have to build the same basic infrastructure before they can start solving customer problems. Kubernetes makes scheduling, orchestration and infrastructure management easier than anyone could have imagined, but it still leaves users with a few pieces to assemble before they can use it as an application delivery platform.

These were some of the original whiteboard sketches on what later became the Tekton Pipeline CRD.

Things like basic DAG orchestration, artifact management, and even reliable log storage are outside the bounds of the core Kubernetes APIs. Our plan was to use the new Custom Resource Definition feature to try to define a few more “nouns”, on top of the existing Kubernetes primitives, that were better suited for Continuous Delivery workloads. In doing so, we would make it easier for people and trans to create delivery systems designed for their exact use cases, while making sure the underlying primitives allowed for some degree of compatibility.

Vague ideas are great, but it’s much more productive to collaborate with others when you have something concrete to share and get feedback on. So in August of 2018, we released the sketch and principles we used to design it on knative-dev and to a few other interested parties. The feedback was amazing. The core pipeline team on Jenkins at Cloudbees jumped in with some hard earned lessons from their experience working on the most widely used orchestration system in history. The Concourse team at Pivotal helped redesign our extensibility system based on what they learned from the successful Concourse resource model.

Then we got to work building it all out! Our goal must have really resonated with others, because we could really feel the power of open source from day one. Even when there were only a few of us working together at Google, we were part of a much larger effort. We really wouldn’t be here today without the help of our contributors, maintainers and governing committee members.

A New Home

Around the time we prepared for our first release, it started to become obvious that Tekton (then called knative/build-pipeline) needed its own home. The knative brand and community helped immensely, but Tekton was meant to provide CI/CD for everyone – not just serverless or even kubernetes users. So in February of 2019, we decided to split out the project, name it “Tekton” and donate it to the newly-forming Continuous Delivery Foundation just in time for the 0.1.0 release.

Kim Lewandowski announcing Tekton at the Open Source Leadership Summit

Open source, governance, and communities are hard. The move out of knative and into a new foundation was a big change for the project, but has proven worth the effort! Tekton still works great with other knative components, but has had the chance to grow its own community and evolve to a spot that its users need. Thanks to the community, Tekton has expanded into multiple projects like the Dashboard and CLI, and in Tekton Pipelines we have been so lucky to gain the expertise of folks like Vincent Demeester and Andrea Frittoli.

The rest of this year has felt like a blur, but I wanted to call out some major milestones we hit:

March – The first Tekton Pipelines release (v0.2.0) using Tekton itself!
June – First release of the Tekton CLI!
July – The first Tekton Pipelines release (v0.5.0) by a community member not at Google!
July – Tekton Friends repo created!
August – First release of the Tekton Dashboard!
September – First Triggers release!
October – Tekton passed 100 contributors in October!

What’s Coming Next?

We’re rapidly approaching the first Tekton beta release! As part of this effort, we evaluated our API surface and identified quite a few areas that need hardening. This includes finishing up the table-stakes requirements for a CI/CD platform – things like triggers, metadata storage and building up our catalog. The Triggers v0.1.0 release has made Tekton usable in so many new ways, and we’re just getting started there still!

Scott Seaward has has just started work on refactoring PipelineResources into an extensible system that will form the basis for the Tekton catalog, and Jason Hall is working on a metadata storage system that will help power some new ideas around software supply chain management.

If you’d like to get involved in the Tekton project, you can find us on Github, Slack or our email list . We’ll also be at Kubecon next week! Come attend one of the many sessions or the CD Summit. Here’s to Tekton in 2020!

Oct 11

Love0

CD Foundation is participating in Outreachy!

By cdfoundation Blog, Staff

Outreachy is a program for open source internships that specifically targets people in demographics that are underrepresented in tech. Since 2010 Outreachy has had over 400 participants making contributions to open source projects. Outreachy is one of the most effective programs for improving the diversity of open source communities. For those familiar with Google Summer of Code, Outreachy follows a very similar format.

The Continuous Delivery Foundation is a neutral home for the next generation of continuous delivery collaboration. We know that the greater number of diverse voices we have collaborating, the more effective we are as a community. Which is why we are thrilled to be participating in Outreachy for this upcoming round.

Three of the CD Foundation projects: Jenkins, Jenkins X and Tekton are offering Outreachy internship projects. The Jenkins project has participated in the previous 2 rounds of Outreachy, having a total of 4 Outreachy interns working with the community on the Jenkins Audit Log Plugin. Jenkins mentor Matt Sicker shares in this post “Outreachy has helped open my eyes to the struggles that developers from around the world are dealing with which can be improved upon to help expand our communities. For example, many countries do not have reliable internet or electricity, so the use of synchronous videoconferencing and other heavyweight, synchronous processes common to more corporate-style development are inadequate in this international context.”

In that way Outreachy is also beneficial to the mentors participating. The actual tech contributions are a bonus side effect. Jenkins participates in the program with no expectation that the interns remain part of the community – but takes a wider, long term view that this generally improves open source and tech communities as a whole. Tracy Miranda, Outreachy coordinator for the Jenkins project says “As of this year I know of 2 Outreachy alumni (non-Jenkins projects) who were hired by my employer and both mention the Outreachy program as an important stepping stone in their career journeys.”

Here are the details of this round’s CD Foundation projects which we are looking for interns for. Please help us spread the word.

Jenkins

Jenkins X

Tekton

Kara de la Marck, the Jenkins X Outreachy coordinator can personally speak to the benefits of the program: “Outreachy is a fantastic mentoring program that helps to onboard new contributors to a project and to open source more generally. Many participants go on to become long term contributors to open source. As an alumna, I have carried forward a deep appreciation of open source as an enabler of global collaboration, technological innovation, and community. I’m incredibly happy to welcome Outreachy participants to Jenkins X.”

Please help us spread the word, and we look forward to working with Outreachy interns and welcoming them into our community.

Aug 14

Love0

Jenkins Celebrates 15 Years of Transforming Software Delivery

By cdfoundation Announcement

Jenkins Community Celebrates 15 years of Continuous Delivery Automation and Innovation at DevOps World | Jenkins World

DEVOPS WORLD | JENKINS WORLD, SAN FRANCISCO – AUGUST 14, 2019 – The Jenkins project, comprised of the community of practitioners using Jenkins^®, today celebrated its 15^th birthday at DevOps World | Jenkins World with a recap of milestones showcasing the community’s growth and the project’s defining impact on the global software industry.

Jenkins is the world’s leading open source automation server, used by companies large and small to implement continuous integration (CI) and continuous delivery (CD). Originally developed in 2004 and called Hudson, Jenkins’ impact has grown consistently over the years to the point where experts regularly describe Jenkins as the de facto tool for CI/CD.

So far this year, the Jenkins project has been a key driver in the formation of the Continuous Delivery Foundation, has continued to experience strong uptake in the use of Jenkins and will recognize key contributors to the Jenkins project around the globe during Thursday’s keynote session at DevOps World | Jenkins World.

In establishing the Continuous Delivery Foundation, the Jenkins community worked with The Linux Foundation, CloudBees, Google and Netflix to create a new foundation for the diverse CI/CD space. In addition to Jenkins, the Continuous Delivery Foundation was established with several other CI/CD open source projects, including Jenkins X, Spinnaker and Tekton. It serves as a vehicle to develop, nurture and promote open source projects, best practices and industry specifications related to continuous delivery.

The Continuous Delivery Foundation fosters vendor-neutral collaboration between the industry’s top developers, end users and vendors to further CI/CD best practices and industry specifications. Its mission is to grow and sustain projects that are part of the broad and growing continuous delivery ecosystem.

“This has been a great year for Jenkins, the Continuous Delivery Foundation and open source collaboration as a whole,” said Chris Aniszczyk, vice president at the Linux Foundation. “We all share a common mission – to support community-based development of projects that advance the state of software delivery. The Jenkins project has been squarely behind this effort from day one and today the community is stronger than ever.”

Also playing a key role in Jenkins’ transition to the CDF was CloudBees’ Tracy Miranda. Miranda took on the dual roles of CloudBees director of open source community and member of the governing board of the CDF. “CD is becoming a differentiator for organizations in every industry, yet adoption remains challenging. It’s an industry-wide problem that needs an industry-wide solution. From the CloudBees perspective, we see it as critical to have a neutral foundation where all agents of change can collaborate and contribute openly,” said Miranda. “Looking ahead to the next 15 years, we need to solve the complexity of CD adoption. With the CDF, we are well-equipped to do this in open source – building on top of all that we have learned in the Jenkins community over the years.”

During the period from July 1, 2018, through June 30, 2019, the Jenkins project achieved these milestones:

46% growth in active Jenkins installations¹ reporting usage data was in the period August 1, 2018 – July 31, 2019
The community experienced approximately 46 percent growth in active installations, reaching 265,956 installations as of July 31, 2019, compared to 182,236 installations as of August 1, 2018. Active installations are defined as Jenkins instances that report usage information back to the Jenkins project. This number is not representative of the total Jenkins instances in use worldwide; that number is significantly greater.

Approximately 15.8 million Jenkins developers
A recent Datanyze analysis of the CI vendor landscape showed that about 66 percent of continuous integration is being run on Jenkins. With an estimated 24 million developers, globally, according to Evans Data in its 2019 Global Developer Population and Demographics Study, approximately 15.8 million developers are using Jenkins.

254% growth in Jenkins Pipeline jobs
Finally, the combined number of defined Jenkins jobs increased during this same period from 19,946,119 in July 2018 to 30,281,905, or 52 percent growth. Specifically, Jenkins Pipeline jobs grew 254 percent in the same period. The dramatic growth in Jenkins Pipeline jobs demonstrates that organizations are accelerating their investment in modern software pipeline automation practices with Jenkins.

“Over the past 15 years, the Jenkins project has revolutionized the way software is built and delivered,” said Jenkins creator Kohsuke Kawaguchi, who also serves as chief scientist at CloudBees. “We have touched every industry and made a difference to every software team in the world.”

Additional Resources

¹The Jenkins community tracks statistics from active Jenkins installations that transmit usage information back to the project. The numbers do not represent a majority of Jenkins installations, only those who choose to report. Therefore, the numbers are conservative.

About the Continuous Delivery Foundation
Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. The Continuous Delivery Foundation (CDF) is a Linux Foundation initiative that serves as the vendor-neutral home for many of the fastest-growing projects, including Jenkins, Jenkins X, Spinnaker, and Tekton. The CDF fosters collaboration between the industry’s top developers, end users and vendors to further continuous delivery best practices. For more information about the CDF, please visit https://cd.foundation.

Jul 19

Love0

Join Us At the CDF Contributor Summit!

By cdfoundation Announcement

Every year before the DevOps World | Jenkins World conference, Jenkins contributors gather for a contributor summit. Earlier this year, the CD Foundation became the new home for Jenkins as well as Jenkins X, Tekton and Spinnaker projects. CD Foundation serves as the new home for many of the fastest-growing projects for continuous delivery as well as a neutral home for collaboration in the CI/CD space. In that spirit of collaboration, this year’s contributor summit is being expanded to be a CDF Contributor Summit.

CDF Contributor Summit will be held on Monday August 12, 2019 in San Francisco, USA just before DevOps World | Jenkins World. It is a free event open to contributors of the open source projects. So far we will have community members from the Jenkins, Jenkins X and Tekton projects attending the summit.

The summit brings together community members to learn, meet and help shape the future of the projects. In the CDF community we value all types and sizes of contributions and love to welcome new participants who want to contribute to one or more projects.

The morning portion of the summit will consist of presentations by core committers of the projects. Presentations will highlight what each effort is about and what community members can do to help. We will cover a range of topics from technical aspects to community topics.

In the afternoon we will break into Birds of a Feather table for in-depth discussion and collaboration on different topics of interest. Bring your laptop, come prepared with questions and ideas, and be ready to meet and get coding with other contributors.

Agenda: (tentative)

9:00 am – Kickoff & Welcome with coffee/pastries
10:00 am – Project Updates
12:00 pm – Lunch
1:00 pm – BoF/Unconference
3:00 pm – Break
3:30 pm – BoF/Unconference
4:30 – Ignite Talks
5:00 pm – Wrap-up

To join the summit, please sign up with the main project you contribute to or are interested in contributing to:

Jenkins: https://www.meetup.com/jenkinsmeetup/events/262686097/

Jenkins X: https://www.eventbrite.com/e/jenkins-x-contributor-summit-2019-san-francisco-tickets-65105473223

Tekton: https://www.eventbrite.com/e/tekton-contributor-summit-2019-san-francisco-tickets-64992545453

We hope to see you there!

Mar 12

Love0

Introducing the Continuous Delivery Foundation, the new home for Tekton, Jenkins, Jenkins X and Spinnaker

By cdfoundation Blog, Staff

By Dan Lorenc and Kim Lewandowski, DevOps at Google Cloud

This is a contributed blog from our Premier founding member Google on the donation of Tekton and Spinnaker to CDF. Originally published on the Google Open Source Blog.

We’re excited to announce that Google is a founding member of the newly formed Continuous Delivery Foundation (CDF). Continuous delivery (CD) is a critical part of modern software development and DevOps practices, and we’re excited to collaborate in a vendor-neutral foundation with other industry leaders.

We’re also thrilled to announce the contribution of two projects as part of our membership: Tekton, and in collaboration with Netflix, Spinnaker. These donations will enter alongside Jenkins and Jenkins X, providing an exciting portfolio of projects for the CDF to expand upon.

Continuous Delivery Foundation

Currently, the continuous integration/continuous delivery (CI/CD) tool landscape is highly fragmented. As companies migrate to the cloud and modernize their infrastructure, tooling decisions become increasingly complicated and difficult. DevOps practitioners constantly seek guidance on software delivery best practices and how to secure their software supply chains but gathering this information can be difficult. Enter the CDF.

The CDF is about more than just code. Modern application development brings new challenges around security and compliance. This foundation will work to define the practices and guidelines that, together with tooling, will help application developers everywhere deliver better and more secure software at speed.

At a foundation level, the CDF will help make CI/CD tooling easier. And at a project level, Tekton helps address complexity problems at their core. We will team up with the open source community and industry leaders to design and build the critical pieces common to CI/CD systems.

Tekton

Tekton is a set of shared, open source components for building CI/CD systems. It provides a flexible, extensible workflow that accommodates deployment to Kubernetes, VMs, bare metal, mobile or even emerging use cases.

The project’s goal is to provide industry specifications for pipelines, workflows, source code access and other primitives. It modernizes the continuous delivery control plane by leveraging all of the built-in scaling, reliability, and extensibility advantages of Kubernetes, and moves software deployment logic there. Tekton was initially built as a part of Knative, but given its stand-alone power, and ability to deploy to a variety of targets, we’ve decided to separate its functionality out into a new project.

Today, Tekton includes primitives for pipeline definition, source code access, artifact management, and test execution. The project roadmap includes adding support for results and event triggering in the coming months. We also plan to work with CI/CD vendors to build out an ecosystem of components that will allow you to use Tekton with existing tools like Jenkins X, Knative and others.

Spinnaker

Spinnaker is an open source, multi-cloud continuous delivery platform originally created by Netflix and jointly led by Netflix and Google. It is typically used in organizations at scale, where DevOps teams support multiple development teams, and has been battle-tested in production by hundreds of teams and in millions of deployments.

Spinnaker is a multi-component system that conceptually aligns with Tekton, and that includes many features important to making continuous delivery reliable, including support for advanced deployment strategies, and Kayenta, an open source canary analysis service.

Given Google’s significant contributions to both Tekton and Spinnaker, we’re very pleased to see them become part of the same foundation. Spinnaker’s large user community has a great deal of experience in the continuous delivery domain, and joining the CDF provides a great opportunity to share that expertise with the broader community.

Next Steps

To learn more about the CDF, listen to this week’s Kubernetes Podcast from Google, where the guest is Tracy Miranda, Director of Open Source Community from our partner CloudBees.

If you’d like to participate in the future of Tekton, Spinnaker, or the CDF, please join us in Barcelona, Spain, on May 20th at the Continuous Delivery Summit ahead of KubeCon/CloudNativeCon EU. If you can’t make it, don’t worry, as there will be many opportunities to get involved and become a part of the community.

We look forward to working with the continuous delivery community on shaping the next wave of CI/CD innovations, alignments, and improvements, no matter where your applications are delivered to.