Tekton Pipelines, the core component of the Tekton project, is moving to beta status with the release of v0.11.0 this week. Tekton is an open source project creating a cloud-native framework you can use to configure and run continuous integration and continuous delivery (CI/CD) pipelines within a Kubernetes cluster.
Tekton development began as Knative Build before becoming a founding project of the CD Foundation under the Linux Foundation last year.
The Tekton project follows the Kubernetes deprecation policies. With Tekton Pipelines upgrading to beta, most Tekton Pipelines CRDs (Custom Resource Definition) are now at beta level. This means overall beta level stability can be relied on. Please note, Tekton Triggers, Tekton Dashboard, Tekton Pipelines CLI and other components are still alpha and may continue to evolve from release to release.
Tekton encourages all Tekton projects and users to migrate their integrations to the new apiVersion. Users of Tekton can see the migration guide on how to migrate from v1alpha1 to v1beta1.
Now is a great time to contribute. There are many areas where you can jump in. For example, the Tekton Task Catalog allows you to share and reuse the components that make up your Pipeline. You can set a Cluster scope, and make tasks available to all users in a namespace, or you can set a Namespace scope, and make it usable only within a specific namespace.
From Dailymotion, a French video-sharing technology platform with over 300 million unique monthly users
At Dailymotion, we are hosting and delivering premium video content to users all around the world. We are building a large variety of software to power this service, from our player or website to our GraphQL API or ad-tech platform. Continuous Delivery is a central practice in our organization, allowing us to push new features quickly and in an iterative way.
We are early adopters of Kubernetes: we built our own hybrid platform, hosted both on-premise and on the cloud. And we heavily rely on Jenkins to power our “release platform”, which is responsible for building, testing, packaging and deploying all our software. Because we have hundreds of repositories, we are using Jenkins Shared Libraries to keep our pipeline files as small as possible. It is an important feature for us, ensuring both a low maintenance cost and a homogeneous experience for all developers – even though they are working on projects using different technology stacks. We even built Gazr, a convention for writing Makefiles with standard targets, which is the foundation for our Jenkins Pipelines.
In 2018, we migrated our ad-tech product to Kubernetes and took the opportunity to set up a Jenkins instance in our new cluster – or better yet move to a “cloud-native” alternative. Jenkins X was released just a few months before, and it seemed like a perfect match for us:
It is built on top and for Kubernetes.
At that time – in 2018 – it was using Jenkins to run the pipelines, which was good news given our experience with Jenkins.
It comes with features such as preview environments which are a real benefit for us.
And it uses the Gitops practice, which we found very interesting because we love version control, peer review, and automation.
While adopting Jenkins X we discovered that it is first a set of good practices derived from the best performing teams, and then a set of tools to implement these practices. If you try to adopt the tools without understanding the practices, you risk fighting against the tool because it won’t fit your practices. So you should start with the practices. Jenkins X is built on top of the practices described in the Accelerate book, such as micro-services and loosely-coupled architecture, trunk-based development, feature flags, backward compatibility, continuous integration, frequent and automated releases, continuous delivery, Gitops, … Understanding these practices and their benefits is the first step. After that, you will see the limitations of your current workflow and tools. This is when you can introduce Jenkins X, its workflow and set of tools.
We’ve been using Jenkins X since the beginning of 2019 to handle all the build and delivery of our ad-tech platform, with great benefits. The main one being the improved velocity: we used to release and deploy every two weeks, at the end of each sprint. Following the adoption of Jenkins X and its set of practices, we’re now releasing between 10 and 15 times per day and deploying to production between 5 and 10 times per day. According to the State of DevOps Report for 2019, our ad-tech team jumped from the medium performers’ group to somewhere between the high and elite performers’ groups.
But these benefits did not come for free. Adopting Jenkins X early meant that we had to customize it to bypass its initial limitations, such as the ability to deploy to multiple clusters. We’ve detailed our work in a recent blog post, and we received the “Most Innovative Jenkins X Implementation” Jenkins Community Award in 2019 for it. It’s important to note that most of the issues we found have been fixed or are being fixed. The Jenkins X team has been listening to the community feedback and is really focused on improving their product. The new Jenkins X Labs is a good example.
As our usage of Jenkins X grows, we’re hitting more and more the limits of the single Jenkins instance deployed as part of Jenkins X. In a platform where every component has been developed with a cloud-native mindset, Jenkins is the only one that has been forced into an environment for which it was not built. It is still a single point of failure, with a much higher maintenance cost than the other components – mainly due to the various plugins.
In 2019, the Jenkins X team started to replace Jenkins with a combination of Prow and Tekton. Prow (or Lighthouse) is the component which handles the incoming webhook events from GitHub, and what Jenkins X calls the “ChatOps”: all the interactions between GitHub and the CI/CD platform. Tekton is a pipeline execution engine. It is a cloud-native project built on top of Kubernetes, fully leveraging the API and capabilities of this platform. No single point of failure, no plugins compatibility nightmare – yet.
Since the beginning of 2020, we’ve started an internal project to upgrade our Jenkins X setup – by introducing Prow and Tekton. We saw immediate benefits:
Faster scheduling of pipelines “runners” pods – because all components are now Kubernetes-native components.
Simpler pipelines – thanks to both the Jenkins X Pipelines YAML syntax and the ability to easily decouple a complex pipeline in multiple small ones that are run concurrently.
Lower maintenance cost.
While replacing the pipeline engine of Jenkins X might seem like an implementation detail, in fact, it has a big impact on the developers. Everybody is used to see the Jenkins UI as the CI/CD UI – the main entry point, the way to manually restart pipelines executions, to access logs and test results. Sure, there is a new UI and a real API with an awesome CLI, but the new UI is not finished yet, and some people still prefer to use web browsers and terminals. Leaving the Jenkins Plugins ecosystem is also a difficult decision because some projects heavily rely on a few plugins. And finally, with the introduction of Prow (Lighthouse) the Github workflow is a bit different, with Pull Requests merges being done automatically, instead of people manually merging when all the reviews and automated checks are green.
So if 2019 was the year of Jenkins X at Dailymotion, 2020 will definitely be the year of Tekton: our main release platform – used by almost all our projects except the ad-tech ones – is still powered by Jenkins, and we feel more and more its limitations in a Kubernetes world. This is why we plan to replace all our Jenkins instances with Tekton, which was truly built for Kubernetes and will enable us to scale our Continuous Delivery practices.
This blog post has been written by the owners of the different projects, and in particular, huge thanks to Christie Wilson, Andrea Frittoli, Adam Roberts and Vincent Demeester!
At the end of last year Dan wrote the blog post: A Year of Tekton. It was a great retrospective on what happened since the bootstrap of the project; a highly recommended read! Now that we’re getting into the swing of 2020, let’s reflect again back on 2019 and look forward to what we can expect for Tekton this year!
Tekton in 2019
We can safely say 2019 (more or less the project’s first year!) was a great year for Tekton. Just like a toddler we tried things, sometimes failed and learned a lot; we are growing fast!
The year 2019 saw 9 releases of Tekton Pipelines, from the first one (0.1.0) to the latest (0.9.2). We shared the work of creating the releases as much as possible, though many contributors are behind the work in each!!
0.1.x (Jason Hall) First Tekton Pipelines release!
0.2.x (Christie Wilson) Tekton Pipelines with graphs; without init containers!
0.3.x Chartreux C-3PO (Vincent Demeester) Released using Tekton itself!
0.4.x Aegean Brackenridge (Dan Lorenc) Exposes digests of built images
If you are curious about the naming of the release starting from 0.3.x, we decide to spice things up a bit and name our release with a composition of a cat breed and a robot name (in reference to our amazing logo, a robot cat).
Chip, our 0.5.0 mascot!
Aside from the initial project (tektoncd/pipeline), we bootstrapped a bunch of new projects:
tektoncd/cli: This project aims to provide an easy to use command line interface to interact with the tekton components. As Tekton objects are Kubernetes components you can always interact with them via the Kubernetes CLI — kubectl, but the kubectl experience can be very ‘raw’ and not very focused. The `tkn` CLI has the ambition to provide an easy to use user experience without having to know anything about kubectl (or Kubernetes for that matter).
tektoncd/dashboard: Alongside the CLI project, the Tekton Dashboard provides a user interface for the Tekton components, in a browser. It allows users to manage and view Tekton PipelineRuns and TaskRuns and the resources involved in their creation, execution, and completion.
tektoncd/catalog: Tekton pipeline is designed to provide highly shareable objects (Task, Pipeline, Condition, …), so creating a repo to store a catalog of shared Tasks and Pipelines came naturally!
tektoncd/experimental: With growing interest in Tekton came a growing number of “feature requests”. In order to be careful about how we expand the scope of Tekton pipeline while still allowing contributors to experiment, we created this repository to allow experiments to happen more easily. Experiments can graduate with enough traction. The biggest project so far is the webhooks extension which combines using the Dashboard project and Triggers to allow users to create webhooks for Git that trigger PipelineRuns.
tektoncd/operator: This project aims to provide an operator to manage installation, updation and uninstallation of tektoncd projects (pipeline, …). It has yet to be published in the community OperatorHub.
tektoncd/triggers: And speaking of the experimental repo, we have Triggers which started its life there! This project provides lightweight event triggering for Pipelines.
Looking forward into 2020 🔮
We’ve come a long way but we’ve got more to do! Though we can’t predict what will happen for sure, here is a preview of what we’d like to make happen in 2020!
Beta API, GA
As you’ve seen, we’ve made a lot of changes! Going forward we want to make sure folks who are using and building on top of Tekton can have more stability guarantees. With that in mind, we are pushing for Tekton Pipelines to have a beta release early in 2020. If you are interested in following along with your progress, please join the beta working group! Or keep an eye on our Twitter for the big announcement.
Once we’ve announced beta, users should be able to expect increased stability as we’ll be taking our lead from kubernetes and mirroring their deprecation policy, for example any breaking changes will need to be rolled out across 9 months or 3 releases (whichever is longer).
And once we get to beta, why stop there? We’d love to be able to offer users GA stability as soon as we possibly can. After we get to beta, we’ll be looking to progress the types that we didn’t promote to beta (e.g. Conditions), add any important features we don’t yet have (we’re looking at you on failure handling and “pause and resume” aka “the feature that enables manual approval”!), and then we should be ready to announce GA!
Task Interfaces and PipelineResources
Speaking of types that won’t be going beta right away: PipelineResources! PipelineResources are a type in Tekton that is meant to encapsulate and type data as it moves through your Pipelines, e.g. an image you are building and deploying, or a git commit you’re checking out and building from.
This concept was introduced early on in Tekton and bares a close resemblance to Concourse resources. However as we started trying to add more features to them, we started discovering some interesting edges to the way we had implemented them that caused us to step back and give them a re-think. Plus, some folks in our community asked the classic question “why PipelineResources” and we found our answer wasn’t as clear as we’d like!
As we started down the path of re-designing, and re-re-designing again, we started to get some clarity on what exactly it was we were trying to create: the interface between Tasks in a Pipeline! And thanks to a revolutionary request to improve our support for volumes, we finally feel we are on the right path! The next steps along this path are to add a few key features, namely the concept of workspaces (i.e. files a Task operates on) and allow Tasks to output values (aka “results”).
Once we have these in place we’ll revisit our designs and our re-designs.
tekton.dev
Hand in hand with our beta plans, we’re revamping our website! Soon at tekton.dev you’ll be able to find introductory material, tutorials, and versioned docs.
The Tekton Catalog
Besides making it easy for folks to implement cloud native CI/CD, one of the most important goals of Tekton is for folks to be able to share and reuse the components that make up your Pipeline. For example, say you want to update Slack with the results of a Task – wouldn’t it be great if there were one battle tested way to do that, with a clean interface?
But there’s so much more we want to do! We want to offer versioning and test guarantees that can make it painless for folks to depend on Tasks in the Catalog – and for companies to create Catalogs of their own.
Plus, the Catalog is a great place for us to build better interoperability even between the Tekton projects, for example with the Task that runs tkn (the Tekton CLI).
Shout outs 😻
A community is nothing without its users, contributors, adopters and friends, so we want to explicitly shout outs to our community for their tremendous effort and support in 2019 and hopefully even more in 2020.
We welcome friend requests! Please submit a PR to https://github.com/tektoncd/friends, this repository acts as a place that allows members of the ecosystem (known as “Tekton Friends”) to self-report in a way that is beneficial to everyone. We’d love to have you as a friend if your company is using Tekton and/or contributing to it 😀
Projects
Adoption of Tekton has grown and became a part of both free and commercial offerings by various companies, demonstrating that Tekton’s valuable and ready for anything
In mid-2019, Puppet launched a new cloud-native CD service called Project Nebula that’s built on Tekton Pipelines. It provides a friendly YAML workflow syntax and niceties like secrets management and a spiffy GUI on top of Tekton instance running in GCP. To coincide with the public beta of Nebula, Scott Seaward keynoted at the Puppetize PDX user conference to talk about how Tekton works under the hood. Since then, the Nebula team has contributed several PRs to the Pipelines repo and are looking forward to working on step interoperability, triggers, and other awesome upstream features in 2020.
It has been such a privilege to see more and more people get excited about Tekton and share it with the world! Here are some (but not all!!) of the great talks and tweets we saw about Tekton in 2020, not to mention our Tekton contributor summit!
Tara Hernandez recommends standards over specifications at DevOps World
Tekton birds of a feather at Devops Days Bangalore, India
Join us!
If you are interested in contributing to Tekton, we’d love to have you join us! Every tektoncd project has a CONTRIBUTING.md that can point you in the right direction, and our community contains helpful links and guidelines. Feel free to open issues, join slack, or pop into one of our working groups! Hope to see you soon 😀
The Jenkins X project started the beginning of 2019 by celebrating its first birthday on the 14th January, a big event for any open source project, and we have just celebrated our 2nd – hooray!
Jenkins X has evolved from a vision of how CI/CD could be reimagined in a cloud native world, to a fast-moving, innovative, rapidly maturing open source project.
Jenkins X was created to help developers ship code fast on Kubernetes. From the start, Jenkins X has focused on improving the developer experience. Using one command line tool, developers can create a Kubernetes cluster, deploy a pipeline, create an app, create a GitHub repository, push the app to the GitHub repository, open a Pull Request, build a container, run that container in Kubernetes, merge to production. To do this, Jenkins X automates the installation and configuration of a whole bunch of best in breed open source tools, and automates the generation of all the pipelines. Jenkins X also automates the promotion of an application through testing, staging, and production environments, enabling lots of feedback on proposed changes. For example, Jenkins X preview environments allow for fast and early feedback as developers can review actual functionality in an automatically provisioned environment. We’ve found that preview environments, created automatically inside the pipelines created by Jenkins X, are very popular with developers, as they can view changes before they are merged to master.
Jenkins X is opinionated, yet easily extensible. Built to enable DevOps best practices, Jenkins X is designed to the deployment of large numbers of distributed microservices in a repeatable and manageable fashion, across multiple teams. Jenkins X facilitates proven best practices like trunk based development and GitOps. To get you up and running quickly, Jenkins X comes with lots of example quickstarts.
Highlights of 2019
February 2019: The rise of Tekton!
In the second half of 2018, Jenkins X embarked on a journey to provide a Serverless Jenkins and run a pipeline engine only when required. That pipeline engine was based on the knative build-pipeline project which evolved into Tekton with much help and love from both the Jenkins and Jenkins X communities. The Jenkins X project completed its initial integration with Tekton in February 2019. Tekton is a powerful and flexible kubernetes-native open source framework for creating CI/CD pipelines, managing artifacts and progressive deployments.
March 2019: Jenkins X joined The Continuous Delivery Foundation!
Jenkins X joined the Continuous Delivery Foundation (CDF) as a founding project alongside Jenkins, Spinnaker, and Tekton. Joining a vendor-neutral foundation, focused on Continuous Delivery, made a lot of sense to the Jenkins X community. There had already been a high level of collaboration with the Jenkins and Tekton communities, and there have been some very interesting and fruitful (in terms of ideas) discussions about how to work better with the Spinnaker communities also.
June 2019: Project Lighthouse
When Jenkins X embarked on its serverless jenkins journey, it chose to use Prow, an event handler for GitHub events and ChatOps. Prow is used by the Kubernetes project for building all of its repos and includes a powerful webhook event handler. Prow is well proven, but heavily tied to GitHub, and not easily extendable to other SCM providers. At the end of June 2019, work commenced on a lightweight, extensible alternative to Prow, called Lighthouse. Lighthouse supports the same plugins as Prow (so you can still ask via ChatOps for cats and dogs) and the same config – making migration between Prow and Lighthouse extremely easy.
June 2019: Jenkins X Boot!
We were very busy in June – a frantic burst of activity before summer vacations! One common problem Jenkins X users were facing was the installation of Jenkins X on different Kubernetes clusters. Installing services, ensuring DNS and secrets are correct, and done in the right order is completely different from vendor to vendor, and sometimes cluster to cluster. We realised that to simplify the install, we really needed a pipeline, and whilst this may sound a little like the plot to a film, running a Jenkins X pipeline to install jx really is the best option. The jx boot command interprets the boot pipeline using your local jx binary. The jx boot command can also be used for updating your cluster.
July 2019: A New Logo!
As part of the move to the CDF the Jenkins X project took the opportunity to redesign its logo. An automaton represents the ability of Jenkins X to provide automated CI/CD on Kubernetes and more!
Second half 2019: Big focus on Stability and Reliability
The Jenkins X project has been fast paced with lots of different components and moving parts. This fast pace unfortunately led to some instability and a growth of serious issues that risked undermining all the great work there had been on Jenkins X. There has been a concerted effort by the community to increase stability and reduce outstanding issues – the graph below shows the trend over the last year, with a notable downward trend in the number of issues being created in the last 6 months.
CloudBees also aided this effort by introducing the CloudBees Jenkins X Distribution with increased testing around stabilized configurations and deployments and regular releases every month.
October 2019: Jenkins X Steering Committee inaugural meeting
The Jenkins X Bootstrap Steering Committee is tasked with organising the transition to an elected steering committee, as well as determining what responsibilities the steering committee will have in guiding the Jenkins X project.
December 2019: First Jenkins X Outreachy mentee!
Neha Gupta is adding support for Kustomize in Jenkins X, to enable Kubernetes native configuration management, while participating in Outreachy from December 2019 to March 2020. We welcome Neha’s work on Jenkins X and look forward to building on our culture of continuous mentoring!
The Jenkins X project is going to be encouraging the community to get involved with more innovation. There are a lot of great ideas to extend the continuous story with integrated progressive delivery (A/B testing, Canary and Blue/Green deployments) and Continuous Verification, alongside more platforms support. We are expecting lots of awesome new features in the CloudBees UI for Jenkins X too.
Expect lots more exciting new announcements from Jenkins X in 2020!
2019 was a crazy time to be writing software. It’s hard to believe how careless we were as an industry. Everyone was just having fun slinging code. Companies were using whatever code they found laying around on NPM, Pip, or Maven Central. No one even looked at the code these package managers were downloading for them. We had no idea where these binaries came from or even who wrote most of this stuff.
And don’t even get me started on containers! There was no way to know what was inside most of them or what they did. Yet there we were, pulling them from Dockerhub, slapping some YAML on them, and running them as root in our Kubernetes clusters. Whoops, I just dated myself. Kubernetes was a primitive system written mostly in YAML and Bash that people used to interact with before Serverless came and saved us all.
Looking back, it’s shocking that the industry is still around! How we didn’t have to cough up every Bitcoin in the world to stop our databases from getting leaked or our servers from being blown up is beyond me. Thankfully, we realized how silly this all was, and we stopped using whatever code had the most Github stars and started using protection.
We’re Under Attack
No, really. Every time you pip install, go get, or mvn fetch something, you’re doing the equivalent of plugging a thumb drive you found on the sidewalk into your production server.
You’re taking code from someone you’ve never met and then running it with access to your most sensitive data. Hopefully, you at least know their email address or Github account from the commit, but there’s no way to know if this is accurate unless you’re checking PGP signatures. And let’s be honest, you’re probably not doing that.
This might sound like I’m just fear-mongering, but I promise I’m not. This is a real problem that everyone needs to be aware of. Attacks like this are called supply-chain attacks, and they are nothing new. Just last month, an active RCE vulnerability was found in an open source package on PyPi that was being used to steal SSH and GPG credentials.
There are lots of variations on this same play that make use of different social-engineering techniques in interesting ways. One attacker used a targeted version of this to steal cryptocurrency from a few specific websites. Another group performed a “long-con” where they actually produced and maintained a whole set of useful open source images on Dockerhub for years before slowly adding, you guessed it, crypto-mining.
The possibilities are endless, terrifying, and morbidly fascinating. And they’re happening more and more often. If reading about attacks like these is your kind of thing, the CNCF has started cataloging known instances of them. Snyk also just published a post detailing how easy it is to inject code like this in most major languages — Github even hides these diffs in code review by default! Russ Cox has also been writing about this problem for a while.
Vision
OK, there’s a bit of hyperbole up there (Kubernetes doesn’t have that much bash in it), but open source is under attack, and it’s not OK. Some progress is being made in this area — GitHub and others are scanning repositories, binaries, and containers, but these tools all only work on known vulnerabilities. They have no mechanism to handle intentional, malicious ones before they are discovered, which are at least as dangerous.
The brutal fact is that there is no way to be confident about the code you find on most artifact repositories today. The service might be compromised and serve you a different package from the one the author uploaded. The maintainer’s credentials might have been compromised, allowing an attacker to upload something malicious. The compiler itself might have been hacked, or even the compiler that compiler used (PDF warning)! Or, the maintainer could have just snuck something in on purpose.
For any given open source package, we need to be able to confidently assert what code it’s comprised of, what toolchains and steps were used to produce the package, and who was responsible for each piece. This information needs to be made available publicly. A reliable, secure view of the supply-chain of every open source package will help make these attacks easier to prevent and easier to detect when they do happen. And the ability to tie each line of code and action back to a real individual will allow us to hold attackers accountable.
How Do We Get There?
We need to work as an industry to start securing open source software, piece by piece.
Artifact repositories need to support basic authentication best practices like 2FA, artifact signing, and strong password requirements. DockerHub, PyPi, and NPM support 2FA, but there’s no way to see if a maintainer of a package is using it. Most container registries don’t support signatures yet, though work is ongoing.
Software build systems need to make reproducible, hermetic builds possible and easy. Debian has started doing some great work here, but they’re basically alone. Every docker build gives you a new container digest. Tar and gzip throw timestamps everywhere. It’s possible to get reproducible builds in Go, Java, and most other major languages, but it’s not necessarily easy. See the recently published whitepaper on how Google handles much of this internally for more information.
SCM providers need strong identity mechanisms so we can associate code back to authors confidently. Git commit logs can be easily forged, and signed commits are not in common use. Even with them, you still have no idea who is on the other end of a PR, only that the signature matches. This isn’t just an issue for security. It can also be a licensing nightmare if you don’t know the real author or license of code you’re accepting.
There is value in allowing developers to work anonymously, but there is also a cost. We need to balance this with systems that apply a higher level of scrutiny to anonymous code. We also need to allow other individuals to “vouch for” patches that they’ve examined, maybe similar to how Wikipedia handles anonymous edits.
And finally, all of this needs to be tied together in secure CI/CD systems and platforms that implement binary transparency for public packages. Putting the packaging steps in the hands and laptops of developers leaves way too large an attack surface. The ability to push a package that will run in prod is the same as having root in prod. By moving the build and upload steps into secure CI/CD systems, we can reduce the need to trust individuals.
OK, but What Can I Do Now?
First, start by securing your code as much as possible. Make sure you have copies of every dependency you’re using stored somewhere. Make sure you review all code you’re using, including OSS. Set up and mandate the use of 2FA across your organization. Publish, and actually check the signatures and digests of the software you’re using.
Log enough information in your build system so you can trace back every artifact to the sources. And every deployment to the artifacts. Once you’ve done all of this, you’ll be pretty far ahead of everyone else. You’re not completely safe, though.
That’s where we need to work together. If you’re interested in helping out, there are many ways to get involved, and I’m sure there are a lot of efforts going on. We’re just getting started on several initiatives inside the Continuous Delivery Foundation, like our new Security SIG. We’re also hoping to make it easier to build and use secure delivery pipelines inside the TektonCD open source project.
We would love your help, no matter your expertise! For example, I’m far from a security expert, but I’ve spent a lot of time working on developer tools and CI/CD systems. Feel free to reach out to me directly if you have any questions or want to get involved. I’m on Twitter and Github.
Another 6 weeks, another Tekton release. It’s mostly common knowledge that Tekton’s logo is a robot cat, but it’s lesser known that the releases are named after robots and cats! Each Tekton Pipelines release is given a codename of a type of cat followed by a famous robot.
On Monday, December 2nd, Andrea Frittoli of IBM cut the v0.9.0 release, dubbed “Bengal Bender”. This release contained commits from 21 different individuals. I wanted to take the time to highlight some of the new features and API changes, as well as to point out what much of the “under the covers” work is laying the groundwork for.
So let’s jump in!
New Features and Bug Fixes
“Bengal Bender” includes a solid set of features, bug fixes and performance improvements. Apologies if I missed anything here, this list is simply what I find most exciting.
If you were at Kubecon San Diego you might have come away with the impression that Go is the language of the cloud. And while that is true to some extent, good old bash and yaml also play a huge part, especially when it comes to “glue” systems like delivery pipelines.
If you’ve spent much time working with containers, you’ve probably seen a yaml file with something like this in it:
This is a lot of complicated boilerplate just to run a simple bash script inside a container. And if you’re not deeply familiar with how bash, shells, entrypoints and shebangs all work at the system level, this is a bit opaque. It’s also prone to subtle and confusing bugs. Even if you are familiar with these things, you’ve probably wasted time debugging issues when the shell in your container is set to something you’re not used to, or the entrypoint is overridden, or “bash -c” doesn’t do what you expect.
This makes delivery pipelines harder to write, understand and maintain.
Enter script mode! Inspired by some ideas from Ahmet Alp Balkan, Jason Hall put together a proposal to make it much easier to define tasks that just need to run a simple bash script. Here’s what it looks like:
You’ll notice that there are far fewer lines of boilerplate. No more need to specify args, an entrypoint or remember the tricky “-c” part. Just specify the interpreter you want to use and the commands to run. This has already let us simplify dozens of test cases and examples!
Tekton has long suffered from poor performance around starting PipelineRuns. Christie Wilson and I did a debugging/coding session last spring to try to improve this, and identified PVC mounting as a major contributor. Unfortunately, our attempted fix didn’t work and needed to be rolled back.
A more general longer term fix is being handled by Scott Seaward and the re-resources effort, but I decided to take another stab at a short term fix. Fingers crossed that it works, but initial testing shows improvements of anywhere between 5 and 20 seconds per PipelineRun!
API Changes
There have been a few changes to the API as we start to firm things up for a beta release. We’re hoping to get the majority of these breaking changes in over the next few releases so users can start to build production systems on top of stable versions.
The breaking changes in v0.9.0 include:
Standardization of output paths for image digests
Tekton currently provides a mechanism to store the digests of container images built by Tasks. This mechanism predated the PipelineResource subsystem, and required Task authors to write these digests to a specific location at /builder/image-outputs. This change moves that to the standard path for output resources, at /workspace/output/<resource-name>.
Simplification of the cluster resource
Cluster PipelineResources make deploying and working with Kubernetes clusters from within Tasks simple. They provide mechanisms for users to declare where a cluster endpoint is and how to authenticate with it. Then, during Task execution, they automatically configure a .kubeconfig file so Kubernetes tooling can find that cluster. This release contained a few changes to make these cluster resources easier to work with.
Previously, users had to specify a name parameter twice: once in the resource name and once as a parameter to the resource. The second parameter has been removed.
Ground Work
Following the age-old programming advice above, most of the work contained in each Tekton release is in service of features that won’t be exposed until a later release. Digging into the commit log shows a detailed picture of what the community is working toward.
Here’s my editorialized version of what’s coming soon 🙂
Revamped PipelineResources
A lot of work went into cleaning up the existing PipelineResource subsystem, including the interface exposed to PipelineResource types and the types themselves. Getting these both solid will form the basis for the larger Re-resources effort, which is currently underway. This project will make the resource types extensible, allowing anyone to add and use their own types. It will also hopefully leave us with some embeddable components, so that other systems can make use of Tekton PipelineResources and the coming catalog.
API Versioning
One of the most important steps in shipping a stable API is figuring out how to make changes in a backwards compatible way. No API is perfect, so the ability to upgrade one is paramount. Vincent Demeester and his colleagues at Red Hat have been hard at work designing and implementing an API versioning system that will allow users to upgrade Tekton API versions without breaking existing workloads. This will be key to the coming beta release.
Wrapping Up
The Tekton project has been amazing to watch grow. This post only details the changes in the Tekton Pipelines release, but there has also been some awesome work done in the Triggers, CLI and Dashboard projects. Triggers now support out-of-the-box Github and Gitlab validation. The CLI has improved support for creating PipelineResources and starting Tasks interactively. Visualization is coming soon to the Dashboard! I’d like to thank everyone that has made Tekton what it is today.
The Tekton community has been hard at work shipping the right APIs and components to build cloud-native software delivery systems. If you’re using Tekton, or interested in learning more about Tekton, we’d love to hear from you. Please consider joining the community, becoming a Tekton Friend or contributing directly.
Author: Dan Lorenc (dlorenc@google.com, twitter.com/lorenc_dan, github.com/dlorenc)
Last year at Kubecon Seattle, Tekton was just an idea in the heads of a few people, and a bit of code inside knative/build-pipeline. Fast forward to today, and we have a thriving community, independent governance inside a great foundation, and we’re quickly approaching our beta release! This year has flown by, so I wanted to highlight some of the original goals of the Tekton project, and some of the milestones we’ve hit toward them.
The Start
When Christie, Jason and I started sketching out the original Tekton APIs on a whiteboard in San Francisco, we had a straightforward goal in mind: make the hard parts of building CI/CD systems easy. There are already dozens of CI/CD systems designed to run on kubernetes, and for the most part they all have to build the same basic infrastructure before they can start solving customer problems. Kubernetes makes scheduling, orchestration and infrastructure management easier than anyone could have imagined, but it still leaves users with a few pieces to assemble before they can use it as an application delivery platform.
These were some of the original whiteboard sketches on what later became the Tekton Pipeline CRD.
Things like basic DAG orchestration, artifact management, and even reliable log storage are outside the bounds of the core Kubernetes APIs. Our plan was to use the new Custom Resource Definition feature to try to define a few more “nouns”, on top of the existing Kubernetes primitives, that were better suited for Continuous Delivery workloads. In doing so, we would make it easier for people and trans to create delivery systems designed for their exact use cases, while making sure the underlying primitives allowed for some degree of compatibility.
Vague ideas are great, but it’s much more productive to collaborate with others when you have something concrete to share and get feedback on. So in August of 2018, we released the sketch and principles we used to design it on knative-dev and to a few other interested parties. The feedback was amazing. The core pipeline team on Jenkins at Cloudbees jumped in with some hard earned lessons from their experience working on the most widely used orchestration system in history. The Concourse team at Pivotal helped redesign our extensibility system based on what they learned from the successful Concourse resource model.
Then we got to work building it all out! Our goal must have really resonated with others, because we could really feel the power of open source from day one. Even when there were only a few of us working together at Google, we were part of a much larger effort. We really wouldn’t be here today without the help of our contributors, maintainers and governing committee members.
A New Home
Around the time we prepared for our first release, it started to become obvious that Tekton (then called knative/build-pipeline) needed its own home. The knative brand and community helped immensely, but Tekton was meant to provide CI/CD for everyone – not just serverless or even kubernetes users. So in February of 2019, we decided to split out the project, name it “Tekton” and donate it to the newly-forming Continuous Delivery Foundation just in time for the 0.1.0 release.
Kim Lewandowski announcing Tekton at the Open Source Leadership Summit
Open source, governance, and communities are hard. The move out of knative and into a new foundation was a big change for the project, but has proven worth the effort! Tekton still works great with other knative components, but has had the chance to grow its own community and evolve to a spot that its users need. Thanks to the community, Tekton has expanded into multiple projects like the Dashboard and CLI, and in Tekton Pipelines we have been so lucky to gain the expertise of folks like Vincent Demeester and Andrea Frittoli.
The rest of this year has felt like a blur, but I wanted to call out some major milestones we hit:
March – The first Tekton Pipelines release (v0.2.0) using Tekton itself!
June – First release of the Tekton CLI!
July – The first Tekton Pipelines release (v0.5.0) by a community member not at Google!
July – Tekton Friends repo created!
August – First release of the Tekton Dashboard!
September – First Triggers release!
October – Tekton passed 100 contributors in October!
What’s Coming Next?
We’re rapidly approaching the first Tekton beta release! As part of this effort, we evaluated our API surface and identified quite a few areas that need hardening. This includes finishing up the table-stakes requirements for a CI/CD platform – things like triggers, metadata storage and building up our catalog. The Triggers v0.1.0 release has made Tekton usable in so many new ways, and we’re just getting started there still!
Scott Seaward has has just started work on refactoring PipelineResources into an extensible system that will form the basis for the Tekton catalog, and Jason Hall is working on a metadata storage system that will help power some new ideas around software supply chain management.
If you’d like to get involved in the Tekton project, you can find us on Github, Slack or our email list . We’ll also be at Kubecon next week! Come attend one of the many sessions or the CD Summit. Here’s to Tekton in 2020!
Outreachy is a program for open source internships that specifically targets people in demographics that are underrepresented in tech. Since 2010 Outreachy has had over 400 participants making contributions to open source projects. Outreachy is one of the most effective programs for improving the diversity of open source communities. For those familiar with Google Summer of Code, Outreachy follows a very similar format.
The Continuous Delivery Foundation is a neutral home for the next generation of continuous delivery collaboration. We know that the greater number of diverse voices we have collaborating, the more effective we are as a community. Which is why we are thrilled to be participating in Outreachy for this upcoming round.
Three of the CD Foundation projects: Jenkins, Jenkins X and Tekton are offering Outreachy internship projects. The Jenkins project has participated in the previous 2 rounds of Outreachy, having a total of 4 Outreachy interns working with the community on the Jenkins Audit Log Plugin. Jenkins mentor Matt Sicker shares in this post “Outreachy has helped open my eyes to the struggles that developers from around the world are dealing with which can be improved upon to help expand our communities. For example, many countries do not have reliable internet or electricity, so the use of synchronous videoconferencing and other heavyweight, synchronous processes common to more corporate-style development are inadequate in this international context.”
In that way Outreachy is also beneficial to the mentors participating. The actual tech contributions are a bonus side effect. Jenkins participates in the program with no expectation that the interns remain part of the community – but takes a wider, long term view that this generally improves open source and tech communities as a whole. Tracy Miranda, Outreachy coordinator for the Jenkins project says “As of this year I know of 2 Outreachy alumni (non-Jenkins projects) who were hired by my employer and both mention the Outreachy program as an important stepping stone in their career journeys.”
Here are the details of this round’s CD Foundation projects which we are looking for interns for. Please help us spread the word.
Kara de la Marck, the Jenkins X Outreachy coordinator can personally speak to the benefits of the program: “Outreachy is a fantastic mentoring program that helps to onboard new contributors to a project and to open source more generally. Many participants go on to become long term contributors to open source. As an alumna, I have carried forward a deep appreciation of open source as an enabler of global collaboration, technological innovation, and community. I’m incredibly happy to welcome Outreachy participants to Jenkins X.”
Please help us spread the word, and we look forward to working with Outreachy interns and welcoming them into our community.
Jenkins Community Celebrates 15 years of Continuous Delivery Automation and Innovation at DevOps World | Jenkins World
DEVOPS WORLD | JENKINS WORLD, SAN FRANCISCO – AUGUST 14, 2019 – The Jenkins project, comprised of the community of practitioners using Jenkins®,
today celebrated its 15th birthday at DevOps
World | Jenkins World with a recap of milestones showcasing
the community’s growth and the project’s defining impact on the global software industry.
Jenkins is the world’s leading open source
automation server, used by companies large and small to implement continuous
integration (CI) and continuous delivery (CD). Originally developed in 2004 and
called Hudson, Jenkins’ impact has grown consistently over the years to the
point where experts regularly describe Jenkins as the de facto tool for CI/CD.
So far this year, the Jenkins project has been
a key driver in the formation of the Continuous
Delivery Foundation,
has continued to experience strong uptake in the use of Jenkins and will
recognize key contributors to the Jenkins project around the globe during
Thursday’s keynote session at DevOps World | Jenkins World.
In
establishing the Continuous
Delivery Foundation, the Jenkins community worked with The
Linux Foundation, CloudBees, Google and Netflix to create a new foundation for
the diverse CI/CD space. In addition to Jenkins, the Continuous Delivery
Foundation was established with several other CI/CD open source projects,
including Jenkins X, Spinnaker and Tekton. It serves as a vehicle to develop, nurture and promote open source
projects, best practices and industry specifications related to continuous
delivery.
The
Continuous Delivery Foundation fosters vendor-neutral collaboration between the
industry’s top developers, end users and vendors to further CI/CD best
practices and industry specifications. Its mission is to grow and sustain
projects that are part of the broad and growing continuous delivery ecosystem.
“This has been a great year for Jenkins, the
Continuous Delivery Foundation and open source collaboration as a whole,” said
Chris Aniszczyk, vice president at the Linux Foundation. “We all share a common
mission – to support community-based development of projects that advance the
state of software delivery. The Jenkins project has been squarely behind this
effort from day one and today the community is stronger than ever.”
Also
playing a key role in Jenkins’ transition to the CDF was CloudBees’ Tracy
Miranda. Miranda took on the dual roles of CloudBees director of open source
community and member of the governing board of the CDF. “CD is becoming a
differentiator for organizations in every industry, yet adoption remains
challenging. It’s an industry-wide problem that needs an industry-wide
solution. From the CloudBees perspective, we see it as critical to have a
neutral foundation where all agents of change can collaborate and contribute
openly,” said Miranda. “Looking ahead to the next 15 years, we need to solve the complexity of CD adoption. With
the CDF, we are well-equipped to do this in open source – building on top of
all that we have learned in the Jenkins community over the years.”
During
the period from July 1, 2018, through June 30, 2019, the Jenkins project
achieved these milestones:
46% growth in active Jenkins installations1
reporting usage data was in the period August 1, 2018 – July 31, 2019
The community experienced approximately 46 percent growth in active
installations, reaching 265,956 installations as of July 31, 2019, compared to
182,236 installations as of August 1, 2018. Active installations are defined as
Jenkins instances that report usage information back to the Jenkins project.
This number is not representative of the total Jenkins instances in use
worldwide; that number is significantly greater.
Approximately 15.8 million Jenkins developers A recent Datanyze analysis of the CI vendor landscape showed that about 66
percent of continuous integration is being run on Jenkins. With an estimated 24
million developers, globally, according to Evans Data in its 2019 Global
Developer Population and Demographics Study, approximately 15.8 million developers are
using Jenkins.
254% growth in Jenkins Pipeline jobs Finally, the combined number of defined Jenkins jobs increased during this same period from 19,946,119 in July 2018 to 30,281,905, or 52 percent growth. Specifically, Jenkins Pipeline jobs grew 254 percent in the same period. The dramatic growth in Jenkins Pipeline jobs demonstrates that organizations are accelerating their investment in modern software pipeline automation practices with Jenkins.
“Over the past 15 years, the Jenkins project has revolutionized the way software is built and delivered,” said Jenkins creator Kohsuke Kawaguchi, who also serves as chief scientist at CloudBees. “We have touched every industry and made a difference to every software team in the world.”
1The Jenkins community tracks statistics from active Jenkins installations that transmit usage information back to the project. The numbers do not represent a majority of Jenkins installations, only those who choose to report. Therefore, the numbers are conservative.
About the Continuous Delivery Foundation Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. The Continuous Delivery Foundation (CDF) is a Linux Foundation initiative that serves as the vendor-neutral home for many of the fastest-growing projects, including Jenkins, Jenkins X, Spinnaker, and Tekton. The CDF fosters collaboration between the industry’s top developers, end users and vendors to further continuous delivery best practices. For more information about the CDF, please visit https://cd.foundation.
Every year before the DevOps World | Jenkins World conference, Jenkins contributors gather for a contributor summit. Earlier this year, the CD Foundation became the new home for Jenkins as well as Jenkins X, Tekton and Spinnaker projects. CD Foundation serves as the new home for many of the fastest-growing projects for continuous delivery as well as a neutral home for collaboration in the CI/CD space. In that spirit of collaboration, this year’s contributor summit is being expanded to be a CDF Contributor Summit.
CDF Contributor Summit will be held on Monday August 12, 2019 in San Francisco, USA just before DevOps World | Jenkins World. It is a free event open to contributors of the open source projects. So far we will have community members from the Jenkins, Jenkins X and Tekton projects attending the summit.
The summit brings together community members to learn, meet and help shape the future of the projects. In the CDF community we value all types and sizes of contributions and love to welcome new participants who want to contribute to one or more projects.
The morning portion of the summit will consist of presentations by core committers of the projects. Presentations will highlight what each effort is about and what community members can do to help. We will cover a range of topics from technical aspects to community topics.
In the afternoon we will break into Birds of a Feather table for in-depth discussion and collaboration on different topics of interest. Bring your laptop, come prepared with questions and ideas, and be ready to meet and get coding with other contributors.
Agenda: (tentative)
9:00 am – Kickoff & Welcome with coffee/pastries
10:00 am – Project Updates
12:00 pm – Lunch
1:00 pm – BoF/Unconference
3:00 pm – Break
3:30 pm – BoF/Unconference
4:30 – Ignite Talks
5:00 pm – Wrap-up
To join the summit, please sign up with the main project you contribute to or are interested in contributing to:
