Skip to main content
All Posts By


Let’s Continuous CI/CD in China!

By Blog

We held the first CI/CD Meetup in China on February 29th. There were more than 5000 attendees who joined us together.  It is time to do more CI/CD, so we will hold the second CI/CD Meetup in China on June 19th.

There will be four topics focused on: Pipeline, CI, SCM, and Automation Testing. 

1. Build a dedicated pipeline engine: Jenkins shared library in-depth design and practice

Combining actual business, design patterns, and Jenkins features step by step, tells how to support large-scale, complex pipelined practice based on Jenkins SharedLibrary expansion library: from how native solutions solve actual business needs to structured design to solve atoms. The minimum unit) abstraction and basic capabilities are enhanced, and then to dynamic refactoring to build a pipeline execution engine.

Gu Zheng

JD Software Architect

2. Construction of engineering efficiency data in Continuous Integration

Engineering performance data is an important tool for improving R&D efficiency. Through years of tool construction, Didi has already possessed a relatively complete R&D tool chain, and the improvement of engineering efficiency has shifted from the original single-point capability optimization to the optimization of the entire process. As the most important continuous integration and continuous release process in the entire R&D process, how to perceive and improve it through data driving will be the key to continuously improve the engineering efficiency.

Zhou Fan

Didi Chuxing, Development Tool Team Leader

Personal profile: Graduated from Beijing University of Posts and Telecommunications in 2007 with a bachelor’s degree and a master’s degree in communications engineering. After graduation, he joined Google directly. He has worked in the Beijing office and the US headquarters for more than 10 years. Gained experience in development, testing and operation and maintenance. In 2018, he joined Didi Chuxing as the head of the R&D tool team, leading the team to improve the company’s R&D tools and engineering efficiency.

3. Ways to improve software quality: code review

Code review is a common topic, so why exactly do code review, how to do it, how to improve efficiency, those that can be manual, those that can be automated, what detailed considerations, let industry veterans take you around.

Li Peng

Senior experts in SCM and DevOps

Personal profile

With more than 20 years of experience in the software industry, he has comprehensive experience in development, operation, maintenance and management. He has worked in many companies such as Xin’an Century, Motorola, Ericsson, Alpha Motors, Horizon Robotics and so on. Familiar with the industry’s CMMI, TL9000, ISO9001: 2000 standards, proficient in various development methodologies; proficient in software quality management, configuration management, CICD, automated testing and other key aspects of DevOps.

4. Test environment, practice and implementation of full-stack DevOps platform

Introduce the development and testing side of Minsheng Bank, the DevOps platform architecture, core capabilities, and implementation. There are currently more than 4,000 active users on the DevOps platform; more than 240 supported projects; full-stack support for back-end, front-end and client.

Hu Wenan

Minsheng Bank, DevOps architect

Personal profile:

Hu Wenan, DevOps Architect of Minsheng Bank, is responsible for organization-level R&D specifications and process formulation, as well as the planning and construction of the PaaS platform and DevOps platform.

Register Now (in Chinese):

Kata Containers in Screwdriver

By Blog

Written by Lakshminarasimhan Parthasarathy


Screwdriver is a scalable CI/CD solution which uses Kubernetes to manage user builds. Screwdriver build workers interfaces with Kubernetes using either “executor-k8s” or “executor-k8s-vm” depending on required build isolation. 

executor-k8s runs builds directly as Kubernetes pods while executor-k8s-vm  uses HyperContainers along with Kubernetes for stricter build isolation with containerized Virtual Machines (VMs). This setup was ideal for running builds in an isolated, ephemeral, and lightweight environment. However, hyperd is now deprecated, has no support, is based on an older Docker runtime and it also required non-native Kubernetes setup for build execution. Therefore, it was time to find a new solution.

Why Kata Containers ?

Kata Containers is an open source project and community that builds a standard implementation of lightweight virtual machines (VMs) that perform like containers, but provide the workload isolation and security advantages of VMs. It combines the benefits of using a hypervisor, such as enhanced security, along with container orchestration capabilities provided by Kubernetes. It is the same team behind HyperD where they successfully merged the best parts of Intel Clear Containers with RunV. As a Kubernetes runtime, Kata enables us to deprecate executor-k8s-vm and use executor-k8s exclusively for all Kubernetes based builds.

Screwdriver Journey to Kata

As we faced a growing number of instabilities with the current HyperD – like network and devicemapper issues and IP cleanup workarounds, we started our initial evaluation of Kata in early 2019 ( and identified two major blockers to move ahead with Kata:

1. Security concern for privileged mode (required to run docker daemon in kata)

2. Disk performance. 

We recently started reevaluating Kata in early 2020 based on a fix to “add flag to overload default privileged host device behaviour” provided by Containerd/cri (, but still we faced issues with disk performance and switched from overlayfs to devicemapper, which yielded significant improvement. With our two major blockers resolved and initial tests with Kata looking promising, we moved ahead with Kata.

Screwdriver Build Architecture

Replacing Hyper with Kata led to a simpler build architecture. We were able to remove the custom build setup scripts to launch Hyper VM and rely on native Kubernetes setup. 


To use Kata containers for running user builds in a Screwdriver Kubernetes build cluster, a cluster admin needs to configure Kubernetes to use Containerd container runtime with Cri-plugin.


Screwdriver build Kubernetes cluster (minimum version: 1.14+) nodes must have the following components set up for using Kata containers for user builds. 


Containerd is a container runtime that helps with management of the complete lifecycle of the container.


CRI-Containerd plugin:

Cri-Containerd is a containerd plugin which implements Kubernetes container runtime interface. CRI plugin interacts with containerd to manage the containers.


Image credit: containerd / cri. Photo licensed under CC-BY-4.0.


Image credit: containerd / cri. Photo licensed under CC-BY-4.0





To debug, inspect, and manage their pods, containers, and container images.



Builds lightweight virtual machines that seamlessly plugin to the containers ecosystem.


Image credit: kata-containers Project licensed under Apache License Version 2.0



Routing builds to Kata nodes in Screwdriver build cluster

Screwdriver uses Runtime Class to route builds to Kata nodes in Screwdriver build clusters. The Screwdriver plugin executor-k8s config handles this based on: 

  1. Pod configuration:
apiVersion: v1
kind: Pod
  name: kata-pod
  namespace: sd-build-namespace
    sdbuild: "sd-kata-build"
    app: screwdriver
    tier: builds
  runtimeClassName: kata
  - name: "sd-build-container"
    image: <<image>>
    imagePullPolicy: IfNotPresent
  1. Update  the plugin to use k8s in your buildcluster-queue-worker configuration

    # Default executor
    plugin: k8s
        - 'rhel6'
      weightage: 0
            # The host or IP of the kubernetes cluster
            host: kubernetes.default
            # Privileged mode, default restricted, set to true for trusted container runtime use-case
            privileged: false
            automountServiceAccountToken: false
            dockerFeatureEnabled: false
                    # Number of cpu cores
                    micro: "0.5"
                    low: 2
                    high: 6
                    turbo: 12
                    # Memory in GB
                    micro: 1
                    low: 2
                    high: 12
                    turbo: 16
            # Default build timeout for all builds in this cluster
            buildTimeout: 90
            # Default max build timeout
            maxBuildTimeout: 120
            # k8s node selectors for approprate pod scheduling
            nodeSelectors: {"dedicated":"screwdriver-kata"}
            preferredNodeSelectors: {}
            annotations: {}
            # support for kata-containers-as-a-runtimeclass
            runtimeClass: "kata"
        # Launcher image to use
        launchImage: screwdrivercd/launcher
        # Container tags to use
        launchVersion: stable
        # Circuit breaker config
                # in milliseconds
                timeout: 10000
        # requestretry configs
            # in milliseconds
            retryDelay: 3000
            maxAttempts: 5

Production rollout

  1. Test out the new setup with pilot users
  2. Route a percentage of traffic to Kata nodes using the weightage configuration
  3. Based on the limitation “Kata default guest kernel does not support IA32 bit binaries”, maintain a list of containers to exclude; only route builds to nodes with Kata when the container is not in the list


The below tables compare build setup and overall execution time for Kata and Hyper when the image is pre-cached or not cached.

Image: node12with Image cached in nodeKata (with 1 min wait in build)Hyper (with 1 min wait in build)
Setup time28 secs50 secs
Overall execution time1 min 32 secs1 min 56 secs
Image: node12without Image cached in nodeKata (with 1 min wait in build)HyperD (with 1 min wait in build)
Setup time51 secs1 min 32 secs
Overall time1 min 55 secs2 min 40 secs

Known problems

While the new Kata implementation offers a lot of advantages, there are some known problems we are aware of with fixes or workarounds:

  1. Run images based on Rhel6 containers don’t start and immediately exit 
  1. Yum install will hang forever
Before fixAfter fix
sh-4.1# time yum remove wget -yreal 6m22.190suser 2m38.387ssys 3m38.619s
sh-4.1# time yum install wget -yreal 6m23.407suser 2m39.387ssys 3m42.606s
sh-4.1# time yum remove wget -yreal 0m4.774suser 0m0.783ssys 0m0.123s
sh-4.1# time yum install wget -yreal 0m2.169suser 0m1.760ssys 0m0.298s
  1. 32-bit executable cannot be loaded refer kata issue 
  • To workaround/mitigate we maintain a container exclusion list and route to current hyperd setup and we have plans to eol these containers by Q4 of this year.
  1. Containerd IO snapshotter – Overlayfs vs devicemapper for storage driver
  • Devicemapper gives better performance with kata
1024000000 bytes (976.6MB) copied, 19.325605 seconds, 50.5MB/s1024000000 bytes (976.6MB) copied, 5.860671 seconds, 166.6MB/s
  1. Image stored in both sys-root and devicemapper volume, consuming both volume disk space 

Compatibility List

In order to use this feature, you will need these minimum versions:


Thanks to the following contributors for making this feature possible:

Questions & Suggestions

We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.

Introducing Our Newest CDF Ambassador – Oscar Medina

By Blog

Hi Everyone,

My name is Oscar Medina, and I am thrilled to be part of this fantastic community. I have spent over 22 years in the technology industry, and have seen things come and go.

One thing that excites me these days (aside from the plethora of outdoor activities), is the paradigm shift I’ve seen throughout my career when it comes to systems architecture.  

Microservices and container orchestration is not going away as other things have. This is why I am committed to spreading the word and helping educate folks on what the Continuous Delivery Foundation is all about.

Over the past 4.5 years or so, I have spent a lot of time in the open-source world. I am currently a Developer Advocate for the Jenkins X project, which is also now under the CD Foundation umbrella along with other projects such as Spinnaker, Jenkins, Tekton, and Jenkins X, of course.

I look forward to meeting you at different organized events, virtual or hopefully in person in the future.

Standup Paddle Boarding on Lake Tahoe, California

My coding buddy, Ginger loves the outdoors too!


Introducing Our Newest CDF Ambassador – BMK Lakshminarayanan

By Blog

Hello CDF Members,

I am BMK Lakshminarayanan from New Zealand. I am excited to join on the other great ambassadors’ line-up with Continuous Delivery Foundation (CDF) as a newly appointed CDF Ambassador.

About me:

I am a passionate Solutions Architect over 20 years of ICT experience working with Bank of New Zealand. I am a hands-on engineer, architect and worked on various challenging assignments ranging from desktop applications to distributed systems.

I am #DevOps #ContinuousDelivery advocate and evangelist for modern engineering & developer practices including helping developer productive, effective and efficient at the same time simple methods, approaches to software architecture.

I am passionate about sharing and learning with the community. Outside of my work, I run community groups and host CNCF New Zealand #meetup for Cloud-Native enthusiasts and The Future ICT to help students, people returning to work or looking for career opportunities in ICT.

I am also CNCF & DevOps Institue ambassador with a commitment to connecting the Humans of DevOps and Modern IT to advance the Skills, Knowledge, Ideas & Learning (SKIL). I am a frequent speaker at local meetups and international in-person & virtual conferences and also the Core organiser of DevOpsDays New Zealand and co-chair Cloud-Native Summit Wellington conferences. I am passionate about engaging, connecting & learning with various community members & leaders.

I am excited to associate with CDF as an Ambassador in promoting core values of CDF, open-source and vendor-neutral CI/CD tools. I am enthusiastic about the opportunity of contributing to and supporting the global community of CD Foundation & growing continuous delivery ecosystem.

Let us learn, share, care and grow together.

Please feel free to reach me out on LinkedIn or Twitter: @LBMKRISHNA

Introducing Our Newest CDF Ambassador – Eduardo Arango

By Blog

Carlos Eduardo Arango Gutierrez – Red Hat (

Eduardo is a performance engineer at Red Hat, working on the OpenShift performance & latency sensitive applications (PSAP) . Eduardo is also a Computer Science PhD student at Universidad del Valle, Cali, Colombia, working on containerized distributed systems for research computing, with high focus on automated workflows and GitOps.

His research interests include High Performance Computing, Distributed systems, Dependency management, Linux containers and most recently, Container orchestration. 

Over the past 5 years Eduardo has focused on enabling researchers to build and deploy performance sensitive applications with containers on distributed environments, by creating tutorials, talks and meetups around how to bridge Research computing and Cloud Native ecosystems.

Introducing Our Newest CDF Ambassador – Helen Beal

By Blog

Hello there! I’m Helen Beal, a new CD Foundation Ambassador from Chichester in the UK. I’m also Chief Ambassador at DevOps Institute so you can tell I’m a huge fan of the power of community. I’m also a DevOps coach, writer and speaker and strategic advisor. Books and words are a huge part of my life – I read constantly and have also written several novels with more in the pipeline. I also love playing Scrabble and Bananagrams.

This is me at home in Chichester, in the beautiful Priory Park. Behind me is the priory where William Blake was tried for sedition in 1803.

I’m really excited to have this opportunity to work closely with the Continuous Delivery community as it’s such an integral part of what we do in DevOps and I’ve been working with the software development lifecycle for my whole career, starting with Lotus Notes (remember that?!) in 1995! I write and speak about many different aspects of DevOps – recently I’ve been really focused on neuroscience in the workplace and value stream management. I also just did my first talk on the relationship between community and capitalism for TechStrongCon. Here’s a beautiful visual rendition of a recent talk by the wonderful MindsEyeCCF.

When I’m not DevOpsing, I tend to be out enjoying the beautiful British countryside. I’m a volunteer warden at a local nature reserve, Kingley Vale, where I pick up litter, ask people to put their dogs on leads and monitor species like the Chalk Hill Blue butterfly and one of our two UK snake species, the (lightly venomous) adder. Here’s me with a baby tawny owl.

I’m excited to be contributing to this community and meeting new people and learning new things. You can follow me on InfoQ here and Medium here. Find me on LinkedIn here and Twitter here.

Introducing Our Newest CDF Ambassador – Alexander Raul

By Blog

My name is Alexander Raul – and I am extremely happy to join the Continuous Delivery Foundation as a Community Ambassador!  

I am the CEO of Rackner, which is a cloud native consultancy focused on Kubernetes and Open Source – so my day to day is really driven by projects in the ecosystem.  Continuous Delivery is a piece which doesn’t get as much credit as it deserves and where there’s still plenty of work to be done.  

I am looking forward to introducing projects like Spinnaker and Tekton to developers all over the globe – and let’s be clear, Continuous Delivery should make the developer’s job easier while improving operational capability.  If it only does one of the two, there’s probably a better solution.

Introducing Our Newest CDF Ambassador – Shlomo Bielak

By Blog

Shlomo Bielak here. I am a CTO at a boutique SI in Canada trying to help anyone and everyone understand how to implement Continuous Deployment.

I think we have got down Continuous Delivery and Integration. Not-so-much the auto-deploy to production; that **** is complicated. I enjoy sharing on our webinars or with customers our Star-Trek approach to governance and CI/CD pipeline fitness.

Demo…Demo…Demo = Believe. I am all about sharing the practicing side of DevSecOps within CI/CD. Having invented governance engineering which is the operating model for security within CI/CD we know the complexities of making security fast within a DevOps mode. I share the model, process, milestones, and clear tasks to achieve those milestones, using an inner-source training plan to simplify your CI/CD journey.

I love publishing whitepapers and archetypes/exemplars to the market to better understand CI/CD which is their thought-leadership’s starting point – knowing your goal.I have been the keynote at CDF sponsor companies where they are looking to understand what the enterprise requires to be successful. I make sure my content hits the pain points and some self-deprecating laugh-points.

Today my desk is set for COVID. Tomorrow it is ready for conferences. Happy to be here supporting the CDF. Its business goal is at the core of our practice.

Introducing Our Newest CDF Ambassador – Tiffany Jachja

By Blog

Hi Readers,

2020 has been a crazy year, yet the opportunities remain to connect, learn, and share throughout our communities, and so I’m thrilled to join the Continuous Delivery Foundation. As a newly minted member of the CDF Ambassadors program, I look forward to getting to know everyone. 

A little bit more about me: my name is Tiffany Jachja. I’ve lived in Maryland almost all my life (go Old Bay!). One of my goals is to become a catalyst for better software delivery. 

Me, the one time I decided to leave Maryland and live 2,000 miles away from home. 

I work as an evangelist at Harness. This is my team.

We believe in empowering developers to move fast without breaking things.

I joined at the start of 2020, excited to travel, connect, and share my experiences around software delivery. 

Of course with the shelter in place policies, the travel bit did not pan out. But I’m grateful and fortunate for the opportunities to contribute to digitally! 

Observe2020 was a day-long conference held in April about Observability. 

ONUG Digital Live was ONUG’s first virtual event held in May 2020. 

I’ve been enjoying the fact that many industry events and sessions are now free to attend. It gives people who normally would not be able to attend an event, the opportunity to grow new skills and learn more about specific topics.

As you can tell, I do enjoy being on stage.I look forward to a healthier and safer time. 

I’m grateful for all the had opportunities I’ve had to help organizations and teams accelerate their DevOps journeys. It’s very rewarding to be a part of a team that’s hit their stride and can deliver effectively.

Before joining Harness, I was a consultant at Red Hat. I focused on cloud-native application development, so helping enterprises adopt and work with applications living in the cloud. I spent the latter half of my time at Red Hat, focusing on DevOps practices and culture. 

It’s important to work with your people, processes, and technology properly when going on transformation journeys.

An area we can improve on within the tech space is sharing stories and leveraging the experiences of others. 

I believe becoming a CDF Ambassador gives me the opportunities to help drive that mission further. 

Stay passionate, caring, and safe during these times. 



From Jenkins – Join us for online UI/UX hackfest on May 25-29!

By Blog

Originally published by Oleg Nenashev on the Jenkins blog.

On behalf of the Jenkins User ExperienceDocumentation and Advocacy and Outreach special interest groups, we are happy to announce the online UI/UX hackfest on May 25-29! Everyone is welcome to participate, regardless of their Jenkins development experience.

The goal is to get together and work on improving Jenkins user experience, including but not limited to user interface and user documentation. We also invite you to share experiences about Jenkins and to participate in UX testing. The event follows the Jenkins is the Way theme and the most active contributors will get special edition swag and prizes!

register button

Event plan

This hackfest is NOT a hackathon. We do not expect participants to dedicate all their time during the event timeframe, but hop-in/hop-out as their time allows. Everybody can spend as much time as they are willing to dedicate. Spending a few days or just a few hours is fine, any contributions matter regardless of their size. Jenkins development experience is not required, we have newcomer-friendly stories for those who want to start contributing to the project. We will also have a 24/7 jenkinsci/hackfest Gitter chat for Q&A and coordination between contributors.

There will be 3 main tracks:

  • User Interface – Improve look&feel and accessibility for Jenkins users, work on new read-only interface for instances managed with configuration as code, create and update Jenkins themes, and many other topics. This track is coordinated by the UX SIG.
  • User Documentation – Improve and create new user documentationtutorials and solution pages. Also, there is ongoing documentation migration from Wiki to and plugin repositories. This track is coordinated by the Documentation SIG.
  • Spread the word – Write user stories for Jenkins Is The Way site and the Jenkins blog, post about your Jenkins user experience and new features, record overview and HOWTO videos, etc. This track is coordinated by the Advocacy and Outreach SIG.

We are working on publishing project ideas and issues for the listed tracks. The current list can be found on the UI / UX hackfest event page, this list will be finalized by the beginning of the hackfest. You are welcome to propose your own projects within the User Experience theme.

During the event, we will organize online meetups and ad-hoc training sessions in different timezones. All these sessions will be recorded and shared on our YouTube channel. There are no mandatory sessions you must attend, you are welcome to join ones remotely or watch the recordings. After the event we will invite participants to demo their projects at online meetings or recorded sessions.


register button

P.S: Note that the registration form has a question top 3 things we could change in Jenkins to improve your user experience. We would appreciate your response there!


Please use the following contacts to contact organizers:


Swag and Prizes

Thanks to our sponsors (CloudBees, Inc. and Continuous Delivery Foundation), we are happy to offer swag to active contributors!

  • 50 most-active contributors will get an exclusive “Jenkins Is The Way” T-shirt and stickers
  • Active contributors will get Jenkins stickers and socks
  • We are working on special prizes for top contributors, to be announced later
Jenkins Is The Way T-shirt
Jenkins Stickers


We thank all contributors who participate in this event as committers! We especially thank all reviewers, organizers and those who participated in the initial program reviews and provided invaluable feedback. In particular, we thank User ExperienceDocumentation and Advocacy and Outreach SIG members who heavily contributed to this event.

We also thank sponsors of the event who make the swag and prizes possible: CloudBees, Inc. and Continuous Delivery Foundation (CDF). In addition to swag, CloudBees donates working time for event hosts and reviewers. CDF also sponsors our online meetup platform which we will be using for the event.