Skip to main content
Category

Project

From Jenkins – GitHub App authentication support released

By Blog, Project

Originally posted on the Jenkins blog by Tim Jacomb

I’m excited to announce support for authenticating as a GitHub app in Jenkins. This has been a long awaited feature by many users.

It has been released in GitHub Branch Source 2.7.0-beta1 which is available in the Jenkins experimental update center.

Authenticating as a GitHub app brings many benefits:

  • Larger rate limits – The rate limit for a GitHub app scales with your organization size, whereas a user based token has a limit of 5000 regardless of how many repositories you have.
  • User-independent authentication – Each GitHub app has its own user-independent authentication. No more need for ‘bot’ users or figuring out who should be the owner of 2FA or OAuth tokens.
  • Improved security and tighter permissions – GitHub Apps offer much finer-grained permissions compared to a service user and its personal access tokens. This lets the Jenkins GitHub app require a much smaller set of privileges to run properly.
  • Access to GitHub Checks API – GitHub Apps can access the the GitHub Checks API to create check runs and check suites from Jenkins jobs and provide detailed feedback on commits as well as code annotation

Getting started

Install the GitHub Branch Source plugin, make sure the version is at least 2.7.0-beta1. Installation guidelines for beta releases are available here

Configuring the GitHub Organization Folder

Follow the GitHub App Authentication setup guide. These instructions are also linked from the plugin’s README on GitHub.

Once you’ve finished setting it up, Jenkins will validate your credential and you should see your new rate limit. Here’s an example on a large org:

GitHub app rate limit

How do I get an API token in my pipeline?

In addition to usage of GitHub App authentication for Multi-Branch Pipeline, you can also use app authentication directly in your Pipelines. You can access the Bearer token for the GitHub API by just loading a ‘Username/Password’ credential as usual, the plugin will handle authenticating with GitHub in the background.

This could be used to call additional GitHub API endpoints from your pipeline, possibly the deployments api or you may wish to implement your own checks api integration until Jenkins supports this out of the box.

Note: the API token you get will only be valid for one hour, don’t get it at the start of the pipeline and assume it will be valid all the way through

Example: Let’s submit a check run to Jenkins from our Pipeline:

pipeline {
  agent any

  stages{
    stage('Check run') {
      steps {
        withCredentials([usernamePassword(credentialsId: 'githubapp-jenkins',
                                          usernameVariable: 'GITHUB_APP',
                                          passwordVariable: 'GITHUB_JWT_TOKEN')]) {
            sh '''
            curl -H "Content-Type: application/json" \
                 -H "Accept: application/vnd.github.antiope-preview+json" \
                 -H "authorization: Bearer ${GITHUB_JWT_TOKEN}" \
                 -d '{ "name": "check_run", \
                       "head_sha": "'${GIT_COMMIT}'", \
                       "status": "in_progress", \
                       "external_id": "42", \
                       "started_at": "2020-03-05T11:14:52Z", \
                       "output": { "title": "Check run from Jenkins!", \
                                   "summary": "This is a check run which has been generated from Jenkins as GitHub App", \
                                   "text": "...and that is awesome"}}' https://api.github.com/repos/<org>/<repo>/check-runs
            '''
        }
      }
    }
  }
}

What’s next

GitHub Apps authentication in Jenkins is a huge improvement. Many teams have already started using it and have helped improve it by giving pre-release feedback. There are more improvements on the way.

There’s a proposed Google Summer of Code project: GitHub Checks API for Jenkins Plugins. It will look at integrating with the Checks API, with a focus on reporting issues found using the warnings-ng plugin directly onto the GitHub pull requests, along with test results summary on GitHub. Hopefully it will make the Pipeline example below much simpler for Jenkins users 🙂 If you want to get involved with this, join the GSoC Gitter channel and ask how you can help.

Tekton: Wow! We’re beta now!

By Blog, Project

You may have heard that Tekton Pipelines is now beta! That’s not beta like the video format but beta like Kubernetes! Okay I’ll stop trying to make jokes, because compatibility is no laughing matter for folks who want to build on top of and use Tekton, and that’s why we’ve declared beta, so that you can feel more confident in using it.

What exactly does beta mean for Tekton?

So what does beta mean exactly? It means for Tekton what it means for Kubernetes, and it boils down to two things:

  1. Features that are beta will not be removed; they might change but you can count on the features themselves sticking around
  2. Backwards incompatible changes to the API will be avoided; if they do have to happen you will be given at least 9 months worth of releases to migrate to the new way of doing things

You might be wondering what “the API” means in this context – good question! It’s the specifications of the CRDs themselves and runtime details like the special directories that Tekton makes.

Not all of Tekton is beta however! Right now it’s just Tekton Pipelines and it’s only the following CRDs:

  • Tasks, ClusterTasks and TaskRuns
  • Pipelines and PipelineRuns


This means that other types that you might like, such as Conditions and PipelineResources (see the next section!) are still alpha and don’t (yet!) have the same beta level guarantees.

You can always refer to our API compatibility docs in our repo if you forget!

What about PipelineResources?

What about them indeed! If you are part of the Tekton community, you’ll know that we keep going back and forth on our love/hate-able PipelineResources – the feature you love until it doesn’t work.

A few months ago, our “difficult to understand, hard to debug” friend was challenged by the community: what would the Tekton world look like without PipelineResources? And when we went on that journey, we discovered features which PipelineResources gave us which were super useful on their own:

So we focused on adding those features and brought them to beta. In the meantime, we keep asking the question: do we still need PipelineResources? And what would they look like if redesigned with workspaces and results? We’re still asking those questions and that’s why PipelineResources aren’t beta (yet)!

We know some users really love them: “There are dozens of us,” – @dlorenc. So we haven’t given up on them yet, and there are some things that you just still can’t do well without them: for example, how do you consistently represent artifacts such as images moving through Pipelines? You can’t! So the investigation continues.

In the meantime, we’ve made Task equivalents of some of our PipelineResources in the Tekton catalog, such as PullRequests, GCS, and git.

Tekton Website is Live Now!

Hooray! Our shiny new site is live! Right this way -> https://tekton.dev/

Tekton Documentation is now hosted on the website at https://tekton.dev/docs/. And interactive tutorials are hosted at https://tekton.dev/try/. There is just one interactive tutorial hosted right now but more are in process to get published, so watch this space!

What’s coming up next?

We’re hard at work on more nifty Tekton stuff to make your CI/CD Pipelines more powerful and more portable by achieving Tekton’s mission:

Be the industry-standard, cloud-native CI/CD platform components and ecosystem.

Check out more on our mission and our 2020 roadmap in our community repo.

THANK YOU!!! ❤️

Thanks to all of the many amazing contributors who have gotten us to this point! The list below is people credited in Tekton Pipelines release notes, but for the complete list of everyone contributing to Tekton check out our devstats!

From Jenkins – Validating JCasC configuration files using Visual Studio Code

By Blog, Project

Originally posted on the Jenkins Blog by Sladyn Nunes

Configuration-as-code plugin

Problem Statement: Convert the existing schema validation workflow from the current scripting language in the Jenkins Configuration as Code Plugin to a Java based rewrite thereby enhancing its readablity and testability supported by a testing framework for the same. Enhance developer experience by developing a VSCode Plugin to facilitate autocompletion and validation which would help the developer write correct yaml files before application to a Jenkins Instance.

The Configuration as Code plugin has been designed as an opinionated way to configure Jenkins based on human-readable declarative configuration files. Writing such a file should be feasible without being a Jenkins expert, just translating into code a configuration process one is used to executing in the web UI. The plugin uses a schema to verify the files being applied to the Jenkins instance.

With the new JSON Schema being enabled developers can now test their yaml file against it. The schema checks the descriptors i.e. configuration that can be applied to a plugin or Jenkins core, the correct type is used and help text is provided in some cases. VSCode allows us to test out the schema right out of the box with some modifications. This project was built as part of the Community Bridge initiative which is a platform created by the Linux Foundation to empower developers — and the individuals and companies who support them — to advance sustainability, security, and diversity in open source technology. You can take a look at the Jenkins Community Bridge Project Page

Steps to Enable the Schema Validation

a) The first step includes installing the JCasC Plugin for Visual Studio Code and opening up the extension via the extension list. Shortcut for opening the extension list in VSCode editor using Ctrl + Shift + X.

b) In order to enable validation we need to include it in the workspace settings. Navigate to File and then Preference and then Settings. Inside settings search for json and inside settings.json include the following configuration.

{
"yaml.schemas": {
        "schema.json": "y[a]?ml"
    }
}

You can specify a glob pattern as the value for schema.json which is the file name for the schema. This would apply the schema to all yaml files. eg: .[y[a]?ml]

c) The following tasks can be done using VSCode:

a) Auto completion (Ctrl + Space):
  Auto completes on all commands.
b) Document Outlining (Ctrl + Shift + O):
Provides the document outlining of all completed nodes in the file.

d) Create a new file under the work directory called jenkins.yml. For example consider the following contents for the file:

jenkins:
  systemMessage: “Hello World”
  numExecutors: 2
  1. The above yaml file is valid according to the schema and vscode should provide you with validation and autocompletion for the same.

Screenshots

vscode
userDocs1
userDocs2

We are holding an online meetup on the 26th February regarding this plugin and how you could use it to validate your YAML configuration files. For any suggestions or dicussions regarding the schema feel free to join our gitter channel. Issues can be created on Github.

From Jenkins X – Asking and Finding Help: Outreachy

By Blog, Project

Originally posted on the Jenkinx X blog by Neha Gupta

Neha Gupta is adding support for Kustomize in Jenkins X, to enable Kubernetes native configuration management, while participating in Outreachy from December 2019 to March 2020.

Outreachy open-source contribution for applicants — Asking/Finding help

This blog might be helpful for beginners who are fear-stricken or I would say hesitant to ASK, to get lost in the new world while trying to understand any open source project, fear of asking questions that may sound stupid later on or are very obvious! First of all.. Relax!

  • Everyone starts from somewhere and has a learning curve!..
  • There are some pre-requisites that may help you get into open-source development better..
  • Learn basics of git operations. (https://learngitbranching.js.org , I find this easy and helpful).
  • Try to find an open-source project (remember : you’re going to contribute to a part of it, so it’s okay if some/many things doesn’t make sense in the beginning, because it’s easier to write code than to understand someone else’s code).
  • For selecting a project you may also look for Google-Summer-of-Code, Outreachy, Google-Code-In, RSoC and other open-source programs and their organisations that helps people/students/aspiring developers to find your best interest communities and projects.

NOTE : Beware! seeing too many organisations and projects will only confuse you, so start with only one or max 2 projects, try to deep-dive and focus on them.

After selecting the project :

  • Connect with the community through their communication channels for both developers and users (example : Slack, IRC-Cloud, Zulip, Riot etc )
  • Try to read the documentation and understand the overall structure and purpose of the project you’re starting to work on.
  • If you don’t understand something functionality wise — just ask! Ask on the communication channel.
  • If you are facing any error — Google search it, or try to look into the existing issues, if you’re not able to move forward and you’re stuck on the same error for more than 45 mins, just ask! Trust me! There’s no harm. In-fact, people of open-source communities appreciate it, feels motivated when there are users asking them about something that they’re passionately building. It also sometimes, helps the community to re-define and re-align the product and some features.

Happy learning! 🙂

From Jenkins X – Outreachy: Motivation to apply!

By Blog, Project

Originally posted on the Jenkins X blog by Neha Gupta

Neha Gupta is adding support for Kustomize in Jenkins X, to enable Kubernetes native configuration management, while participating in Outreachy from December 2019 to March 2020.

Motivation to apply to Outreachy:

In my graduation class of fifty people we were three girls struggling to set up our space and comfort with the weird reactions we got from fellow students for trying to understand technology. When my professor asked us to make an autonomous drone. I couldn’t make one, I was shattered, until a friend from computer science batch helped me make one. He showed me some of the cool apps he made, that sparked an interest, and I started building apps, realising that computer science is beyond just coding, it’s more about solving real life problems.

I’ve transitioned from mobile to web apps, server-end development, robotics, cloud architecting, and also cofounding a startup. I’ve been focusing on using AI to make smarter apps, and help students think beyond, and see the bigger picture. I’m hoping to start an accelerator, regulating the perception about technology, focusing especially on hidden potential behind fear-stricken girls.

When I heard about Outreachy program and I liked how women and other minority communities are being supported and motivated. It felt something similar to what I’m trying to do with the young girls around me (breaking the stereotypical phenomenon of “girls can’t code”). I felt participating in Outreachy will not only boost my but other girls motivation too! and it’ll also definitely help me grow technologically, socially and mentally.

Why excited about Outreachy?

To me it feels really cool to work with a team remotely. The interactions, networking and feel is completely different, especially when it’s open-source (Like ..I get anxious before asking questions on public channels, if the question is too logical and stupid). Also, I’m a fan of open-source contributions, so… (here was the chance).

Another reason was to interact with the minority community (people who are facing similar issues in STEM like me) and share some instances with them, be on the same page, enlighten and get enlightened (all that networking sounds fun..). I am also excited about the trip. Why Jenkins-X?

The Cloud Storage backed Helm repository idea seemed interesting, so I started exploring it. The project was also quite different from other listed (maybe because it was meant for me 😀 ), I only contributed to it and focused on it.

Also the community was very welcoming and communications with my mentor were good. He helped me making contributions to the project, he guided me to some good first issues, helped me correct my PR’s. Jenkins-X looked as an interesting open-source project so I’m glad I tried to be a part of it and got selected.

What would I tell someone who is worried about applying to Outreachy?

If you are someone who has just started open source contributions and are fear stricken on how is it gonna work? These all things seems so confusing and you’re overwhelmed.

Don’t worry! ..

I was too! Every one is.. and this is just step — 1. Anyone who’s going to pick up a new project which is production ready and thousands/millions of people are using it, is going to be confused! This is normal and natural (the initial learning curve), but once you overcome it. Things become so normal and understanding, people are here to help you out.

If you think you can’t make it because of the competition, how does matter? If not this time, next time (I myself got selected after 3 years of trying), you anyway has to start one day, so let it be today. But the learnings you take away from the process, are insanely valuable and every-time it’s gonna become easier.. Feel free to reach me if you are facing any issues regarding starting with open-source contributions or if you have question saying — shall I apply to Outreachy this time?

Good Luck! 🙂

From Jenkins – WebSocket

By Blog, Project

Originally posted on the Jenkins blog by Jesse Glick

I am happy to report that JEP-222 has landed in Jenkins weeklies, starting in 2.217. This improvement brings experimental WebSocket support to Jenkins, available when connecting inbound agents or when running the CLI. The WebSocket protocol allows bidirectional, streaming communication over an HTTP(S) port.

While many users of Jenkins could benefit, implementing this system was particularly important for CloudBees because of how CloudBees Core on modern cloud platforms (i.e., running on Kubernetes) configures networking. When an administrator wishes to connect an inbound (formerly known as “JNLP”) external agent to a Jenkins master, such as a Windows virtual machine running outside the cluster and using the agent service wrapper, until now the only option was to use a special TCP port. This port needed to be opened to external traffic using low-level network configuration. For example, users of the nginx ingress controller would need to proxy a separate external port for each Jenkins service in the cluster. The instructions to do this are complex and hard to troubleshoot.

Using WebSocket, inbound agents can now be connected much more simply when a reverse proxy is present: if the HTTP(S) port is already serving traffic, most proxies will allow WebSocket connections with no additional configuration. The WebSocket mode can be enabled in agent configuration, and support for pod-based agents in the Kubernetes plugin is coming soon. You will need an agent version 4.0 or later, which is bundled with Jenkins in the usual way (Docker images with this version are coming soon).

Another part of Jenkins that was troublesome for reverse proxy users was the CLI. Besides the SSH protocol on port 22, which again was a hassle to open from the outside, the CLI already had the ability to use HTTP(S) transport. Unfortunately the trick used to implement that confused some proxies and was not very portable. Jenkins 2.217 offers a new -webSocket CLI mode which should avoid these issues; again you will need to download a new version of jenkins-cli.jar to use this mode.

The WebSocket code has been tested against a sample of Kubernetes implementations (including OpenShift), but it is likely that some bugs and limitations remain, and scalability of agents under heavy build loads has not yet been tested. Treat this feature as beta quality for now and let us know how it works!

From Jenkins – Atlassian’s new Bitbucket Server integration for Jenkins

By Blog, Project

Originally posted on the Jenkins blog by Daniel Kjellin

We know that for many of our customers Jenkins is incredibly important and its integration with Bitbucket Server is a key part of their development workflow. Unfortunately, we also know that integrating Bitbucket Server with Jenkins wasn’t always easy – it may have required multiple plugins and considerable time. That’s why earlier this year we set out to change this. We began building our own integration, and we’re proud to announce that v1.0 is out.

The new Bitbucket Server integration for Jenkins plugin, which is built and supported by Atlassian, is the easiest way to link Jenkins with Bitbucket Server. It streamlines the entire set-up process, from creating a webhook to trigger builds in Jenkins, to posting build statuses back to Bitbucket Server. It also supports smart mirroring and lets Jenkins clone from mirrors to free up valuable resources on your primary server.

Our plugin is available to install through Jenkins now. Watch this video to find out how, or read the BitBucket Server solution page to learn more about it.

Once you’ve tried it out we’d love to hear any feedback you have. To share it with us, visit https://issues.jenkins-ci.org and create an issue using the component atlassian-bitbucket-server-integration-plugin.

Screwdriver: Introducing Queue Service

By Blog, Project
Introducing Queue Service

Pritam Paul, Software Engineer, Verizon Media

We have recently made changes to the underlying Screwdriver Architecture for build processing. Previously, the executor-queue was tightly-coupled to the SD API and worked by constantly polling for messages at specific intervals. Due to this design, the queue would block API requests. Furthermore, if the API crashed, scheduled jobs might not be added to the queue, causing cascading failures.

Hence, keeping the principles of isolation-of-concerns and abstraction in mind, we designed a more resilient REST-API-based queueing system: the Queue Service. This new service reads, writes and deletes messages from the queue after processing. It also encompasses the former capability of the queue-worker and acts as a scheduler.

Authentication

The SD API and Queue Service communicate bidirectionally using signed JWT tokens sent via auth headers of each request.

Build Sequence
image
Design Document

For more details, check out our design spec.

Using Queue Service

As a cluster admin, to configure using the queue as an executor, you can deploy the queue-service as a REST API using a screwdriver.yaml and update configuration in SD API to point to the new service endpoint:

# config/default.yaml
ecosystem:
    # Externally routable URL for the User Interface
    ui: https://cd.screwdriver.cd

    # Externally routable URL for the Artifact Store
    store: https://store.screwdriver.cd

    # Badge service (needs to add a status and color)
    badges: https://img.shields.io/badge/build–.svg

    # Internally routable FQDNS of the queue service
    queue: http://sdqueuesvc.screwdriver.svc.cluster.local

executor:
    plugin: queue
    queue: “

For more configuration options, see the queue-service documentation.

Compatibility List

In order to use the new workflow features, you will need these minimum versions:

  • UI – v1.0.502
  • API – v0.5.887
  • Launcher – v6.0.56
  • Queue-Service – v1.0.11
Contributors

Thanks to the following contributors for making this feature possible:

Questions and Suggestions

We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.

Screwdriver : Recent Enhancements and Bug Fixes

By Blog, Project

Recent Enhancements and Bug Fixes

Screwdriver Team from Verizon Media

UI

Previously, users could not start builds during a freeze window unless they made changes to the freeze window setting in the screwdriver.yaml configuration. Now, you can start a build by entering a reason in the confirmation modal. This can be useful for users needing to push out an urgent patch or hotfix during a freeze window.

image
image

Store

  • Feature: Build cache now supports local disk-based cache in addition to S3 cache.

Queue Worker

  • Bugfix: Periodic build timeout check
  • Enhancement: Prevent re-enqueue of builds from same event.

Compatibility List

In order to have these improvements, you will need these minimum versions:

  • UI – v1.0.479
  • API – v0.5.835
  • Store – v3.10.3
  • Launcher – v6.0.42
  • Queue-Worker – v2.9.0

Contributors

Thanks to the following contributors for making this feature possible:

Questions and Suggestions

We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.

Screwdriver: Improvements and Fixes

By Blog, Project

Part 2 from the Screwdriver Team at Verizon Media

UI
  • Enhancement: Upgrade to node.js v12.
  • Enhancement: Users can now link to custom test & coverage URL via metadata.
  • Enhancement: Reduce number of API calls to fetch active build logs.
  • Enhancement: Display proper title for Commands and Templates pages.
  • Bug fix: Hide “My Pipelines” from Add to collection dialogue.
  • Enhancement: Display usage stats for a template.
image
API
Store
Compatibility List

In order to have these improvements, you will need these minimum versions:

  • UI – v1.0.491
  • API – v0.5.851
  • Store – v3.10.5
Contributors

Thanks to the following contributors for making this feature possible:

Questions and Suggestions

We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.