Security Principles for Cloud Native CI/CD Systems with Jenkins X

By July 7, 2020July 10th, 2020Blog

Contributed by: by Cosmin Cojocar

CDF Newsletter – July 2020 Article
Subscribe to the Newsletter

Jenkins X is a CI/CD platform built on Kubernetes that provides a cloud-native application delivery experience for developers. In addition, Jenkins X relies on other cloud services to create a production-ready system from the get-go. Sounds great? Let’s go! 

Hold on. 

What if someone hacks into your build server? They could replace part of your software with a virus and you’d ship that to your customers. No one wants that. That’s why this post highlights some of the security features available in Jenkins X and why you should enable them. 

Reminder: Before trying to secure any system, it’s important to adopt some security design principles that go beyond technology. You may have heard of them, but they also apply to cloud-native systems: minimize the attack surface, apply defence in depth, least privilege, establish secure defaults, fail securely, separation of duties, don’t trust services, keep security simple, and avoid security by obscurity. 

Configuration as Code and GitOps

Configuration as code is a DevOps practice that promotes storing of application configuration as code in a source-code repository. It is particularly important for security when building a complex cloud-native system. It allows you to specify the security controls of every cloud service that’s part of the platform. All changes can be easily tracked and updates can be performed consistently.

Jenkins X embraces configuration as code at all levels: from cloud services provisioning to Kubernetes Helm charts installation. Furthermore, it uses GitOps to store its configuration in git repositories and to keep this configuration in sync with the installation at all times.
The cloud resources, including the Kubernetes cluster, are provisioned with Terraform. Jenkins X provides Terraform modules with a set of predefined defaults that are validated for security issues with tfsec. Upon completion of a Terraform provisioning, a jx-requirements.yml file is created. This file contains several of features that are installed with Jenkins X along with the required cloud resources; consider it a recipe for jx boot command. This command bootstraps the Jenkins X installation and stores the current installation’s configuration in a git repository.

Any changes made to the installation should be done via this git repository. By doing so, Jenkins X will detect it and start a deployment pipeline that updates the installation. 

The following diagram shows how Jenkins X manages its configuration:

As you can see at the top of the diagram, the cloud project, where the Jenkins X resources are created, is the first place to secure. Why? If this cloud project is breached, the attacker can escalate its access further into the Kubernetes cluster where Jenkins X is running. To avoid this kind of unpleasant situation, set up restricted access control on the cloud project level and isolate this Kubernetes cluster into a dedicated cloud project when possible. Access should be monitored and audited to prevent any suspicious activities.

Afterwards, check the configuration of the Kubernetes cluster created with Terraform to make sure that your cluster implements the security controls recommended by the Kubernetes community. It’s not always easy to achieve this by simply inspecting the configuration. You can scan the cluster to find out potential security issues, adjust the configuration accordingly, and then reapply it. Jenkins X tries to facilitate this process by providing a command jx scan cluster. This command runs underneath the kube-hunter tool from within the cluster and collects the results into a YAML file. 

Now that we made sure that the cloud environment where Jenkins X is running has proper security controls in place, we can look at some of the security features that can be enabled in Jenkins X.

Automate DNS and TLS provisioning

Having TLS enabled on all publicly exposed endpoints is a security prerequisite for any modern cloud application; Jenkins X is no exception. To enable TLS automatically from the beginning, Jenkins X integrates with open-source projects such as cert-manager and external-dns. These components allow you to fully automate this process during the installation by defining a few configuration parameters in the jx-requirements.yml file:

ingress:
  cloud_dns_secret_name: external-dns-gcp-sa
  domain: cluster-domain.jenkins-x.example
  externalDNS: true
  namespaceSubDomain: -jx.
  tls:
    email: my-email@jenkins-x.example
    enabled: true
    production: true

Jenkins X detects when DNS and TLS are enabled. It installs and configures the required components to ensure that all exposed endpoints are TLS ready at the end of the installation.

Secrets management

Separating the secrets from the configuration is a no brainer in an environment where all the configuration is managed as code and using GitOps. Store your secrets in a secrets management solution instead of hardcoding sensitive information in plain text. 
Jenkins X integrates with HashiCorp Vault. All the secrets required by Jenkins X are stored in a vault that’s provisioned during the installation. The vault can be enabled by adding the following configuration value in the jx-requirements.yml file:

secretStorage: vault

Jenkins X also makes sure that a vault is ready in the Kubernetes cluster before starting its installation as shown in the following diagram:

This vault is configured to use the cloud services for storage and data encryption. Kubernetes authentication is also enabled, allowing a Jenkins X pipeline to retrieve secrets from the vault during a pipeline execution. 

If you have an existing Vault instance and you want Jenkins X to store its secrets there, simply enable Kubernetes authentication and specify the following configuration values in the jx-requirements.yml file:

secretStorage: vault
vault:
  kubernetesAuthPath: "secret"
  secretEngineMountPoint: "kubernetes"
  serviceAccount: my-sa
  url: https://my-vault.com

The secrets from vault can be referenced only by their path in the Jenkins X configuration as shown in the following example:

adminUser:
  password: vault:mycluster/adminUser:password
  username: admin
pipelineUser:
  token: vault:mycluster/pipelineUser:token
  username: pipeline-user

Securing the Git integration in a GitOps environment

The integration with the git provider needs to be properly secured, especially in a GitOps environment where an unauthorized git event can trigger an undesired action inside of Jenkins X. The interaction with the git provider is bidirectional as described in the following diagram:

Jenkins X exposes a webhook-handler to receive events from the git provider whenever an action, like the creation of a pull request, takes place. This endpoint has TLS enabled and expects an HMAC token for authentication. This token is shared with the git provider to prevent unauthenticated events from entering the system. Jenkins X also requires git and API access to git providers to clone repositories or perform various API calls. The permissions for this access is reduced to a minimum and depends on the git provider.
Sometimes in a large project with many collaborators, you need to be able to control who can trigger a pipeline automatically when a pull request is created or who can approve that pull request. Jenkins X allows you to define a list of users, exactly for this purpose, in an OWNERS file stored in the git repository which looks as follows:

approvers:
- user1
reviewers:
- user1
- user2

Enabling all these features in Jenkins X is an important first step to ensure that your cloud-native applications are securely delivered into Kubernetes without being surprised by hackers. You can learn more about them and other features available from Jenkins X Documentation.