CD Foundation

Contributed by Priya Wadhwa, Software Engineer @ Google

We’re excited to announce Tekton Chains v0.3.0! The latest release comes with new features like:

• Signing with AWS KMS and Azure KMS
• Binary transparency log support with Rekor 
• Keyless signing with Fulcio on GKE
• In-toto provenance, with a custom Tekton Chains predicate format
• Support for AWS DynamoDB as a storage backend

So, what’s the meta-chain?  Basically, this is the first Chains release that has been signed by Chains itself. The release comes with the v0.3.0 controller image, which was signed during the release process with a key in GCP KMS. Provenance for the build was generated using our custom predicate format, and stored in the Rekor transparency log.

All of this means that you can now verify the Chains controller image is legitimate before deploying it to your cluster, and you can find build provenance for the image in the Rekor transparency log.

Let’s run through a quick demo to see this in action! Since the controller image is reproducible, we’ll run a TaskRun to rebuild the image ourselves and generate the expected digest for the image. We can use this generated digest to query the transparency log and find provenance for the official image. Since the digests should be the same, this should just work.

To rebuild the controller image, you’ll need a cluster with Tekton installed and the following tools installed locally:

• kubectl
• cosign
• rekor-cli
• jq

To rebuild the image yourself, you can run this TaskRun (make sure you remember to update the KO_DOCKER_REPO variable to a registry you have push access to, and that you set up the default service account with the correct authentication). 

Once the TaskRun has completed, you can get the digest of the built image by running:

$ kubectl get tr <TASKRUN_NAME> -o json | jq -r .status.taskResults[0].value 
sha256:1189a2207be3e93e91aaeff323dc2804576f188527afe3cc2e9a9a0c688344df

Now that we have the expected digest for the official image, we should be able to search the transparency log with the digest and find provenance for the official image.

$ rekor-cli search –sha sha256:1189a2207be3e93e91aaeff323dc2804576f188527afe3cc2e9a9a0c688344df
Found matching entries (listed by UUID):
60a1e4f9c78ae76b2b2a06745340b8ed74c8b2ea2c124b8520ba319d03957906
3873a54462deab6320d1cac993b31b36bb28ff5c2f0d16993909b61907235ec6

$ rekor-cli get --uuid 3873a54462deab6320d1cac993b31b36bb28ff5c2f0d16993909b61907235ec6 --format json | jq -r .Attestation | base64 --decode | jq

{
  "_type": "publish-chains-release",
  "predicateType": "https://tekton.dev/chains/provenance",
  "subject": [
    {
      "name": "gcr.io/tekton-releases/github.com/tektoncd/chains/cmd/controller",
      "digest": {
        "sha256": "1189a2207be3e93e91aaeff323dc2804576f188527afe3cc2e9a9a0c688344df"
      }
    }
  ],
  "predicate": {
    "invocation": {
      "parameters": [
        "package={string github.com/tektoncd/chains []}",
        "versionTag={string v0.3.0 []}",
        "imageRegistry={string gcr.io []}",
        "imageRegistryPath={string tekton-releases []}",
        "releaseAsLatest={string true []}",
        "platforms={string linux/amd64,linux/arm64 []}",
        "serviceAccountPath={string release.json []}",
        "package=github.com/tektoncd/chains",
        "images=controller",
        "imageRegistry=gcr.io",
        "imageRegistryRegions=us eu asia",
        "releaseAsLatest=true",
        "platforms=linux/amd64,linux/arm64,linux/s390x,linux/ppc64le"
      ],
      "recipe_uri": "task://publish-chains-release",
      "event_id": "c26f0ca6-1a00-4313-ac6b-ee098ad859da",
      "builder.id": "tekton-chains"
    },
    "recipe": {
      "steps": [
        {
          "entryPoint": "#!/busybox/sh\nset -ex\n\n# Login to the container registry\nDOCKER_CONFIG=$(cat ${CONTAINER_REGISTY_CREDENTIALS} | \\\n  crane auth login -u _json_key --password-stdin $(params.imageRegistry) 2>&1 | \\\n  sed 's,^.*logged in via \\(.*\\)$,\\1,g')\n\n# Auth with account credentials for all regions.\nfor region in ${REGIONS}\ndo\n  HOSTNAME=${region}.$(params.imageRegistry)\n  cat ${CONTAINER_REGISTY_CREDENTIALS} | crane auth login -u _json_key --password-stdin ${HOSTNAME}\ndone\ncp ${DOCKER_CONFIG} /workspace/docker-config.json\n",
          "arguments": null,
          "environment": {
            "container": "container-registy-auth",
            "image": "docker-pullable://gcr.io/go-containerregistry/crane@sha256:3095b8be43318d89e593a7d067430b79070cd9da32b4c9484438cef117a31a98"
          },
          "annotations": null
        },
        {
          "entryPoint": "#!/usr/bin/env sh\nset -ex\n\n# Setup docker-auth\nDOCKER_CONFIG=~/.docker\nmkdir -p ${DOCKER_CONFIG}\ncp /workspace/docker-config.json ${DOCKER_CONFIG}/config.json\n\n# Change to directory with our .ko.yaml\ncd ${PROJECT_ROOT}\n\n# For each cmd/* directory, include a full gzipped tar of all source in\n# vendor/. This is overkill. Some deps' licenses require the source to be\n# included in the container image when they're used as a dependency.\n# Rather than trying to determine which deps have this requirement (an(params.imageRegistryd\n# probably get it wrong), we'll just targz up the whole vendor tree and\n# include it. As of 9/20/2019, this amounts to about 11MB of additional\n# data in each image.\nTMPDIR=$(mktemp -d)\ntar cfz ${TMPDIR}/source.tar.gz vendor/\nfor d in cmd/*; do\n  if [ -d ${d}/kodata/ ]; then\n    ln -s ${TMPDIR}/source.tar.gz ${d}/kodata/\n  fi\ndone\n\n# Rewrite \"devel\" to params.versionTag\nsed -i -e 's/\\(chains.tekton.dev\\/release\\): \"devel\"/\\1: \"$(params.versionTag)\"/g' -e 's/\\(app.kubernetes.io\\/version\\): \"devel\"/\\1: \"$(params.versionTag)\"/g' -e 's/\\(version\\): \"devel\"/\\1: \"$(params.versionTag)\"/g' ${PROJECT_ROOT}/config/*.yaml\n\n# Publish images and create release.yaml\nmkdir -p $OUTPUT_RELEASE_DIR\n\nko resolve --platform=$(params.platforms) --preserve-import-paths -t $(params.versionTag) -f ${PROJECT_ROOT}/config/ > $OUTPUT_RELEASE_DIR/release.yaml\n\n# Publish images and create release.notags.yaml\n# This is useful if your container runtime doesn't support the `image-reference:tag@digest` notation\n# This is currently the case for `cri-o` (and most likely others)\nko resolve --platform=$(params.platforms) --preserve-import-paths -t $(params.versionTag) -f ${PROJECT_ROOT}/config/ > $OUTPUT_RELEASE_DIR/release.notags.yaml\n",
          "arguments": null,
          "environment": {
            "container": "run-ko",
            "image": "docker-pullable://gcr.io/tekton-releases/dogfooding/ko@sha256:ff918ec2c8bbe416d5a9b6f9d25dfe9012dce673922fe7b2d5d69a99b02df0ac"
          },
          "annotations": null
        },
        {
          "entryPoint": "set -ex\n\nIMAGES_PATH=${CONTAINER_REGISTRY}/$(params.package)\n\nfor cmd in $(params.images)\ndo\n  IMAGES=\"${IMAGES} ${IMAGES_PATH}/cmd/${cmd}:$(params.versionTag)\"\ndone\n\n# Parse the built images from the release.yaml generated by ko\nkoparse \\\n  --path $OUTPUT_RELEASE_DIR/release.yaml \\\n  --base ${IMAGES_PATH} --images ${IMAGES} > /workspace/built_images\n",
          "arguments": null,
          "environment": {
            "container": "koparse",
            "image": "docker-pullable://gcr.io/tekton-releases/dogfooding/koparse@sha256:5945f709f5533347e2fac2f7e757a2acde2ce25418a7193489bf49027aa0497f"
          },
          "annotations": null
        },
        {
          "entryPoint": "#!/busybox/sh\nset -ex\n\n# Setup docker-auth\nDOCKER_CONFIG=~/.docker\nmkdir -p ${DOCKER_CONFIG}\ncp /workspace/docker-config.json ${DOCKER_CONFIG}/config.json\n\nREGIONS=\"us eu asia\"\n\n# Tag the images and put them in all the regions\nfor IMAGE in $(cat /workspace/built_images)\ndo\n  IMAGE_WITHOUT_SHA=${IMAGE%%@*}\n  IMAGE_WITHOUT_SHA_AND_TAG=${IMAGE_WITHOUT_SHA%%:*}\n  IMAGE_WITH_SHA=${IMAGE_WITHOUT_SHA_AND_TAG}@${IMAGE##*@}\n\n  echo $IMAGE_WITH_SHA, >> $(results.IMAGES.path)\n\n  if [[ \"$(params.releaseAsLatest)\" == \"true\" ]]\n  then\n    crane cp ${IMAGE_WITH_SHA} ${IMAGE_WITHOUT_SHA_AND_TAG}:latest\n  fi\n\n  for REGION in ${REGIONS}\n  do\n    if [[ \"$(params.releaseAsLatest)\" == \"true\" ]]\n    then\n      for TAG in \"latest\" $(params.versionTag)\n      do\n        crane cp ${IMAGE_WITH_SHA} ${REGION}.${IMAGE_WITHOUT_SHA_AND_TAG}:$TAG\n      done\n    else\n      TAG=\"$(params.versionTag)\"\n      crane cp ${IMAGE_WITH_SHA} ${REGION}.${IMAGE_WITHOUT_SHA_AND_TAG}:$TAG\n      echo ${REGION}.$IMAGE_WITH_SHA, >> $(results.IMAGES.path)\n    fi\n  done\ndone\n",
          "arguments": null,
          "environment": {
            "container": "tag-images",
            "image": "docker-pullable://gcr.io/go-containerregistry/crane@sha256:3095b8be43318d89e593a7d067430b79070cd9da32b4c9484438cef117a31a98"
          },
          "annotations": null
        }
      ]
    },
    "metadata": {
      "buildStartedOn": "2021-07-28T15:23:18Z",
      "buildFinishedOn": "2021-07-28T15:29:12Z",
      "reproducible": false
    }
  }
}

Provenance for the build image is comprehensive and uses the custom Chains format, which was designed specifically for generating provenance in container based systems like Tekton. Key sections include:

• subject: the artifact that was built (in this case, it’s the Chains controller image)
• invocation: describes the event that kicked off the build, and any parameters that were set 
• materials: defines the inputs to the build system (e.g. Git repos, pinned by commit)
• recipe: the set of steps executed to build the subject — each step in the recipe corresponds to a step in the TaskRun. It includes the image that was run, environment variables set, and any scripts that were passed in

With all of this information, you can see step-by-step how the final image was produced. In the future if you need this record again for auditing or querying, it can be easily found in the transparency log.

Last but not least, we can use cosign and the Tekton public key to verify the image itself!

$ cosign verify -key https://raw.githubusercontent.com/tektoncd/chains/main/tekton.pub gcr.io/tekton-releases/github.com/tektoncd/chains/cmd/controller:v0.3.0

Verification for gcr.io/tekton-releases/github.com/tektoncd/chains/cmd/controller:v0.3.0 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
  - Any certificates were verified against the Fulcio roots.
{"Critical":{"Identity":{"docker-reference":"gcr.io/tekton-releases/github.com/tektoncd/chains/cmd/controller"},"Image":{"Docker-manifest-digest":"sha256:1189a2207be3e93e91aaeff323dc2804576f188527afe3cc2e9a9a0c688344df"},"Type":"Tekton container signature"},"Optional":{}}

Congratulations! You’ve now built the Chains controller image for yourself, verified the digest, found the build provenance for the official image, and verified the signature of the official image. And with Chains v0.3.0, you can implement any of these for your own images.

So what’s up next? We’re planning an integration between Tekton Chains, Tekton and SPIRE to create a zero trust supply chain on Kubernetes. We’re also working on adding support for signing other types of software. All of this work will culminate in the first Beta release for Tekton Chains, which is coming soon. If you’d like to contribute to the next release, please reach out on the #chains Slack channel!