Screwdriver: Introducing Queue Service

By April 9, 2020Blog
Introducing Queue Service

Pritam Paul, Software Engineer, Verizon Media

We have recently made changes to the underlying Screwdriver Architecture for build processing. Previously, the executor-queue was tightly-coupled to the SD API and worked by constantly polling for messages at specific intervals. Due to this design, the queue would block API requests. Furthermore, if the API crashed, scheduled jobs might not be added to the queue, causing cascading failures.

Hence, keeping the principles of isolation-of-concerns and abstraction in mind, we designed a more resilient REST-API-based queueing system: the Queue Service. This new service reads, writes and deletes messages from the queue after processing. It also encompasses the former capability of the queue-worker and acts as a scheduler.

Authentication

The SD API and Queue Service communicate bidirectionally using signed JWT tokens sent via auth headers of each request.

Build Sequence
image
Design Document

For more details, check out our design spec.

Using Queue Service

As a cluster admin, to configure using the queue as an executor, you can deploy the queue-service as a REST API using a screwdriver.yaml and update configuration in SD API to point to the new service endpoint:

# config/default.yaml
ecosystem:
    # Externally routable URL for the User Interface
    ui: https://cd.screwdriver.cd

    # Externally routable URL for the Artifact Store
    store: https://store.screwdriver.cd

    # Badge service (needs to add a status and color)
    badges: https://img.shields.io/badge/build–.svg

    # Internally routable FQDNS of the queue service
    queue: http://sdqueuesvc.screwdriver.svc.cluster.local

executor:
    plugin: queue
    queue: “

For more configuration options, see the queue-service documentation.

Compatibility List

In order to use the new workflow features, you will need these minimum versions:

  • UI – v1.0.502
  • API – v0.5.887
  • Launcher – v6.0.56
  • Queue-Service – v1.0.11
Contributors

Thanks to the following contributors for making this feature possible:

Questions and Suggestions

We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.