Introducing Queue Service
Pritam Paul, Software Engineer, Verizon Media
We have recently made changes to the underlying Screwdriver Architecture for build processing. Previously, the executor-queue was tightly-coupled to the SD API and worked by constantly polling for messages at specific intervals. Due to this design, the queue would block API requests. Furthermore, if the API crashed, scheduled jobs might not be added to the queue, causing cascading failures.
Hence, keeping the principles of isolation-of-concerns and abstraction in mind, we designed a more resilient REST-API-based queueing system: the Queue Service. This new service reads, writes and deletes messages from the queue after processing. It also encompasses the former capability of the queue-worker and acts as a scheduler.
Authentication
The SD API and Queue Service communicate bidirectionally using signed JWT tokens sent via auth headers of each request.
Build Sequence
Design Document
For more details, check out our design spec.
Using Queue Service
As a cluster admin, to configure using the queue as an executor, you can deploy the queue-service as a REST API using a screwdriver.yaml and update configuration in SD API to point to the new service endpoint:
# config/default.yaml
ecosystem:
# Externally routable URL for the User Interface
ui: https://cd.screwdriver.cd# Externally routable URL for the Artifact Store
store: https://store.screwdriver.cd# Badge service (needs to add a status and color)
badges: https://img.shields.io/badge/build–.svg# Internally routable FQDNS of the queue service
queue: http://sdqueuesvc.screwdriver.svc.cluster.localexecutor:
plugin: queue
queue: “
For more configuration options, see the queue-service documentation.
Compatibility List
In order to use the new workflow features, you will need these minimum versions:
- UI – v1.0.502
- API – v0.5.887
- Launcher – v6.0.56
- Queue-Service – v1.0.11
Contributors
Thanks to the following contributors for making this feature possible:
Questions and Suggestions
We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.
