Data teams ship critical workloads, but Spark jobs often live outside the DevOps/CI/CD guardrails. This session shows how to bring Continuous Delivery discipline to Apache Spark on container-orchestrated platforms (with Kubernetes as a concrete example). We’ll cover how to package Spark apps as immutable artifacts, add automated quality gates (code, dependency, and data tests), and promote jobs through environments using pipeline-as-code.
We will discuss the end-to-end flow: commit → CI build → artifact + test → CD submit to a local container orchestrator → run/observe/roll back. We’ll close with a production checklist for platform teams (multi-tenant quotas, secrets, cost controls, and supply-chain security) and share a template repo you can adapt. If you’re a DevOps, platform, or data engineer looking to make Spark delivery as robust as your apps and services, this is your fast on-ramp.
Key Takeaways:
