How Avast Scaled its Continuous Delivery with Spinnaker

CHALLENGE

Testing of pull requests before merging
Configuration consistency and rollback after incidents
Meeting stringent SLAs in production

SOLUTION

Tested and deployed Spinnaker for one product line
Mobile app that provides advanced parental controls
Facilitated the next phase of growth by saving time
Spinnaker to deploy EC2 instances

IMPACT

Spinnaker significantly reduces the amount of work
Cut down the time and work needed for the versioning of ConfigMaps
Rollback and no- downtime features
Spinnaker promises automated canary analysis

Avast maintains the world’s largest global security network for protecting consumers and small to medium-sized businesses (SMBs). The company’s suite of protection and pr products secures the devices of 435 million people in 49 countries and neutralizes 1.5 billion malware attacks each month. Avast does all this with just 1,700 employees worldwide so for its engineering and operations teams, how they resource matters: It has the potential to affect the safety of one in 10 internet users.

The Spinnaker team spoke with David Byron, Staff Software Engineer at Avast, about why his team selected and deployed Spinnaker.

Stats

Spinnaker footprint:

1 product family
17 Kubernetes clusters, 650 pods, and 90 nodes
Approximately 70 pipelines for each environment

Challenge

The Avast team wants to move towards continuous delivery in a multi-cloud environment. At their scale and given their security needs, this process presents challenges:Testing of pull requests before merging

Testing of pull requests before merging
Configuration consistency and rollback after incidents
Meeting stringent SLAs in production

“The whole goal here is to not wake people up in the middle of the night.”
– David Byron, Avast

Solution

The team tested and deployed Spinnaker for one product line—a mobile app that provides advanced parental controls and location features for Android and iPhone—as they grew into more AWS functionality. Spinnaker didn’t so much replace anything as it facilitated the next phase of growth by saving time. It allowed engineers to build pipelines to automate launches, testing, and rollbacks.

Avast’s code runs on AWS and they use Spinnaker to deploy EC2 instances and auto-scaling groups, as well as their Kubernetes workloads.

Impact

Spinnaker significantly reduces the amount of work needed to deploy in a multi-cloud environment. “Spinnaker saves us from creating a gigantic forest of Jenkins jobs,” says David.

Spinnaker helped the team cut down the time and work needed for the versioning of ConfigMaps. “That’s a pain to do other ways. Maybe it seems simple, but it’s a big deal to our team,” says David. “The whole goal here is to not wake people up in the middle of the night. The rollback and no-downtime deployment features are things everybody wants to do.”

“The thing that Spinnaker promises is automat-ed canary analysis. That to me is the holy grail to enabling continuous deployment.”
– David Byron, Avast

The Whole Story

Avast found Spinnaker when an employee from Netflix joined their team. When the team ran into tooling problems, he already knew there was an open source fix. (Spinnaker was open-sourced by Netflix in 2015.)

“We used it right from the beginning,” says David. “In fact, we still have a deployment out there running that’s pre-Halyard—a non-Kubernetes way of running Spinnaker. Now we’re running our deployments with Halyard in a Kubernetes cluster and it’s a lot nicer that way.”

Avast has many products but is only using Spinnaker for one product family so far because that’s the one they were scaling when the need arose. “We do have some on-prem data centers using Kubernetes and I’ll be looking to use Spinnaker there,” says David. “It’s really not a hard sell. Once we have automated canary analysis, it’s going to sell itself. Everybody wants that.”

“Once we have automated canary analysis, it’s going to sell itself. Everybody wants that.”
– David Byron, Avast

David’s team faced the challenge of learning Kubernetes and Spinnaker simultaneously. “When we migrated to the Kubernetes V2 provider, that’s when the engineering organization got into Spinnaker, started using it, and started making and running pipelines.” There was a steep learning curve, but for David, “We’re getting there. It’s just trying to teach people to know what’s going on so they have a chance to figure it out themselves.”

Counterintuitively, David feels that he could have made adopting Spinnaker easier on his team by requiring them to learn more on their own. Early on, he wrote a tool for his engineers to generate pipeline JSON because the SPIN CLI was new and not fully mature. According to David, “I made a company-specific, opinionated tool and custom DSLs that were supposed to make people’s lives easier but in the end, it covered up some details of Spinnaker that I wish I had asked people to learn.”

Infrastructure teams also use Spinnaker to manage cluster-wide Kubernetes resources, often exploring parts of Spinnaker and passing on that knowledge to more product-focused teams.

David and his team have become active members in the Spinnaker community, participating in the 8,000-person strong (and growing!) Spinnaker Slack channel, asking and answering questions, and helping to surface issues. They have also begun to contribute to the OSS code base, and David is a member of the Kubernetes SIG.

What’s next

“We’re headed to automated canary analysis,” says David. His team uses Datadog as their metrics aggregator, and they intend to feed that data into Spinnaker for automated canary analysis.

If you’re looking to learn more about the potential Spinnaker has to transform your business’ approach to app or microservice deployment, be sure to explore www.spinnaker.io to connect with our rich knowledge base of how-to guides and success stories.

Website: https://cd.foundation/ | Slack: join.cdeliveryfdn.slack.com | Blog: https://cd.foundation/news/blog/