Contributed by: Dwayne McDaniel, GitGuardian Developer Advocate | Originally posted on GitGuardian
On the west coast of Canada, you will find Vancouver, British Columbia, home to the Canucks, breathtaking scenery, and the Granville Walk of Fame. You will also find the Vancouver Convention Center, which hosts some of the best views from any event space in the world. It was in this picturesque setting that the CD Foundation and OpenGitOps communities came together for a co-located event, cdCon + GitOpsCon 2023.
These two communities are distinct but have aligned goals and visions for how DevOps needs to evolve. The CD Foundation acts as a host and incubator for open source projects like Spinnaker and Jenkins, the newly graduated project Tekton, and the completely new CDEvents. They have a mission of defining continuous delivery best practices. OpenGitOps was started as a Cloud Native Computing Foundation (CNCF) working group with the goal of clearly defining a vendor-neutral, principle-led meaning of GitOps.
With so much overlap in community missions and membership, this event was a great success, fostering a lot of meaningful conversations. Everyone was excited to share what they saw on the horizon for DevOps. Here are just a few of the highlights from this outstanding event.
What is GitOps
Most people, by now, are familiar with the term DevOps, the movement to combine development and operations into effectively one team, leveraging certain processes and tools. The term “GitOps” might be a new one for you. In their keynote, “State of the Union: OpenGitOps,” project leads Scott Rigby and Christian Hernandez defined GitOps as having these 4 principles:
- Declarative – A system managed by GitOps must have its desired state expressed declaratively.
- Versioned and Immutable – The desired state is stored in a way that enforces immutability and versioning and retains a complete version history.
- Pulled Automatically – Software agents automatically pull the desired state declarations from the source.
- Continuously Reconciled – Software agents continuously observe the actual system state and attempt to apply the desired state.
Find out more about OpenGitOps at https://opengitops.dev
Better security through standardized logging
Over the last couple of years, we have seen a 742% increase in malicious supply chain attacks. It is no wonder that 88% of boards of directors consider cybersecurity a business risk. With application security top of mind, it is also easy to understand why most organizations, 80%, are constantly seeking better visibility into app security logs. Fortunately, we heard from Tracy Ragan from DeployHub, in her session “Learn New Ways of Tracking Security and DevOps Intelligence with Ortelius,” that there is an open source project actively working on the visibility issue.
There are multiple issues that make tracking all security logging difficult. First is the sheer volume of log data. Every single tool throughout the DevOps stack, including all security tools, produce audit logs. While there are good tools for ingesting these logs and storing them, actively watching them all is a daunting challenge.
Compounding the issue is the variety of enterprise logs. We depend on logs to show dependencies, licensing, vulnerabilities, configurations, and many more vital stats. Each tool will also have some variance in how it presents results.
Also at play is the concept of ‘drift,’ the number of different versions of the same microservice running in production across all environments. While we always intend to keep things consistent throughout our entire ecosystem, the reality is some drift will always occur.
This problem set is what Ortelius, a free and open source project from the CD Foundation, set out to solve. The project defines itself as “a unified catalog of supply chain evidence providing an end-to-end view of an organization’s security profile.” In simple terms, it helps you capture actional insights in minutes instead of days. It does this by ingesting data from any services’ logs and categorizing them into three universal object reference types. There are only three categories:
- Containers
- DB
- Files
This approach leverages data you already have, just presenting it in a more centralized, standardized way. The promised benefits include understanding package use and variance across the whole organization. Also, every time a microservice composition changes, Ortelius will be able to easily surface those changes to see if you are still in compliance. It can make it easier for you to track the impact of new CVEs throughout all your applications through a consistent view.
Trusting signatures in git is harder than you think
When you see a signed commit in GitHub, your first instinct might be to trust it. Unfortunately, as we deal with supply chain security, your first instinct should be to question if you can trust that signature. Fortunately, you are not on your own for verifying those signatures, according to Billy Lynch from Chainguard in his session “Identity-based Source Integrity with Gitsign.”
One of the growing issues in supply chain security is impersonated commits. While signing commits has been in git for years, thanks to projects like gpg, signatures have relied on long-lived key pairs. These can easily end up in places they should not go, perhaps by being hardcoded in repositories, as we have seen in the State of Secrets Sprawl year after year. If anyone gets your private key, then they can sign and push legitimate seeming commits whenever they wish.
Compounding the issue is that GitHub uses web-flow.gpg to sign all web commits: GitHub uses the same signature for all web commits (merge, revert, edit, etc) made on GitHub.com. One of the points of signing a commit is to be able to verify it independently. If someone gains access to your GitHub org, then they can use web-flow.gpg to sign their commits, no matter what malicious code they might have inserted.
To fight this, the Sigstore team is developing Sigstore Gitsign, an open source project that leverages OpenID Connect identity to issue you a temporary, ephemeral signing key pair that you can securely sign your commits. Essentially, when you are ready to sign a commit, you will open a browser window to leverage an OpenID provider, such as GitHub or Google, verifying you are really you. Then a new asymmetrical key pair is issued for ten minutes. You will then be able to sign whatever you like in that window of time. The usage, along with all the associated metadata, is stored long-term in Sigstore rekor, an immutable, tamper-resistant ledger generated within a software project’s supply chain.
There are already a lot of projects using Gitsign, including Pypi, npm, OCI, GoReleaser, and Tekton Chains. However, neither GitHub, nor GitLab recognize Sigstore signatures as valid just yet. While this is a very promising route to more secure git commits, it is still a work in progress that could benefit from additional community volunteers.
Be the hero of your project
You can have the best intentions and make the best plans, but if your ideas don’t get implemented, then they won’t benefit anyone. This was the message at the root of the session from Robert Reeves, Co-Founder & Chief Evangelist from Liquibase, “Continuous Delivery Delivers Value and So Do You: How to Get Budget, Build Your Case, and Be the Hero”
Unlike the movies, where there is typically a big, bad villain fighting the hero, there is not a single foe to defeat inside your organization. Your main opponents to overcome are:
- Ambivalence
- Apathy
- Inertia
He laid out a straightforward plan that can help you overcome these challenges and help get the executive team on your side.
It starts by thinking in terms of budget. While your idea might be really cool, you are only going to get people to move if it affects the bottom line. First, lay out what the current path costs in terms of team hours and real numbers. Next, you need to show what the 100% effectiveness rate of your plan can accomplish. After that, it is a matter of finding what people would accept as a realistic effectiveness percentage.
The resulting number, especially if arrived at with executive input, will make it much simpler to sell your project, but to be successful, you still need to paint a picture of what can be. Along with the monetary savings, you also need to show what other opportunity costs are involved, that is, what else you are currently not able to do because you are spending more time and resources on the old approach.
Robert said to start by gathering your collation of the willing, what he referred to as your ‘ride or dies;’ colleagues who are already generally on board with the idea. As you build your story, make sure you are getting feedback from devs and ops teams involved on how these changes could make their lives better in terms of time spent and what other things they could be doing. Being able to cite this info can make selling your plan a lot easier.
Growing and maturing communities
cdCon + GitOpsCon was also co-located with Open Source Summit North America, one of the many events put on by The Linux Foundation. Their aim is to democratize code and scale adoption for all projects. The communities they are helping to build and scale are truly changing how the world delivers code. It was such an honor for me to give my introduction to OWASP and introduce some folks to another open community, helping unite us all in our security journey.
