Contributed by Tracy Ragan, CEO DeployHub, Inc.
We as a collective IT community like to create new terms. And when we do, we spark new interest in old concepts. This is certainly the case for the new SCM. At the heart of the supply chain management discussion is the core of Software Configuration Management – the old SCM. Software configuration management is the process of controlling and versioning changes in software. The supply chain discussion expands our historical SCM practices to also explore both internal and external artifacts that we include in the software we develop. In other words, it extends to tracking all ‘raw materials’ that are used to create software.
Historically, Supply Chain Management is a ‘commerce’ term that refers to tracking the logistics of goods and services moving between producer and consumer. This would include the storage and flow of raw materials with a focus on the channels that support the process.
There is no doubt that the SolarWinds hack taught us to take a closer look at how we consume our ‘raw materials,’ which are the external and internal artifacts that go into our software build and package process. While the SolarWinds hack was a hard lesson to learn, it was bound to happen. In general, this discussion is years overdue.
According to Fireeye, the firm who discovered the ‘backdoor’ Supply Chain hack:
“SolarWinds.Orion.Core.BusinessLayer.dll is a SolarWinds digitally-signed component of the Orion software framework that contains a backdoor that communicates via HTTP to third-party servers. We are tracking the trojanized version of this SolarWinds Orion plug-in as SUNBURST.”
In other words, something went very wrong with the creation of that .dll in the build. And while I am no expert in the field of cybersecurity, and I’m certainly not a hacker, I do understand the build process and have been talking about the potential of these types of breaches for over 25 years.
Let’s look at the core problem of most software compile/link processes. As we all know, most build processes are imperatively defined with an Ant, Make or Maven scripts. In 25 years of working in the build management space, I’ve never run across a scripted build process that built the binaries incrementally. The build starts at the top and ends at the bottom, ‘cleaning all’ objects along the way. This means that each build recreates all the binaries. Incremental builds are difficult to code and need more advanced logic. For this reason, most build processes rebuild the entire set of objects.
Why does this matter? It matters because if the build only recreated the new objects, and left the old ones to be reused, we could clearly see what source code and libraries were updated. If we are building frequently, that means only validating the code updates of a few binaries. But instead, all the binaries are new, so seeing what needs to be validated is a big job. With a full build every time, we lose the ability to determine what source code needs to be checked to confirm that only the approved coding updates are going into the build.
I’m not saying that this is what occurred in the SolarWinds hack, but I can see how it could have. Hackers first get past your firewall and get to your network where your corporate source code assets are stored. They find the build directory, read through the build scripts to see how the application is assembled. They then find the perfect piece of code to update with their ‘bad function.’ The next time the build runs, a ‘clean all’ recompiles everything and like magic, a SUNBURST is created.
The first company I started was called OpenMake Software. We created a declarative build process, called Meister, that ensured that incremental builds were easy. The work products of the build always included a Software Bill of Material report and a Difference report allowing teams to easily audit and compare the actual source changes to confirm that only the approved code updates were included. Meister was embraced by the fintech community who understood the importance of making sure nefarious code was never released. It has been running in some fintech organizations for over 20 years. Rational Software’s ClearCase addressed this problem in a similar way; it uses a process called ‘winkin’ to reuse ‘derived’ objects that had not changed. This process created an incremental build that could be easily audited.
As we continue along this path to improve our software supply chain, we should carefully think about how we manage our build and package process. The truth is most monolithic builds will never be re-written to support an incremental process. Culture and cost get in the way. But that does not mean we can’t solve the supply chain puzzle in a microservice application. As a DevOps professional, I’ve always known how important the ‘build and package’ steps are to the overall release process. Microservices change the way we build and package software, but it does not make the problem any less important.
My newfound passion is in the emerging microservice catalog market. A microservice catalog tracks versions of services to versions of their consuming applications. This is how the new Software Bill of Material reports and Difference reports can be created. In a microservice application, the microservice will have an SBOM, and a collection of microservice SBOMs gives us the application-level insights. The good news is that microservices can be independently built and deployed, meaning incremental. Don’t make the mistake of re-releasing all your microservices in one big Helm script just because we don’t know what changed. Always know what changed, why it changed and confirm the code is what you were expecting.
The topic of builds, packaging, SBOMs and Difference reports has never been sexy and mostly overlooked. But with the renewed interest in the new SCM – Supply Chain Management – we have an opportunity to re-evaluate our methods. Fixing old builds may not be possible, but let’s get this process right as we decompose our monolithic applications and move forward into a new frontier of cloud native software development.
And don’t forget, Trevor Rosen of SolarWinds will be the Keynote at SupplyChainSecurityCon. SupplyChainSecurityCon will be a day 0 event at KubeCon held on October 11, 2021. Attend the event and find out the details of the SolarWinds attack and how they plan to move forward to ensure it never happens again. No doubt, the best keynote at KubeCon this year.
