Yesterday the New York Times ran a story stating that TeamCity, a CI/CD tool from JetBrains, was implicated in the US hacking of 10 federal agencies that in turn affected 250+ organization networks.
“The exact software that investigators are examining is a JetBrains product called TeamCity, which allows developers to test and exchange software code before its release. By compromising TeamCity, or exploiting gaps in how customers use the tool, cybersecurity experts say the Russian hackers could have inconspicuously planted back doors in an untold number of JetBrains’ clients. Because TeamCity is so widely deployed, experts said, it is imperative to determine whether its software contains a vulnerability, or if attackers exploited TeamCity customers via stolen passwords or gaps in unpatched, outdated software.”
JetBrains the company behind TeamCity provided a response to correctly distance themselves from the attack and follow up by commenting on TeamCity’s potential role in the attack:
“It’s important to stress that TeamCity is a complex product that requires proper configuration. If TeamCity has somehow been used in this process, it could very well be due to misconfiguration, and not a specific vulnerability.”
It remains to be seen the role TeamCity, a closed source CI/CD tool, plays in this intrusion and if so what the nature of the exploitation was. For now, the question we need to ask ourselves is how conceivable is it that a CI/CD tool could be implicated in such a hack?
Misconfigured or Badly Configured Software
Jetbrains point to misconfiguration as one possibility. Configuration of a system can leave it vulnerable, at the most simplistic level it is like creating the login account to be ‘admin’ with password ‘admin’. We sometimes talk about CI/CD systems being sharp knives which if not handled properly can cut the user. There is a reasonable amount of knowledge and security awareness needed to set up tools in such a way that they are secure. And let’s be clear it is not just CI/CD tools, plenty of software is vulnerable to being badly configured and insecure such as Amazon S3 buckets.
Out of Date/Unpatched Software
Another issue plaguing software – all software, not just CI/CD tools – is that even after vulnerabilities are found and patched users may still be running out-of-date and vulnerable options. This was the case in 2018 when hackers exploited unpatched Jenkins servers to make $3 million in cryptocurrency mining. In this example, the crypto-operator exploited a vulnerability in the Jenkins Java deserialization implementation. In 2019, TeamCity also had a vulnerability patched where insecure Java deserialization could potentially allow remote code execution. Any versions of TeamCity in use without the fix would be vulnerable to exploitation.
Undisclosed Vulnerabilities
It is entirely possible for software to have recent or undisclosed vulnerabilities without fixes that could potentially be exploited in devastating ways. Back in 2018, I spoke about how there is increased awareness in the industry around security vulnerabilities, the consequences, and the need to do better. This is a good thing, given how pervasive and complex modern software is. Below I outline how the industry can be more proactive in discovering vulnerabilities before hackers do.
Security is everyone’s job: so what can we do about it?
CI/CD tools are a threat vector and as recent events show, as an industry we need to get drastically better at software security. The Continuous Delivery Foundation(CDF) is home to the most popular CI/CD projects such as Jenkins, Spinnaker Tekton and Jenkins X. CDF is committed to driving the industry forward on better security practices and taking action to ensure our software projects are committed to continuous security improvements.
CDF mature open-source projects are expected to ‘graduate’ by (amongst other criteria) auditing their security processes through the CII Best Practice badge and have the project report on their:
- Secure development knowledge
- Use of good cryptographic practices
- Secured delivery against man-in-the-middle (MITM) attacks
- Publicly known vulnerabilities fixed
- Use of good private credential practices (must not leak credentials)
In 2020, the Jenkins project went through the graduation process, improved its security processes and achieved a 100% badge certification.
At CDF we acknowledge that security in Continuous Delivery tools is of paramount importance and we are constantly looking to find ways to drive improvement in this space. We offer our open-source projects the ability to have a security audit to help detect vulnerabilities before hackers do. In 2021 the Tekton project will be the first CDF project to do this – as another way to give CI/CD tool adopters increased confidence.
As an open-source foundation, we are in a unique position to bring together developers, enterprises, end-users companies, startups and everyone really, to work together as a community in an open and transparent way. Together we can highlight the measures being taken and drive for increased security. Our Technical Oversight Committee chair Dan Lorenc of Tekton & Google is also leading us towards more trusted open source, most recently taking an in-depth, hands-on look at secure software supply chains: Who’s At the Helm?
With software playing a very important role in every organization, exploitation of CI/CD tools is indeed the “holy grail of a supply chain hack” – it is imperative we all work together to prevent such attacks from happening.
Resources:
- Join CDF to support our continuous security efforts
- CDF Security Newsletter
- Who’s At the Helm – Dan Lorenc
- Core Infrastructure Best Practices
Photo by James Sutton on Unsplash
