Last week, Tidelift CEO and co-founder Donald Fischer and open source maintainer Jordan Harband sat down with Software Engineering Radio to discuss open source supply chain security. You can listen to the full episode by following this link—or read on for some of the highlights:
Defining the software supply chain and what it means to secure it
The meaning of the phrase “software supply chain” varies from person to person. The term can either be all encompassing or a stand in for a more specific technical area of focus. To start the conversation, Donald offered up his definition of the software supply chain:
“The software supply chain is really anything that affects your software at any point in its development and release—including the original creation, writing the software, continuous integration, continuous deployment pipeline. It goes through all of the channels through which your software flows from the moment of creation, into production; that includes a whole bunch of different systems and sources of software. In particular, one of the areas that's been in the spotlight quite a bit recently, is third party open source software, which is one of the key ingredients that goes into most applications these days.”
And what does it mean to secure something as broad as the open source software supply chain? Jordan offered his insights to help answer this:
“Each dependency is one or more people who maintain that dependency. And there are huge benefits to adding them in terms of there's more eyes on the software, there's more experience involved. You have more specialists who can do the specific task they're charged with doing exceedingly well, [but] you also have more points of failure. And I think that securing your supply chain is really a balancing act about how you accept the large benefits of adding people to your process, while still managing the weakest link in any process on the planet to humans.”
The focus on open source components in the software supply chain
When asked if open source is a major vector of supply chain attacks, Donald had this to say:
“It's a vector for software supply chain attacks, for sure. Now, open source software is still just software, it's going to suffer from a lot of the challenges that any software including privately proprietary software would face in terms of secure development, secure release, and secure handling. And in fact, in a lot of ways, open source software has some advantages over privately built software. There's inherently the many eyes principle—many eyes will find even obscure bugs. There's inherently more transparency around open source than your garden variety internal proprietary software project.”
“But I think the balance there is that because open source [...] is so widely used, the blast radius of an incident can be really vast. I mean, we're talking frequently about open source packages that are downloaded and incorporated into application builds billions of times a month. So one of those packages being compromised in a supply chain attack can have really, really far reaching implications. And I think that's why a lot of the conversation around software supply chain security has recently focused on this part of it—the open source software supply chain security challenge.”
In agreement, Jordan recounted a time when one small change to one of his packages had a bigger impact than expected.
“One time, I refactored the package and changed the text of an error message to make it better, to make it more helpful. And I had no idea that some very popular packages were using the text of that error message, like in their code paths, and I broke Angular and Ember users all over the planet by changing the text of an error message. Luckily, I was able to find it and fix it within a matter of hours. But you know, that wasn't an attack. That was just a mistake. And it still caused widespread damage.”
“So I would say that in the same way that Donald phrased it, [open source] is a vector. When anyone says something is a major vector, are you talking about the scale of its reach? Or are you talking about it having outsized risk, more than it should? And I would say that, because open source is used so widely across many different industries and companies and code bases, it is a major vector in terms of scale. Whether it's an outsized vector or not, I don't believe it to be but obviously, that part would be debatable.”
Industry efforts to improve open source software supply chain security
The technology industry has reacted (alongside the U.S. government) to the recent software supply chain attacks by working to identify best practices and standards that will improve open source software security.
Donald brought up the Open Source Security Foundation (OpenSSF), an industry collaboration where Tidelift has been one of the participating entities gathered to work on some of these efforts to improve software supply chain security. He had this to say about the OpenSSF security scorecard project:
“What security scorecards is about is essentially agreeing on what are some of these practices that we would all like our software, the creation of our software to adhere to—whether it's security of the systems that are being used to host the code or to package the code, or whether it's specific secure coding practices that are being taken. Or different kinds of preventative analysis that can be undertaken around open source code, such as fuzzing it or supplying it with random inputs to find potential avenues of vulnerability.”
“So, the security scorecards project has been great, because it's been a way to catalog or inventory, what's the wish list of what we would like all of our software to follow—in particular open source software projects—and that's an important part of the challenge—just agreeing what our desired state would be. Then I think some of the opportunity in the industry is, how can we cause that to happen? How can we get the right folks, including companies and organizations like the ones that are predominantly authoring standards (like the OpenSSL security scorecards are contributing to them), but also the independent creators? I mean, folks like Jordan, frankly, who are authoring a lot of those open source projects outside of a traditional corporate context. We need to meet in the middle there and convene these two audiences if we're going to turn it from being just a wishlist of attributes that we wish software could have, to a list of attributes that we have confidence or know that software has—because we've engaged with the actual creators.”
"We need to meet in the middle there and convene these two audiences if we're going to turn it from being just a wishlist of attributes that we wish software could have, to a list of attributes that we have confidence or know that software has—because we've engaged with the actual creators."
- Donald Fischer
Donald went on to discuss standardizing open source project certifications and error reporting:
“The OpenSSF security scorecards project does include various ways for some of these facts to be communicated in machine readable formats, but some of them require human attestation. And that attestation can be digitized, but it cannot necessarily be determined without the involvement of the creators there. The traditional approach in the industry has been very much a downstream tools based approach. It's been centered around different software tools that we can apply against either the code that's being written inside an organization or the code that we import [through a] package, the third party open source code that we import, looking for known vulnerabilities that have an identifier, et cetera. But there's only so much that a downstream tool like that can do by looking at just the code itself, without actually having a connection back to the moment of creation, the human or group of humans, who authored that code and know all the surrounding context about, how that code came to be, and what standards and processes were being followed?”
Bringing in the people behind these packages
At Tidelift we believe that working with and paying the maintainers of these open source projects is the key to improving open source software supply chain security. Donald highlighted this in response to a question on how to incorporate the human context in the broader supply chain:
“This is one of the challenges that we're working on at Tidelift—creating both an incentive mechanism that makes it clear to those independent maintainers what we would ask of them. And then rather than just placing yet another obligation on these folks who are creating this code that we rely on in our enterprise and organizational applications, we're also providing a economic incentive to go undertake this non-trivial work to cause these practices to be followed and to document how they're being followed using a combination of machine readable and human validated mechanisms. I think Jordan probably could speak to this as well.”
Jordan responded, “Without speaking to the merits of any particular system, the world is largely capitalism. It's not pure capitalism [as] we have regulation. But that means that there are really only two mechanisms to exert leverage in the world: capital and the law. And so if we want something to happen, like if we want open source software to be more secure, those are the mechanisms: spend money, put money towards it, and or regulate it, and provide incentives/penalties to make what we want happen. And the way that the tech industry has been treating open source for decades has been essentially as a source of free labor. But they keep wanting things from the open source ecosystem: we want you to make your software secure, we want you to fix bugs, we want you to release in a timely fashion, and so forth, that's where we're at. If you don't like that as a mechanism, and you don't want to lobby for regulation, then vote for a different economic system. Similarly to my earlier comments, we need to acknowledge the reality that we're in. And the reality we're in is capitalism. So spend your capital to make it happen.”
"And so if we want something to happen, like if we want open source software to be more secure, those are the mechanisms: spend money, put money towards it, and or regulate it, and provide incentives/penalties to make what we want happen."
- Jordan Harband
Pay the maintainers
Jordan continued to emphasize why paying the people behind these packages is the key to improving the health and security of these packages:
“I can speak to myself personally, as it relates to Tidelift, like Tidelift as a funding source. So I am doing, hopefully, all the best practices of software maintenance for all of my packages. And I'm doing that largely out of a sense of ethics and ideology. However, if at any point in the future, I decided those were no longer enough for me, that would suck for the entire ecosystem. That is, depending on my packages, the best way to make sure that those incentives remain, even if my mind changes, again, is capital. If I have a source of income related to it, then I'm going to think a lot harder about changing my mind on a whim about ethics and morals and whatnot. I think we can't [ignore that], again, acknowledging the reality we're in, at least in the U.S., money is required for healthcare, for raising your children, for having a home, for buying food. And these things are pretty important. So if you want to make sure that something happens, you tie those things to that thing happening. And money is the glue to tie them together.”
"I am doing, hopefully, all the best practices of software maintenance for all of my packages. And I'm doing that largely out of a sense of ethics and ideology. However, if at any point in the future, I decided those were no longer enough for me, that would suck for the entire ecosystem."
- Jordan Harband
Scanning tools and risk mitigation
Diving deeper into how to mitigate risk, Donald brought up the current, more widely-known method of software analysis tools and the role they play:
“There's a long established category of static software analysis tools that look for the structure of software programs and look for coding patterns that can be suggestive of leaving open security vulnerabilities. There's a second class of products that are commonly called the dynamic scanning tools that are essentially running different kinds of exploitation paths against whole applications or parts of applications, often following common exploit paths, like cross site scripting, or techniques like that. And then there's a third category, certainly pertaining to open source software that's had a lot of adoption, in recent years, the category of software composition analysis, the tools that help you look at what are all of the ingredients flowing into your application, and cross-checking those against lists of known vulnerabilities, oftentimes drawn from the National Vulnerability Database, these security vulnerabilities that people know from their naming convention, the ‘CVE vulnerabilities’.”
“What I'm very fascinated by is how we can add a layer of an advanced upstream, or proactive engagement with the open source creators behind our projects to agree in advance what are some of these practices that we would like to see followed and engage in the process of creating and releasing the software, before [it] even flows into our organization. I saw that recently echoed in the words of a senior bank executive who I saw speaking just last week, who said their philosophy at this global bank is the best way to deal with security vulnerabilities is to head them off in the first place, not just to have the fastest scanner, because if you're relying just on reactive scanning tools, there's always going to be a window of vulnerability, no matter how quickly you can go. But if you can head off the issue entirely using a combination of some of these tools like static analysis and dynamic analysis. But I think now importantly, agreeing on the practices that need to be followed before that software flows into your organization. That's really the best solution.”
The buzz around software bill of materials (SBOMs)
The White House cybersecurity executive order 14028 that was issued last year, and the U.S. government’s Office of Management and Budget (OMB) memorandum M-22-18 released a few months ago, both suggested that providing SBOMs will be a requirement for organizations selling software to the government. This has made SBOMs a hot—and controversial—issue. Donald explained the role SBOMs play in securing the open source security supply chain and why, by themselves, they might not be the silver bullet some are hoping for:
“The idea of a software bill of materials is basically that of an ingredients list. For your software application, it's intended to be a comprehensive list of all of the components that comprise your application, whether they be third party open source components that are coming from public software repositories, or whether they're coming from commercial vendors or internally authored. And one of the ideas of software bills of materials is that they allow you to at least know what you're using.”
“One of the things that has happened, though, with the recent popularity of software bills of materials is there's a little bit of backlash from practitioners who protest that just having a list of the components doesn't necessarily solve the problem. It's a prerequisite for being able to answer a lot of questions about your software application—what's flowing into it—but just having a list of the packages and the versions, and so on, it doesn't make that software secure, it gives you a basis for starting the investigation or the analysis of whether it's secure and on what basis.”
At Tidelift our mission is to provide a proactive approach to managing your organization’s open source software. We partner with open source maintainers to validate and improve packages to meet enterprise, industry, and government standards. Most importantly, we make sure that the maintainers we partner with get paid. Learn more about the Tidelift Subscription and finish the story on Software Engineering Radio.
50 Milk St, 16th Floor, Boston, MA 02109