What if your team could save $1 million while improving the security and resilience of an application that your company depends on to deliver revenue? I’d like to share a story about a team that did just that.
First, let me set the stage. Over the past few years, organizations have become increasingly focused on improving the security of the open source powering their applications. For good reason: known vulnerabilities like Log4Shell in 2021 have highlighted the risk to organizations’ revenue, data, and customers that can result from insecure or under-maintained open source packages.
Some industry leaders have pushed for codifying security practices for open source projects, and then measuring how well projects follow these practices. The OpenSSF Scorecard has been the most visible of these efforts. But while Scorecard scores may provide good information about the security, health, and resilience of a package when the data is accurate, it is not actually making the package more secure, healthy, or resilient. The Scorecard is simply a historic snapshot of observable activities.
Our recent 2024 Tidelift state of the open source maintainer report showed that paid maintainers implement, on average, 55% more critical security and maintenance practices than unpaid maintainers. As part of our upcoming maintainer impact report, we wanted to analyze the impact of paying open source maintainers to implement industry-leading secure software development practices.
Paid maintainers are doing more—but what difference is it making in customers’ bottom line and their ability to increase the velocity of their business?
Which brings us back to the story about our customer. We analyzed a real-world Python application that this organization uses to analyze and forecast commercial pricing in a competitive, highly regulated industry. We wanted to see how they were able to improve the application’s security and resilience over a two-year period with help from Tidelift and our open source maintainer partners.
Let’s start with the bottom line results: This customer is able to set accurate pricing, drive profitability, and improve their margins because their developers have been able to reduce the organization’s reliance on abandoned, end-of-life, or otherwise insecure open source packages that are costing them time and money. Specifically, they:
All of this return is in addition to the organization’s ability to continue to take advantage of the increased business velocity that open source provides. Recent research by Harvard Business School estimates that it would cost 3.5 times more to build the software and platforms that run their businesses without open source software. This customer can’t wait 3.5 times as long to remain competitive in their industry—they found a better way to speed up time to market, reduce internal development costs, and avoid vendor lock-in while still taking full advantage of open source.
In 2021, before the customer began to track this application with Tidelift, they had limited visibility into the health, security, and resilience of their open source dependencies. At best, the customer would have visibility into CVEs affecting package releases in the application, alongside information from the OpenSSF Scorecard about package maintenance and discoverable security policies.
After starting to work with Tidelift, their visibility into the health, security, and resilience of their open source software supply chain increased radically. On day one, the customer could answer questions like:
This customer immediately had much more visibility into which packages were known to be following secure software development practices, and which were not. And, since all open source packages are not created equally, with some backed by independent open source maintainers, and some by large corporations or foundations, they were immediately able to see the benefits of Tidelift’s partnerships with independent open source maintainers.
The chart below shows how the organization went from having zero visibility into who was maintaining their software, and the secure development practices behind those open source dependencies, to 100% supplier visibility. This new visibility into supplier information, accurate health and security data, and long term resilience reduced the overwhelm of an endless list of CVEs. The customer now had actionable information, and a way to proactively ensure that the packages they needed to keep in place would be reliable.
The company instantly had contractual commitments from the maintainers of 19% of their dependencies who were already being paid by Tidelift to implement enterprise grade secure software development practices, and continue to keep these practices in place over time. Over the next two years, Tidelift recruited additional maintainers, and now the percentage of packages backed by Tidelift-partnered maintainers is at 29% and continuing to rise. In addition, the customer also now knew which packages were backed by independent open source maintainers who were not paid by Tidelift to follow secure software development practices and didn’t have the safety net of affiliation with corporations or foundations (the 48% we refer to as “no assurances found” in the chart above).
Meanwhile, before beginning work with Tidelift, the customer had limited risk visibility via their software composition analysis (SCA) tool, which could show them which package releases in their application had known vulnerabilities. But known security vulnerabilities are only one indicator of potential risk, and they had no visibility into other risk vectors, including from packages that had been declared end of life or did not have a security policy in place.
With Tidelift, they now have 100% risk visibility, with data on other forms of risk beyond security vulnerabilities. Almost half of the packages in the application (48%) are now known to be reliable (with no known security vulnerabilities, a discoverable security policy available, and no indication that they’ve been declared end of life). Meanwhile, in addition to the 10% of known vulnerable releases, they can also see that 10% of their package versions have been declared end of life, and 32% have no security policy in place. End of life packages and packages with no security process are both important risk vectors that are invisible to SCA tools.
With help from Tidelift, the customer went from having limited visibility into who their open source suppliers were and limited visibility into their open source risk to having full visibility into both their suppliers and their risk.
They now had contractual relationships with the independent maintainers of 29% of their open source dependencies, and data showing foundations or corporations backing another 23% of their dependencies.
Increasing supplier and risk visibility was a good start. But in order to have the largest possible impact on this organization’s ability to use open source effectively, Tidelift’s mission now became “how are we going to improve the security and resilience of the other 48% of independently maintained projects so this organization can continue to increase the business value they are getting from open source?”
With Tidelift in place, the organization had the visibility into their open source dependencies to further focus their development team’s time and efforts. Tidelift could lead the effort of monitoring and ensuring accountability for the independently maintained open source packages in their application. With the increased predictability around their open source software supply chain, the customer had more time and resources to focus on their internal developers’ development lifecycles. The customer now knew:
Next, this customer needed to ensure they’d continue to improve the predictability and reliability of their application over time. Here’s how Tidelift has partnered with them in this effort:
In 2022, when this customer first came to Tidelift, 22% of the application components were already backed by corporations or foundations (and that percentage has remained fairly stable at 23% today). Tidelift has historically focused on providing income primarily to independent open source maintainers rather than those backed by corporations or foundations because (as Tidelift co-founder Luis Villa describes in his blog post Paying maintainers: the HOWTO), smaller, independently maintained projects often have fewer avenues for funding, are sometimes less objectively well-known, so the money can have a larger impact on improving security and reliability.
Conversely, projects backed by corporations or foundations may have some of the maintainers on the payroll, have more avenues for funding, be more well-known, and thus may be less “at risk” than independently maintained projects. One important caveat: this does not mean organizations should assume that the foundations or corporations backing these components have any contractual obligation to ensure they meet secure development requirements, but they can often be considered less “at risk” than components backed by independent open source maintainers who have less support and attention for their work.
Because Tidelift already had contractual commitments with the maintainers of 19% of the application’s dependencies on day one, the recruitment effort focused on the remaining application components whose independent open source maintainers were not yet partnered with Tidelift.
The chart below focuses only on the packages from independent open source maintainers (meaning, it removes the packages that are already backed by foundations or corporations). From a starting point of 24% of independently maintained components in the application with contractual security and maintenance through Tidelift, the percentage continued to rise as Tidelift recruited more maintainers because of package usage by this customer and others.
By 2024, coverage of this application by Tidelift-partnered maintainers had risen to 37%, and is predicted to continue to rise to about 80% by 2026 as more maintainers are recruited. This probably represents almost “full coverage” because not every maintainer is interested in getting paid to implement secure software development practices, and if the customer would like to see this percentage rise beyond 80%, it might require rearchitecting some of the components out of the application whose maintainers are not interested in income for their work.
We led up front with this headline, but it bears repeating. Because this organization was able to reduce its reliance on abandoned, end-of-life, or otherwise insecure open source packages it was able to:
Let’s dive a bit deeper into the business impact beneath the headlines.
With an abundance of open source packages available to modern software developers, it may seem like it would be easy to simply remove an open source package that is, or appears to be, abandoned. But it is rarely that easy. The package may be deeply embedded into the application and its dependency graphs, so the rework required may be extensive. Also, there are other costs to consider, including the time to research and test replacements, document what has changed in the application, training the team on the replacement, all while ensuring customer experience continuity.
We've never accounted for the cost of ripping out a package, but we know it’s large. A proof of concept, legal, security, and other checks are required. Does the new package come with support? Is the package stable? What does the CVE history and licensing look like? What is the adoption rate of the library—are we the first ones to adopt? If any of these answers are unsatisfactory, it can result in restarting the research and approval processes. A minimal change can drag out 3 months, average 6 months, and replacing a platform or major framework can take a year.
Then we have to plan a transition. How we will migrate a feature to a new package or framework. Because we have to ensure stability and a good experience for our users. It depends on the amount of change. If it's a smaller change, we'll roll it out within a quarter. If it's more, it takes half a year. If it’s urgent, due to a vulnerability that is presenting risk, there are multiple tradeoffs that must be made. — Senior IT Director
When an abandoned package turns up with an unfixed urgent vulnerability, either directly or through a transitive dependency, it can also cause a fire drill. For a deeply nested transitive vulnerability, this becomes a multi-week or even multi-month cost where the business is still carrying the risk until that risk is eliminated. For two high profile examples of software supply chain security threats that impacted many companies in this way, read more about colors and left-pad.
As this chart above shows, Tidelift has been able to reduce these potential costs to the business significantly already in this one application. With plans to continue improving contractual security and maintenance coverage over the next two years, the organization can expect to significantly reduce costs related to unplanned work and emergency fire drills that suck time away from other important business priorities. This organization has already saved an estimated $1.1M in costs so far, with the savings expected to rise as maintainer coverage continues to grow.
The number of reported CVEs is rapidly increasing each year, and the rise of AI capabilities to detect and report vulnerabilities in open source projects will make this number grow even faster. Maintainers are telling us that many of these AI-produced reports are time-wasting false positives today. Problems at this scale require innovative solutions—security and development teams’ capacity simply can’t keep up.
“The increase of spam PRs, comments, and false positives from AI tools and users has been enormous and very frustrating.” — maintainer, via Tidelift 2024 maintainer survey
Tidelift’s partnered maintainers are the front lines of managing their dependencies on behalf of this customers’ internal development team. Maintainers are able to mark false positives and apply patches—or move off of a dependency when a patch is not available. This is driving down risk on behalf of the customer.
What happens when CVEs enter an application transitively? Software supply chain security is a true chain, with many inaccessible nodes. Customers may decide to stop using a package directly, as this customer did multiple times, only to have it show back up transitively, through another package they depend upon. Even if teams are putting measures in place to control and approve the surface area of open source they must manage, they have no control over how many dependencies those packages bring in, or if those new packages meet their quality requirements. This customer saw Tidelift as the best way to reduce this risk.
In this example above, aiohttp’s graph is pulling in 16 dependencies where the customer cannot control updates on their own and has little visibility. But because the maintainer of aiohttp is being paid by Tidelift to ensure their dependency graph is kept free of risk, the risk to the customer’s application from transitive dependencies is reduced.
This customer was able to establish a real partnership with the upstream developers who maintain the majority of their code. Tidelift’s partnered maintainers reduced the customer cost of patching endless vulnerabilities and ripping out and replacing packages. The result was hundreds of potential hours of assessment, research, approvals, and rework saved. The customer reduced unplanned vulnerability fire drills that abandoned or unmaintained dependencies create. These partnered maintainers also ensured that transitive package risk was removed for the customer.
Based on Tidelift customers' reporting an average of 4.5 months for cross-functional teams to review a major package change, we estimate the value of this time saved to be a minimum of $1.1m. The scale of this cost reduction for an organization cannot be overstated. And, this same set of Python packages is used across 3,166 other Python applications spanning multiple application teams for this customer.
With this strategy in-hand, the customer maximizes the ROI of their whole organization’s time spent on open source lifecycle maintenance. This ensures that the customer stays competitive in the market, with their developers building functionality that drives revenue instead of wasting time on open source maintenance tasks. From adopting new packages, to monitoring and moving away from problems, Tidelift is their trusted partner helping reduce risk from bad open source packages and ensuring that the open source used in their applications keeps getting better.
Tidelift is also continuing to run analysis for packages in their applications where there are not yet maintainer partnerships in place, and the next recruitment class has been identified—including one of the customer’s currently abandoned packages! (Read more on how Tidelift uniquely does this with partnered maintainers in the socks-js case study.)
This customer will continue to be able to accurately forecast pricing, because they can stay focused on investing their time and resources in improving their internal code, while Tidelift and its maintainer partners take the lead on ensuring their open source code and software supply chain continues to be secure and resilient, now and into the future.