A new report just out last week from the Digital Forensic Research Lab (DFRLab) at the Atlantic Council found that open source projects with funding have better security practices in place. 

Those of you who are familiar with Tidelift’s work paying open source maintainers to implement secure software development practices will no doubt find this conclusion, let’s say, unsurprising 😄. But this is still a big moment for open source, because this report represents one of the first non-Tidelift, data-driven analyses of the correlation between funding maintainers and better security practices, and it is coming from a highly reputable source in the Atlantic Council, an influential policy organization that shapes conversations around key issues in the United States and around the world. 

From the report, emphasis mine:

“...these findings indicate with moderate confidence that there is a meaningful connection between more open-source project funding and improved security posture. Some practices are strongly associated with funding, and more funding generally correlates with more dramatically differentiated security practices.”

If you care about open source software security, I’d strongly encourage you to go read the report so you can parse the findings yourself, but, hey, I know we are all busy, so I’ve put together some highlights below that stood out for me.

Does general financial investment (”more money”) improve security for open source projects?

The authors of the report started from the point of view of trying to answer the question of whether they could find evidence that “more general financial investment (“more money”) improves security for open source software projects?”

They started by looking for existing data about the impact of money on open source security and found there is not much data out there already (with the 2023 Tidelift open source maintainer impact report, written by yours truly, as “to [their] knowledge, the only exception”). Next, they built a methodology that included identifying the top 1000 most downloaded projects in the Python and JavaScript ecosystems, and used a tool called Funder-finder to determine each project’s funding sources, including GitHub sponsors, Tidelift, Open Collective, Google Summer of Code, and NumFOCUS. OpenSSF scorecard scores were used as a quantitative measure of the security posture of open source projects.

They summed up the answer to their original question with three key findings, again emphasis mine:

1. The statistical evidence, which is of only moderate strength, supports the view that some general-purpose open-source funding vehicles do correlate moderately with more secure open-source software projects. In short, more money does seem to correlate with better security practices.
2. Additionally, there is some evidence that a greater number of unique sources of open source funding for a given project also corresponds to a project with better security practices.
3. The detailed quantitative evidence suggests that more funding positively correlates with better compliance with several, though not all, security practices (as measured by the OpenSSF Security Scorecard tool), instead of any single security practice.

And then drew the following conclusion:

“Most importantly, this study presents prima facie evidence of a positive effect of general open source software funding on open source software security.”

(I had to look it up too, but prima facie means “based on first impression, or “accepted as correct until proven otherwise.”)

General funding vs. targeted funding

The research project was specifically focused on what the authors refer to as “general funding” as opposed to targeted funding that is directly tied to a specific security goal. 

I found this interesting because, while the report classified Tidelift as a general funding source, we’d probably classify ourselves as more of a targeted funding source since we recruit, contract, and pay maintainers specifically to implement a set of secure development practices, validate the practices they follow, and contractually commit to continuing those practices into the future as part of our partnership with them (read more about our work with open source maintainers here and read more about the specific practices they are under contract to implement here).

Even the authors admit the assumption that all of the examined funders are general funders is an oversimplification. Specifically about Tidelift, the report states:

“Tidelift, similarly, works with maintainers to explicitly improve security practices and requires supported projects to take some measures that might boost their scorecards. One example is the robust association between Tidelift funding and the presence of a project security policy, particularly in the npm ecosystem. Tidelift terms require such a policy, and so long as the file is named SECURITY.md, it will satisfy the Scorecard check. In this way, Tidelift funding is more specific than general funding, but the other requirements it makes of maintainers appear minimal enough to still constitute as general funding for the purposes of this proof-of-concept study.”

While we do try to ensure that the list of tasks we ask maintainers to complete is not too onerous, we do regularly add new tasks that will improve project security, and we contractually tie recurring, reliable income to completing these tasks. But we also hope that beyond these specific tasks, maintainers can use the funding to do important work to improve the health and security of their projects above and beyond these minimum requirements as well (and you can find some specific stories of what maintainers have been able to do with their income on our website).

Does the amount of funding or the type of funding impact security?

This research project is clearly positioned by the authors from the beginning as an initial proof-of-concept, since it is of limited scope (covering just the top 1000 projects in Python and JavaScript alone) and makes some generalized assumptions about the obligations that those receiving funding do or do not take on, among other things.

Another aspect we believe would be notable to dive deeper into in future studies is not just the sources of funding, but the amount of funding and the impact that has on security posture. In this project, each funding source was simply viewed as a binary checkbox where a project received funding or did not receive funding from each source, probably due to limitations in the data coming from Fund-finder. But bringing a project up to a baseline of security hygiene takes time and sometimes a lot of work, so the amount of funding can make a big difference in how much security work can be implemented and in what timeframe.

Also, in our experience, the type of funding really matters in terms of the work maintainers are able to accomplish. Some funding is one-time-only payments, and we’ve found in our work that there is an enormous difference in what maintainers can do with one-time payments (especially when money has strings attached, like being tied to implementing a new feature or fixing a bug) versus when it is recurring income that they can count on continuing over time.

Tidelift is focused on providing maintainers with recurring monthly income, and we believe this is unique in that it helps maintainers set aside the time they need to not only implement secure software development practices, but ensure the practices continue to remain in place into the future. Our co-founder Luis Villa wrote more on this in his recent blog post Paying maintainers: the HOWTO.

Final thoughts

Trusting the security of our open source software supply chains has never been more critical. The cost of under-investment in open source affects package security, but bottom line costs show up in organizations that are using open source. The cost shows up as late stage vulnerability fire drills that stall time to market, re-work when a critical package is abandoned, and lost time to navigating an ever-increasing list of problems. Ensuring that the open source you rely on has investment does yield security outcomes, and even more practically—it reduces engineering toil and cost.

Overall, I found this report, and its methodology, really refreshing to see. We’ve learned a lot at Tidelift about how to effectively pay open source maintainers over the past seven years, and sometimes it feels like lonely work. 

So thanks, Atlantic Council, and in particular report authors Sara Ann Brackett, John Speed Meyers, and Stewart Scott, for adding your meaning to this important conversation. We are hoping that this proof of concept is the first of more research like this coming out of the Atlantic Council, and would be excited to share some of our learnings and data to help inform future reports.