Have you ever wondered what the open source maintainers that your business relies on do to keep our software healthy and secure? Here’s the third and final post in a series about urllib3, a Python package that is downloaded billions of times a year, and what it takes to keep it well maintained and up to date.

In the first post in this series, we discussed what urllib3 is, who maintains urllib3, how urllib3 maintainers handle CVEs, and why you should care. The second post dove deeper into how the urllib3 team establishes account security, fine tunes process, and more. If you’d like to read the entire case study, you can download the full whitepaper right now at the link below.

Taking lessons learned, and teaching them wider

All this work to improve urllib3 is great, as it helps secure and maintain a foundational piece of the Python ecosystem. Seth and the urllib3 team, however, think bigger—how can they take the lessons they’ve learned and spread them across the ecosystem and make software better as a whole?

Documenting processes for package maintainers

The urllib3 team went through many steps, as noted in the previous post in this series, to secure their accounts and establish how the team used them to develop software. This then became a blog post on how developers should practice secure behaviors.

Improving scores against industry standards

If you can’t measure it, you can’t improve it. - Peter Drucker

The urllib3 team is constantly looking for ways to improve the security posture of their code. Recently, the Open Software Security Foundation introduced their Scorecards project, which scans a project repository and scores it against a number of best practice criteria for secure development.

The urllib3 team investigated their project’s scorecard, and immediately saw room for improvement. Within just a couple of weeks, their scores had improved and they became the first Python project to score at least a 9.0 out of 10 (as of September 2022). This measurable improvement in their security posture shows their commitment to best practices to protect their users.

After working on their own improvements of their OpenSSF scorecard score, Seth was curious how the rest of the Python ecosystem stacks up. This led to the analysis of scorecards across the entirety of PyPI as a public dataset that can be used for analysis to drive improvements.

Improving the standards themselves

As part of improving their own project’s security, Seth and the urllib3 team are improving the standards they are aligning to. So far, this has included discovering multiple bugs in provenance generation across tools, and improving OpenSSF scorecard checks to include SLSA provenance.

Building a standard secure framework for developers

Having one package do the work to use secure, verifiable, development and release practices is great. But what if they all could?

Seth is taking his lessons learned in building a secure framework for the development of urllib3 and creating a secure project template for Python developers. By cloning this template, a project can immediately start taking advantage of Seth’s research and create a project that can build using and aligning to secure practices.

All of these initiatives benefit not just Python, but open source software as a whole as security standards improve. Every user of software that adopts these practices and standards will benefit from the work that the urllib3 team has done.

How urllib3 maintainers can do this work

It’s not a spoiler—all of these things take time.

• Handling CVEs? Takes time to verify reports, produce fixes, and perform releases.
  
  
• Diagnosing breaches? Takes time to investigate what was breached, and what the consequences might be.
  
  
• Building automated release processes? Time to trial and error different setups, test release processes, and hook platforms together.
  
  
• Improving packages against industry standards? Time to investigate each new standard, and perform necessary remediation (or fix the standard).
  
  
• Making reproducible, verifiable builds? Time to research new tools, implement them, and migrate to new build processes.
  
  
• Deprecating extensions and making it secure by default? Time to analyze the ecosystem, find users, and get all of them to update.

Quentin Pradet, one of the urllib3 maintainers, said “the most important thing to maintainers is time. Time to do the development work, but also time to do all the other things—research new software standards, perform non-development maintenance tasks, secure infrastructure, and even recruit new potential maintainers.”

How do you give maintainers time? You don’t do it by giving them a list of tasks from your software composition analysis (SCA) tool. You don’t do it by going to their issue tracker to ask them about a two-year old CVE they already have marked as not exploitable. You don’t do it by giving them new unfunded mandates on how to secure their software.

The easiest way for an organization to ensure maintainers have time… is to pay them.

“People are willing to absorb way more pain [if they are paid],” Seth said. “Paying people to be around a project just makes it more secure intrinsically. There’s always someone around to deal with the burstiness aspect of open source issues.”  

Not every maintainer has time, all the time… but if you’re paying the maintainers, someone will always be there no matter what.

Paying the maintainers also can scale time. urllib3 is taking the money they make via Tidelift, GitHub Sponsors, and other income sources and using it to set up bounty programs to incentivize more time to fix issues. Their deprecation of the ‘[secure]’ extra utilizes bounties for providing fixes for other projects that still depend on it. They set bounties for additional maintenance work,  such as ensuring their software works with upcoming Python cryptography changes ahead of time. They also use these bounties as recruiting tools for new maintainers—contributors who tackle multiple bounties can be invited to become co-maintainers.

Paying the maintainers is the way

In short, the urllib3 maintainers are able to do all these things a professional software organization might do, because they are paid to do professional software development. That’s no accident—they could not do all these things if they were doing it in their spare time while they did other work to keep their lights on.

The urllib3 team is a group of open source professionals, doing professional open source work, all aligned on keeping the code secure, backwards compatible, and easy for the entire world to continue depending on. They then take this work, and make it available for other projects to build on, securing the ecosystem as a whole. That sort of work doesn’t just happen—it requires time, effort, and the money that supports that time and effort. Is your organization paying the maintainers like Seth to provide these professional services? If not, you should be.

We hope you've enjoyed this series about urllib3 and their secure development practices. If you'd like to review the white paper in its entirety, you can download the entire story below.