Recap: Life as an open source maintainer after xz

It’s been six weeks since a developer uncovered a hack of epic scope in the popular Linux compression library called xz utils (previously known as LZMA Utils). This attack wasn’t just sophisticated in a technical sense, but also sinister in a social sense: the attacker, Jia Tan, spent years gaining the trust of the xz maintainer, before he handed over the keys to the project.

Shortly after the attack was uncovered, we gathered together a group of open source maintainers across the Javascript, Java, and Python ecosystems to hear not only how the xz hack impacted their work (spoiler alert: this attack reverberated across ALL ecosystems, not just in the Linux OS!), but also how it made them feel. 

Keeping the flame of open source alive despite the xz utils backdoor

In a recent survey of open source maintainers, we uncovered that 44% of maintainers are solo maintainers; that 44% have experienced burnout; and that 58% have quit or considered quitting their maintenance work. 

Being an open source maintainer can be lonely and thankless. We asked the maintainers on our panel how the xz utils backdoor hack made them feel. 

“I maintain a lot of projects, and I am the single maintainer on almost all of them,” said Jordan Harband, who maintains hundreds of npm packages. “It's certainly not because I don't want additional maintainers. I very much do. But it's hard to find people who will take the time to maintain any package, even a small single purpose one. And so whenever a contribution comes in, I get excited. I'm like, ‘Ooh, a new person, like, how can I make them happy? How can I make them feel welcome? How can I make them feel proud of their contribution?’”

And when it comes to vetting contributions, what are the best practices? Being a maintainer is hard, lonely work, and it’s even more difficult to fully vet people online.

“A dilemma we have on the Apache projects I work on is we're trying to lower the bar for committership,” said Gary Gregory, a prolific maintainer in the Java space. “We're trying to grow communities. [the xz utils hack is] really making me rethink, how are you supposed to vet granting committership? How do we make the community more friendly and easy to get into but also create trust and quality software?” 

And does raising the bar of “committership” wash away the very essence of what makes open source “open source”? 

“I believe, very firmly, that anonymous contributions, especially small, anonymous contributions are good,” said Valeri Karpov, maintainer of Mongoose.js. “The ability for anyone to contribute to an open source project is absolutely one of the core features of open source.”

This point is backed up by Seth Larson, maintainer of urllib3 and security developer-in-residence at the Python Software Foundation.

“So one of the primary things that I concern myself with is keeping the flame of open source alive,” Seth said. “And that is so easy to stamp out, just by making something that seems like a well intentioned change. Like, okay, we're not going to take contributions from people that are anonymous anymore. And so any one of these, small changes, that seems like something good to some people, would end up just destroying open source as we know it today.”

How did this xz utils backdoor hack impact ecosystems beyond Linux?

Beyond shaking the entire open source community to its core, this attack also took up a huge amount of these open source maintainers’ time. Even though xz utils is based in the Linux world,  it caused trouble for a wide range of open source projects.

When he heard about it, Gary Gregory spent a few days trying to figure out if his Java packages were impacted—similar to when Log4Shell happened in December 2021. “Two things that came up: The first is the usual panic mode,” Gary said. “This all of a sudden took over the next two days. And even though most of the code that I maintain is on the Java side, it turned out that this was really on the C side of the project.”

As for the Python ecosystem, Seth also spent a few days assessing the repercussions. “My day job has me overseeing the security of the entire Python ecosystem, which includes the runtime, the packaging infrastructure, and the entire content of the packages themselves,” Seth said. “There was a moment of panic for me because I know with certainty that C Python bundles xz, but we do end up bundling a really old version relative to when this contributor started.”

And, of course, it also affected the Javascript community, as Jordan pointed out. “I also had a moment of panic, like, ‘oh, no, did I, many years ago, lay a time bomb that this guy is exploiting?’” Jordan said. “I'm very lucky this was caught, we're all very lucky and that this was caught very early and before it made things widely vulnerable. But it has downstream effects, I think even far distant from C and things that bundle xz directly. Anyone using Node could have theoretically been impacted by this as well, if they're using it through npm, which the last survey said like 80% of node downloads are.”

What have we learned from this attack?

Jordan raised a great point about automated fixes and AI, and how paying the maintainers is the best solution we have right now. “I think the big lesson is that, while it's great to pursue automated fixes, the real problem is trust of humans,” Jordan said. “And if you want to increase your ability to trust humans, you need to both ensure that they are not incentivized to do bad things and you also want to have leverage over them. And injecting capital into their projects is a pretty good way to achieve both.” 

And Seth pointed out, preparation is key. “Preparation and knowing what you can do in all of these situations and not falling for the panic,” Seth said. “Just know that if you're prepared, and you know what you're going to do in certain situations, whether it's a vulnerability being released, or having a co-maintainer or adding a co-maintainer, it's really hard to be in the moment if you're not prepared for it.”

You can watch the entire panel any time here.