Late last week, a developer noticed some unusual behavior on their computer, investigated it, and uncovered a hack of epic scope, in an obscure but important library called xz. The attack was technically sophisticated, but perhaps worse it was socially sophisticated. The attackers took advantage of an open source maintainer over a long period of time to slowly, but steadily, win his trust—and then subvert the security mechanisms that he had previously put in place.

The maintainer facing this deliberate, long-term attack was, in his own words at the time the hack began, “unpaid.”

“I haven’t lost interest but my ability to care has been fairly limited... it’s also good to keep in mind that this is an unpaid hobby project.”

In the same email, this maintainer said:

“Jia Tan may have a bigger role in the project in the future. He has been helping a lot off-list and is practically a co-maintainer already. :-)”

Like any good horror story, you can see where this is going: it was exactly this “Jia Tan” who, over a period of two years, took over xz and inserted a malicious backdoor that could have exposed computers the world over to remote execution.

We got lucky this time: the problem was caught early. But it will not be the last time this sort of sophisticated attack is tried.

So what does this hack mean?

Over the weekend, the internet quickly blossomed with takes. All were a mix of angry and sad that this maintainer, clearly operating in good faith, had been taken advantage of. Past that, though, there was a lot of disagreement about how to reduce the odds of this happening in the future.

One of those frequently recurring subthreads was of particular interest to me: “how can we work together to pay maintainers like this one, so they are more resilient to attacks like this one?”

Hi, we're Tidelift

From day one at Tidelift, we’ve said that overworking and underappreciating maintainers, like xz’s, is a huge problem. It leads directly to burnout, bugs, and downstream costs.This isn’t the fault of the maintainers—collectively, our industry has made them the basis of a trillion-dollar economy without giving them anything but extremely indirect benefits. We founded Tidelift to address a root cause of that overwork and underappreciation—by paying them.

Forgive the long quote from our website, but we wrote this a long time ago and it still stands:

[M]ost maintainers are volunteers. A recent Tidelift maintainer survey showed that 60% of maintainers are unpaid hobbyists. And countering the regularly cited theory that most maintainers prefer to work on open source as an unpaid hobby, our survey found that 77% of the maintainers who are not paid would prefer to get paid.

The crisis of overworked open source maintainers being asked to do more isn’t just wrong, it is dangerous. 

At Tidelift, we want to change this. We see a world where open source maintainers get full recognition and compensation for the value they create. We see a world where maintainers thrive.

Our solution is so simple that, especially at times like these, it seems almost naïve: we should pay the maintainers. And we need to pay maintainers, not just of famous packages, but of the packages you've never heard of until there's a crisis: today it is xz, but yesterday the packages you’d never heard of were left-pad, event-stream, and minimist. We can’t know what they’ll be tomorrow, but we can know that they’ll exist, and that we have to cast a wide net to find them.

What we do that might have helped

So what is it, exactly, that Tidelift does that might have helped here? As I’ll discuss below, there is no one answer, but we do a lot that helps reduce the probability of an attack like this.

• We pay maintainers to reach security and licensing goals. Put simply, we pay maintainers to help them become better maintainers, and to keep maintaining the software. That money is tied to completing specific tasks that ensure the software is built using secure development practices, well maintained, and properly licensed.
  
  
• We pay known maintainers. You can’t just show up and say “hey, I want to maintain project X.” Instead, we work to ensure that the money goes to the people already doing the maintenance. This both sets the right expectations (what we’re paying for is committed, long-term work) and increases trust (though of course, as we were reminded this weekend, trust can be subverted in many ways).
  
  
• We pay based on project usage, not vibes. We pay maintainers based on an analysis of how many of our customers use their software (and, originally, how projects were used in public dependency trees). In contrast, too many well-intentioned initiatives in this space rely on identifying projects based on “did one of my employees like it?” That’s not bad (I donate that way too!), but in practice it ends up directing money to a handful of well-known projects, and missing more obscure-but-critical projects like xz. Doing objective usage analysis is the only chance we have of getting money to the places where it can help before the crisis happens.
  
  
• When a maintainer partner needs to move on, we help ensure project continuity. Sometimes, for perfectly good reasons, a maintainer wants to stop working on their project. When this happens, we help find trusted maintainers from our network interested in getting paid to continue the work. We’ve done this, for example, with SockJS and minimist.
  
  
• We collect money based on a commercial exchange of value, not fickle charity. I love charity, having worked for Wikipedia and even (an infinity ago) a homeless shelter. But as we have all seen in the past two years in tech, charitable efforts are the first thing to get cut in a downturn—exactly when maintainers are most likely to be stressed about their jobs and their projects. By providing commercial services around this transfer of money, we make it more robust—and more appealing at the large scale necessary.
  
  
• We focus on the middle of the stack. The traditional operating systems vendors have built a lot of infrastructure to vet and support core operating system features, and the Linux Foundation and others have put in a lot of work around the biggest high-level development tools and frameworks. But that has left a huge number of relatively untouched packages in every modern language stack (from Java to Python to Javascript) that need Tidelift-style attention. We believe our highest impact, on open source and for our customers, lies in supporting those packages.

There’s also one important thing we won’t do.

• We never shame maintainers for not doing enough. Unfortunately some responded this weekend with "this wouldn't have happened if the maintainer had done more work to get a good score on our rating system.” As we’ve said for years, the answer to this problem cannot be “do more work, for free”—if the industry (or government!) wants tedious work to be done, no matter how important that work is, it must be paid for. There is no other way for change to happen across tens of thousands of key packages.

But what about...? 

I’ve gotten a decent number of “but what about X, do you do X yet” questions this weekend, and most of them are quite fair. A sampling:

• “Do you offer mental health benefits?” Consistent with our goal of having resilient, thriving maintainers, we would love to offer mental health benefits (and vacation coverage!). We have looked into it repeatedly, but the overhead has been substantial, especially when trying to do it in all of the 40+ countries where we support maintainers. We will continue to explore this, because we agree that it is the next obvious step after direct monetary support.
  
  
• “Shouldn’t government do this?” We agree firmly that open source security is the kind of infrastructure investment where governments should play an important role, just like the investments in other public goods like clean water, safe roads, and reliable airports.  It’s also why we’re excited to have many public sector customers, and why we’ve lobbied repeatedly for the U.S. federal government to pay the maintainers.
  
  
• “Do you do C?” As we mentioned above, we focus on the middle of the stack. Besides those motivations, there are a few other reasons. First, C/C++ dependency handling is relatively primitive, so implementing our dependency analysis would be time-consuming to do well. Second, the deeper we go into C/C++ the less money is left for the (mostly completely unsupported) higher-level languages. If this weekend’s crisis has piqued your interest in paying for support of C/C++, though, let us know!
  
  
• “How much money would it take?” While we do pay some maintainers six figure incomes, we always want to pay more money to more maintainers. If you agree, there’s a great solution—tell your company to contact us :)

Money: not a magic bullet, but a cornerstone

Let me conclude by saying the obvious: paying maintainers is not a magic bullet. Even if we’d been in daily communication with the maintainer of xz, and writing them a check monthly, there is no guarantee Tidelift would have been able to stop this. That’s because every human breakdown of this sort has many causes, including (in this case) concerted attack and deliberate emotional manipulation. 

So our team would be the first to tell you we need more than just money: among many other things, we need to move away from fragile, antique build chains; we need more collaborative efforts to provide trustable reviewers; we need more sandboxing to deal with the failures that will be inevitable no matter how much we try to stop them; so on and so forth.

But at Tidelift we firmly believe that money paid to maintainers is the cornerstone all this other work must be built on. We cannot say “work on my security solution” without creating more time and energy to do that work. 

We cannot say to maintainers “do it for the ideology of open,” because their work has been snarfed into every closed piece of software on the planet. 

We cannot say “do it for the public,” because those maintainers aren’t dumb: they know not just the public but also a very profitable trillion-dollar industry rides on their oft-uncompensated work.

We cannot say “do it for your friends,” because maintainers talk to their friends in the dependency stack—and they know those friends are just as tired and overworked as they are.

We have gone to all the wells in our quest to squeeze more labor from these stones. Paying the maintainers is the only one left on which to build the foundation of a future of secure, reliable, resilient software industry. Join us! The maintainers need your support.