In mid-2024, Tidelift fielded its third survey of open source maintainers. More than 400 maintainers responded and shared details about their work, including how they fund it, who pays for it, and what kinds of security, maintenance, and documentation practices they have in place today or would consider in the future. They also shared their thoughts about some “in the headlines” issues like the recent xz utils hack and the impact of AI-based coding tools. In this post, we share the ninth of twelve key findings. If you don’t want to wait for the rest of the results, you can download the full survey report right now.
In late March 2024, a developer from Microsoft noticed some unusual behavior on their computer, investigated it, and uncovered a hack of epic scope in an obscure but important library called xz utils. The attack was technically sophisticated, but perhaps worse, it was socially sophisticated. The attackers took advantage of an open source maintainer over a long period of time (years) to slowly, but steadily, win his trust—and then subvert the security mechanisms that he had previously put in place.
The maintainer facing this deliberate, long-term attack was, in his own words at the time the hack began, “unpaid.”
“I haven’t lost interest but my ability to care has been fairly limited... it’s also good to keep in mind that this is an unpaid hobby project.”
In the same email, this maintainer said:
“Jia Tan may have a bigger role in the project in the future. He has been helping a lot off-list and is practically a co-maintainer already. :-)”
It was exactly this “Jia Tan” who, over a period of two years, took over xz and inserted a malicious backdoor that could have exposed computers the world over to remote execution. Thankfully this attack was discovered before it could cause extensive damage.
But we wanted to use this year’s maintainer survey to find out whether the xz utils hack has inflicted any collateral damage; namely, has it negatively impacted the way open source maintainers think about their work?
First, we got a baseline read on how many maintainers were even aware of the xz utils hack. We provided a brief description of the hack, and then asked them if they were aware of the hack prior to taking the survey. The vast majority of maintainers (88%) were already aware.
How has the xz utils hack impacted maintainer trust?
Next, we asked those that were aware of the xz utils hack the following question:
How much do you agree with the following statements in terms of how the xz utils hack has impacted the way you approach your work as an open source maintainer?
-
- I’m less trusting of the contributions of my co-maintainers or feel I need to vet them more carefully
- I’m less trusting of pull requests from non-maintainer contributors or feel I need to vet them more carefully
- It has added to my personal stress
Of the three statements, maintainers most agreed with: “I’m less trusting of pull requests from non-maintainer contributors or I feel I need to vet them more carefully.” Two-thirds (66%) either agreed or strongly agreed with that statement, which shows that the xz utils hack has had a significant impact on maintainer trust of contributions.
We also asked maintainers to tell us in their own words how the xz utils hack has impacted their work, and several spoke directly to this point.
"I'm definitely taking code review more seriously and no longer merge code I don't understand. Worst case, the author will have to walk me through line by line.”
Or as another maintainer put it:
“It made me more aware of the sophistication of actors looking to compromise projects, and made me more mindful of vetting non-trivial code changes.”
But that degradation of trust also creates challenges when it comes to encouraging contributions, as another maintainer observed:
“I feel the need to add a layer of vetting, but adding any additional layer of friction to a possible open source contributor would just scare them away. I cannot afford to be pushing people away.”
Thankfully, on the other two statements we asked about, the results were less concerning. Only 37% agreed or strongly agreed with the statement, “I’m less trusting of the contributions of my co-maintainers or feel I need to vet them more carefully.”
One maintainer felt like xz had brought them closer:
“Made me appreciate my co-maintainer.”
Another felt that even though the time horizon on this hack was long, their co-maintainer relationships had proven trustworthy over an even longer timespan:
“While the xz hack was longer than might be expected, the time spent by my co-maintainers is still longer than that, so I feel confident in their reliability.”
And an even smaller percentage (27%) agreed or strongly agreed with the statement “[xz utils] has added to my personal stress.”
One maintainer who agreed with that statement made the following observation:
"Now, checking in code from other contributors is stressful."
Maintainers express the impact of the xz-utils hack in their own words
Overall, by analyzing the open text responses, we were able to categorize maintainers’ concerns expressed in their own words into a few high-level categories. The most-often-shared sentiment was that, in the wake of the xz utils hack, maintainers will need to increase their caution during code review and vetting.
One maintainer expressed concern that even if contributions were technically sound, that might no longer be enough.
“Now I have to be extra careful with reviewing pull requests from some random GitHub user. Even their proficiency doesn't make you believe them.”
Several other maintainers indicated they’d be taking a more cautious approach in the future, for example:
“I am MUCH more suspicious of any code contributions from non-maintainers, especially from individuals who have not been active in our user community.”
Some maintainers were concerned about what the xz utils hack says about how we can rebuild trust and improve community dynamics.
“This incident really highlighted for me that technology is not the problem—culture is. Without authentic, trustworthy support from a real community (not merely an accidental collection of strangers who have a single common interest) this kind of thing will only continue. Security is a wetware problem first and foremost—we need to care about actual, living humans, not just certs and hashes and chains of custody.”
“It's not a new problem, this issue has always been there in some shape or form, it's just shined more light on an existing problem (which is good). But we still need actionable things that can be done to help mitigate these problems. Ring of trust, reproducible builds, etc. lots of these things are great ideas, but they are all half implemented. (e.g. how many companies actually validate GPG signatures from packages they download, with an existing ring of trust, reproducible builds are great, but they don't provide a mechanism to certify the rebuild process).”
And, notably, many maintainers felt that the hack would have no significant impact on their work at all. As one put it:
“Trusting new maintainers by default is the open source way, and how it should remain. Throwing everyone under the bus just because of one threat actor is not a good idea. I added a new maintainer to a security-critical package I maintain recently, and I vetted them as I would have vetted them before xz-utils—see past contributions, and keep an eye on all the changes they are pushing.”
Finally two maintainers nicely summed up the entire discussion with these thoughtful words:
“I've always been aware of the possibility of a sabotage contribution. I've refreshed my thinking on the risk and doubled down on the need to validate each solution for its merits and not accept a change that I wouldn't be happy putting my own name on the commits.”
“I think the case has, despite clearly being a bad thing, helped shine a brighter light on the whole supply chain and how we need to have clear and open standards for every aspect of projects, not just code or documentation.”
Well said.
50 Milk St, 16th Floor, Boston, MA 02109
.svg){kind=icon}