Yesterday, the U.S. Department of Homeland Security released the first report from the recently created Cyber Safety Review Board (CSRB), reviewing the December 2021 Log4Shell vulnerability. The report was delivered to President Biden through Secretary of Homeland Security Mayorkas.
The CSRB report:
- reviews the facts and timeline of the Log4Shell event
- presents a number of the board’s key findings with respect to Log4Shell
- outlines recommendations for future actions by the private and public sector to enhance open source software supply chain security
In this post, we highlight a number of elements of the CSRB report that are most relevant to organizations relying on open source, and to the creators and maintainers behind widely used open source projects.
CSRB findings on impact and response to Log4Shell
It’s no surprise that the CSRB found that organizations with processes and tooling in place to track their open source dependencies, and the standards that they meet, were best positioned respond to the Log4Shell event:
After disclosure of the Log4j vulnerability, organizations had to rapidly assess and manage their potential risk, at both a technical and business level. Generally, the Board found that organizations that responded most effectively to the Log4j event already had technical resources and mature processes in place to identify vulnerable assets, absorb new information, assess potential risk, and mobilize their entire organization and key partners to action.
What’s less obvious from the report is how most organizations can put in place the right processes and tooling if they don’t have them already.
Somewhat surprisingly, the CSRB report found organizations using Software Bills of Materials (SBOMs) didn’t have a significant advantage in their response to Log4Shell:
Software Bills of Materials (SBOMs) provide a list of components included in software, and theoretically should enable organizations to identify vulnerable software components in their environments. The Board spoke with representative groups for organizations currently using SBOMs in their environments, and none reported having leveraged them to identify vulnerable deployments of Log4j.
As designed today, SBOMs are limited, for example by variances in field descriptions and a lack of version information about catalogued components, and lack of automation on the consumption end due to these variances. Addressing SBOM standardization gaps would support a faster software supply chain vulnerability response.
Our take: SBOMs are necessary, but not sufficient, to ensure open source software supply chain security. For a more in depth discussion of why, see Software bills of materials are important—but they won't work at scale if we don't pay the maintainers.
The operational impact of Log4Shell was dramatically illustrated by one datapoint the CSRB found:
Organizations spent significant resources as they struggled with this problem. For example, one federal cabinet department reported dedicating 33,000 hours to Log4j vulnerability response to protect the department’s own networks. These costs, often sustained over many weeks and months, delayed other mission-critical work, including the response to other vulnerabilities.
33,000 hours lost is around 16 person-years of wasted effort responding to Log4Shell for this one organization—and we’ve heard countless other stories along these lines. Based on Glassdoor’s $100k/year base pay average estimate for U.S. federal government software engineers, this particular department was forced to allocate about $1.6m in taxpayer dollars just to remediate this one vulnerability!
And that was just one of 15 federal cabinet departments, suggesting an overall impact of at least $24m across federal departments if it’s representative.
CSRB findings on root causes of Log4Shell
As it turns to examining root causes, the CSRB report describes the challenges that come with public and private sector organizations relying on volunteer projects that don’t necessarily have comprehensive security processes, nor adequate support for the software creators behind them:
“The event also called attention to security risks unique to the thinly-resourced, volunteer-based open source community. This community is not adequately resourced to ensure that code is developed pursuant to industry-recognized secure coding practices and audited by experts. To reduce recurrence of the introduction of vulnerabilities like Log4j, it is essential that public and private sector stakeholders create centralized resourcing and security assistance structures that can support the open source community going forward.”
“The Board also found specific challenges associated with maintaining open source projects like Log4j, which generally rely on volunteer teams and do not necessarily have dedicated security resources throughout the Software Development Lifecycle. Open source projects generally do not have dedicated coordinated vulnerability disclosure and response teams that investigate root causes of reported vulnerabilities and work to bring them to resolution.”
Furthermore the CSRB accurately reflects that asking more of open source volunteers will require sustained financial support:
“The Board also found that the only way to reduce the likelihood of risk to the ecosystem caused by vulnerabilities in Log4j, and other widely used open source software, is to ensure that code is developed pursuant to industry-recognized secure coding practices and an attendant audit by security experts. The open source community, which is volunteer-based, would need sustained financial support and expertise to make this possible.”
CSRB recommendations on open source software supply chain security
To its considerable credit, the CSRB report doesn’t just inventory the known facts and causes of Log4Shell. It also makes direct recommendations for the future.
To begin with, the CSRB report recommends a direct increase in investment in open source software security:
“Recommendation 13: Increase investments in open source software security.
The U.S. government is a significant consumer and producer of open source software, and therefore should play a role in driving security enhancements to the overall open source ecosystem.
- OMB should take appropriate steps to direct federal agency IT staff to contribute to the security and maintenance of open source software upon which they rely, as part of their regular duties.
- ONCD, in coordination with OMB, should consider effective funding mechanisms to invest in widely used open source software tools, and to catalyze improvements in the overall security of the open source software ecosystem.
- NIST should engage with the open source software community to develop practical guidance on how to most effectively apply federal policies and frameworks around secure software development for open source software.
Private sector companies that build commercial software that includes open source libraries or dependencies should commit financial resources toward the open source projects that they deploy. Specific examples include:
- paying developers, security engineers, and other essential roles in supporting secure software development, vulnerability disclosure and handling processes, and vulnerability response for open source projects; and
- contributing funding and knowledge to centralized open source security projects.”
Next, the CSRB report recommends exploring “more sustainable models” for open source software security at scale:
“Recommendation 14: Pilot open source software maintenance support for critical services.
The Board heard from many interviewees about the operational risks introduced to organizations because of limitations in open source software maintenance. Open source developers are usually not paid to maintain their software, especially not older versions that are often deployed deeply and widely across systems. Log4j was a particularly extreme example of this. The burden of maintenance then falls to organizations that use open source software, and can be expensive, time-consuming, and unpredictable. Organizations should be able to project the maintenance costs of open source software, and obtain that information from a trusted source. Funding the maintenance of critical open source software could drive a more sustainable model for security at scale and enable timely and effective distribution of updates and mitigations.
The report then cites the opportunity to leverage industry projects such as the OpenSSF’s Criticality Score and Security Scorecards projects.
Finally, it’s particularly notable that the the CSRB report calls out the need for innovation in incentive structures for open source creators:
“Recommendation 18: Study the incentive structures required to build secure software.
Many interviewees stressed the volunteer nature of the open source software community. In light of that, the Board recommends the U.S. government’s National Academy of Sciences Cyber Resilience Forum undertake a study of incentive structures to build secure software. The Forum should consider:
- legal, structural, and regulatory incentives for organizations, to include, for example, the potential for software liability reform;
- developer incentive structures that increase awareness of safety issues. For example, developers could obtain a trust rating for their contributions through extended education on secure coding or threat modeling. This developer rating could be a component of the software’s overall health rating; and
- challenges to adopting incentives, including, but not limited to, resourcing and volunteer fatigue.”
On this point, we think it’s actually quite simple: pay the maintainers of open source to validate that their packages meet defined production-ready standards, and we’ll all be better off.
Conclusions
With the CSRB’s report, the U.S. federal government has taken a significant step towards improving our collective open source software supply chain security. We’re encouraged that the government is leading the charge with its regulatory and enforcement capabilities, and we’re excited by the report’s tantalizing hints of future efforts to catalyze the necessary direct financial support that open source maintainers need to complete their important work.
50 Milk St, 16th Floor, Boston, MA 02109