In November 2021, a zero-day vulnerability in a ubiquitous piece of open-source code stunned the technology industry and set off an urgent effort to help secure the largely volunteer open-source ecosystem. Nearly four years later, that effort has made important progress but has also been hobbled by multiple setbacks.
The Log4Shell vulnerability in a popular Java logging tool convinced the Biden administration to focus on open-source security and prompted major tech companies including Amazon, Google and Microsoft to pledge tens of millions of dollars to security improvements. Much of that work occurred through the Linux Foundation’s Open Source Security Foundation (OpenSSF), which created numerous tools to help developers analyze and address their code’s risks.
But what began with a White House summit and an ambitious industry-wide “mobilization plan” soon encountered challenges. A tantalizing new technology known as generative AI distracted the tech giants funding the work, and a political transition in the U.S. extinguished government efforts to keep the industry on track.
Overcoming those obstacles and doubling down on open-source security is essential, experts told Cybersecurity Dive, given how pervasive the code is in everything from critical infrastructure to everyday home computing.
“We need to make sure that the momentum that we built doesn’t get lost,” said Jack Cable, a former senior technical adviser at the Cybersecurity and Infrastructure Security Agency (CISA) who worked on open-source security.
Open-source security progress
Since early 2022, an infusion of funding and attention has led to important open-source security improvements.
One of the most significant developments was the campaign to improve the security of open-source package repositories. The repository “is the modern distribution point for the majority of software that’s consumed,” said David Nalley, director of developer experience at Amazon Web Services, which has helped fund improvements in these vital platforms. Christopher Robinson, OpenSSF’s chief security architect, said the goal of this work was to ensure that “all projects within those ecosystems will inherit” strong security practices.
Amazon also helped the developers behind a TLS encryption library for the memory-safe programming language Rust adopt a cryptographic algorithm that met federal standards, making it easier for organizations that need to meet those standards — including companies in regulated industries — to use memory-safe code.
Robinson highlighted OpenSSF’s Sigstore project, which lets developers digitally sign their code to prevent tampering. He also praised tech companies for embedding security experts in communities built around certain programming languages to serve as those groups’ “ambassadors” to the broader ecosystem.
The Cybersecurity and Infrastructure Security Agency (CISA) used its authority and its experts’ reputations to build bridges between agencies using open-source code and the developers producing it.
“[We] really did a lot to make sure that … when incidents occurred, folks would be able to connect with each other,” said Cable, now the CEO and co-founder of the AI coding security firm Corridor. He said those efforts paid off during the 2024 XZ Utils crisis, in which a malicious actor used social engineering to fool an overwhelmed developer and plant a backdoor in their widely used package.
Perhaps most significantly, companies that depend on open-source packages are increasingly taking responsibility for ensuring that they are secure, instead of treating open-source developers like their unpaid labor force.
Open-source developers have complained for years that they feel exploited by the companies using their code for profit, according to Arnaud Le Hors, senior technical staff member of open technologies at IBM. Now, more businesses “realize that you can’t just depend on the community at large to fix vulnerabilities in open-source packages that you decide to use in your products.”
“A lot of good work did happen” over the past few years, Cable said, “and still a lot is ongoing.”
Declining investment
After Log4Shell exposed the precarious state of the open-source ecosystem, leading tech companies met with Biden administration officials and pledged more than $30 million in services, infrastructure and personnel to help.
But while the companies’ efforts have yielded some results, experts said they haven’t lived up to expectations.
Tech firms’ commitments have “not materialized at the amount promised,” leading to “a lot of disappointment,” said Aeva Black, who led CISA’s open-source security program for two years. In addition, many other companies haven’t even committed anything, Cable said, because they “still don’t recognize … the value that they’re getting out of open-source software” and “aren’t thinking about how they interact with maintainers, let alone start[ing] to contribute back.”
Amazon, one of the biggest early funders of OpenSSF’s work, is “investing more today than we were post-Log4Shell,” Nalley said, but its “investments have evolved as we’ve learned what works and what doesn’t. … Some [investments] have paid off well. Others, maybe not as we had hoped.”
As companies have scaled back their ambitions, the U.S. government has gone from prodding them forward to abandoning the field under President Donald Trump. Open-source work “has slowed down under the new administration,” Le Hors said.
The Biden administration pledged $11 million toward open-source security last August, but Black said “those promises are not being followed through.”
The Trump administration’s cuts to CISA and the departures of Black, Cable and other widely respected subject-matter experts have essentially eliminated the agency’s work on open-source security.
CISA’s advocacy helped open-source experts at tech companies convince their employers to “follow through on their commitments” and become more active participants in the community, Black said.
Going forward, Cable said, “it is unclear what CISA and, generally, the federal government’s level of involvement will be” in open source.
CISA “remains laser-focused” on open-source security, Marci McCarthy, the agency’s director of public affairs, said in a statement. “Open-source software is a critical building block in our software supply chain for both the federal government and U.S. critical infrastructure,” McCarthy said. “We are privileged to have a talented team at CISA dedicated to understanding and mitigating risks in open-source software.”
OpenAI accidentally upends everything
On Nov. 30, 2022, only a few months into OpenSSF’s security initiative, OpenAI released ChatGPT. As generative AI chatbots captivated the public, tech companies rushed to embrace the technology. And right around that time, Black said, “several of the big companies that made [open-source security] promises started to reassign their developers away from open-source security and onto AI tools.”
In the ensuing years, Black said, open-source security fell by the wayside as companies made “big pivots to double down on AI.”
Most of the Microsoft experts that Black worked with on open-source security when they were at the company and at CISA “have been moved over to AI teams now,” they said. As CISA was recruiting them away from Microsoft, they recalled, their team at the company was “being picked apart and reassigned onto AI work.”
Legal and policy staffers at Microsoft who had been supporting open-source work were also reassigned to AI, Black said, and Microsoft subsidiary GitHub “did a huge shift.”
Both Microsoft and Google “seem to be reallocating their human resources away from this effort,” Black said.
Microsoft did not dispute Black’s account, but Ryan Waite, its director of open source ecosystems and open source incubations, said the company remained deeply involved in the ecosystem. A Google spokesperson said it “continues to invest a substantial amount of resources and expertise” into open-source security.
Some experts believe AI will improve open-source security by dramatically speeding up the process of finding and fixing vulnerabilities. The Defense Advanced Research Projects Agency (DARPA) recently concluded a prize competition, held in partnership with OpenSSF, to develop AI-powered vulnerability detection software.
But others aren’t as optimistic. Black pointed to a frustrated conference talk by the developer of the widely used curl package. “He is underwater,” Black said, “overwhelmed by AI-based developer slop — people proposing patches that are clearly written by generative AI tools that are garbage, and he has to just keep sifting through them and rejecting them.”
Unsolved problems
A wide range of pressing open-source security issues remain unresolved, according to experts who see varying degrees of commitment to them.
One of the most serious problems is that software developers — including those supplying the U.S. military — often don’t know where the code they use comes from. “People don’t have great insight into what they’re consuming,” Nalley said. This problem is particularly serious because of how many packages are in one piece of software — an average of 180, according to Sonatype — and because of how insecure many of those packages are. Nearly four years after the widely publicized vulnerability in Log4j, the flawed version still accounts for 13% of all downloads of the package, Robinson said.
OpenSSF’s Scorecard project will help developers address these “dependency” risks, Nalley said. Software bills of materials (SBOMs) could also help illuminate packages’ dependencies, although Black said open source’s complexity rendered them less than ideal.
Identifying and aiding the most sparsely maintained but vital projects remains another significant challenge. Some projects that underpin the entire internet may be the work of one or two volunteers. “We need to be making investments in there,” Nalley said. Harvard Business School has been addressing this problem through a periodic census.
The XZ Utils crisis highlighted the importance of reducing projects’ trust gaps and understanding the provenance of every line of code, and Le Hors said OpenSSF’s Supply-chain Levels for Software Artifacts (SLSA) project would help with that.
Tech companies rewriting packages in memory-safe programming languages face adoption challenges. “A lot of the [packages] we tried to rewrite did not see [significant] adoption,” Nalley said. His team helped rewrite the all-important sudo package and encouraged Linux distributions to incorporate it, but they said it had too many dependencies, so the team had to redo it.
More work also remains to secure package repositories. “Investing in the plumbing is really important,” Nalley said.
Even as work in the U.S. slows down, other governments aren’t standing still. New European Union legislation will hold businesses responsible for securing the open-source code they list, which will likely have global ripple effects.
“We’ve made lot of progress since the Log4Shell issue,” Le Hors said. “We’re still making some progress, thankfully, because the U.S. is not the whole world.”
