Funded open source security work at the Linux Foundation

Open source software (OSS) is vitally important to the functioning of society today; it underpins much of the global economy. However, some OSS is highly secure, while others are not as secure as they need to be.

By its very nature, open source enables worldwide peer review, yet while its transparency has the potential for enhanced software security, that potential isn’t always realized. Many people are working to improve things where it’s needed. Most of that work is done by volunteers or organizations outside the Linux Foundation (LF) who directly pay people to do the work (typically as employees). Often those people work together within a foundation that’s part of the Linux Foundation. Sometimes, however, the LF or an LF foundation/project (e.g., a fund) directly funds people to do security work.

At the Linux Foundation (LF), I have the privilege of overseeing focused work to improve OSS security by the very people paid to do it. This work is funded through various grants and foundations, with credits to organizations like Google, Microsoft, the Open Source Security Foundation (OpenSSF), the LF Public Health foundation, and the LF itself.

The LF and its foundations do much more that I don’t oversee, so I’ve only listed the ones I am personally involved with in the interest of brevity. I hope it will give you a sense of some of the things we’re doing that you might not know about otherwise.

The typical LF oversight process for this work is described in “Post-Approval LF Security Funding.” Generally, performers must provide a periodic summary of their work so they can get paid. Most of those summaries are public, and in those cases, it’s easy for others to learn about their interesting work!

Here’s a sample of the work I oversee:

• Ariadne Conill is improving Alpine Linux security, including significant improvements to its vulnerability processing and making it reproducible. For example, as noted in the July 2021 report, this resulted in Alpine 3.14 being released with the lowest open vulnerability count in the final release in a long time. Alpine Linux’s security is important because many containers use it. For more information, see “Bits relating to Alpine security initiatives in June” and “Bits relating to Alpine security initiatives in July.”
• kpcyrd is doing a lot of reproducible build work on Linux distributions, especially Alpine Linux (including on the Raspberry Pi) and Arch Linux. Reproducible builds are a strong countermeasure against build system attacks (such as the devastating attack on SolarWinds Orion). More than half of the currently unreproducible packages in Arch Linux have now been reviewed and classified.
• David Huseby has been working on modifying git to have a much more flexible cryptographic signing infrastructure. This will make it easier to verify the integrity of software source code; git is widely used to manage source code.
• Theo de Raadt has also been receiving funding to secure the critical “plumbing” behind modern communications infrastructure:
  • This funding is being used towards improving OpenSSH (a widely-used tool whose security is critical). These include various smaller improvements, an updated configuration file parser, and a transition to using the SFTP protocol rather than the older RCP protocol inside the scp(1) program.
  • It is also being used to improve rpki-client, implementing Resource Public Key Infrastructure (RPKI). RPKI is an important protocol for protecting the Internet’s routing protocols from attack. These improvements implement the RPKI Repository Delta Protocol (RRDP) data transfer protocol and fix various edge cases (e.g., through additional validation checks). The https://irrexplorer.nlnog.net/ service is even using rpki-client behind the scenes.

• Nathan Chancellor is improving the Linux kernel’s ability to be compiled with clang (instead of just gcc). This includes eliminating warning messages from clang (which helps to reduce kernel bugs even when gcc is used) and fixing/extending the clang compiler (which helps clang users when compiling code other than the Linux kernel). Unsurprisingly this involves changing both the Linux kernel and the clang/LLVM compiler infrastructure, and sometimes other software as well.
  • In the long run, eliminating warnings that by themselves aren’t bugs is important; developers will ignore warnings if there are many irrelevant ones, but if there are only a few warnings, they’ll examine them (making warnings more useful).
  • Of notable mention for security implications is clang support for Control-Flow Integrity (CFI); this can counter many attacks on arm64, and work will eventually enable x86_64 support.

• I oversee some security audits conducted via the Open Source Technology Improvement Fund (OSTIF) when funded through the LF. We (the LF) often work with OSTIF to conduct security audits. We work with OSTIF to define the audit scope, and then OSTIF runs a bidding process where qualified security audit firms propose to do the work. We then work with OSTIF to select the winner (who isn’t always the cheapest — we want good work, not a box-check). OSTIF & I then oversee the process and review the final result. 
  • Note that we don’t just want to do audits, we also want to fix or mitigate any critical issues the audits identify, but the audits help us find the key problems. Subject matter experts perform the audit reports, and handling bidding is OSTIF’s primary focus, so my main contribution is usually to help ensure these reports are clear to non-experts while still being accurate. Experts sometimes forget to explain their context and jargon, and it’s sometimes hard to fix that (you must know the terminology & technology to explain it).
  • This work included two security audits related to the Linux kernel, one for signing and key management policies and the other for vulnerability reporting and remediation. 
  • I’ve also overseen audits of the exposure notification applications COVID Shield and COVID Green: 
  • It’s not part of my oversight of OSTIF on behalf of the LF, but I also informally talk with OSTIF about other OSS they’re auditing (such as flux2, lodash, jackson-core, jackson-databind, httpcomponents-core, httpcomponents-client, laravel, and slf4j). A little coordination and advice-sharing among experts can make everything better.

The future is hard to predict, but we anticipate that we will be doing more. In late July, the OpenSSF Technical Advisory Council (TAC) recommended approving funding for a security audit of (part of) Symfony, a widely-used web framework. The OpenSSF Governing Board (GB) approved this on 2021-08-05 and I expect OSTIF will soon take bids on it.

The OpenSSF is also taking steps to raise more money via membership dues (this was delayed due to COVID; starting a new foundation is harder during a pandemic). Once the OpenSSF has more money, we expect they’ll be funding a lot more work to identify critical projects, do security audits, fix problems, and improve or create projects to enhance OSS security. The future looks bright.

Please remember that this is only a small part of ongoing work to improve OSS security. Almost all LF projects need to be secure, so most foundations’ projects include security efforts not listed here. As noted earlier, most development work is done by volunteers or by non-LF organizations directly paying people to do the work (typically employees). 

The OpenSSF has several working groups and many projects where people are working together to improve OSS security. These include free courses on how to develop secure software and the CII Best Practices badge project. We (at the LF) also have many other projects working to improve OSS security. For example, sigstore is making cryptographic signatures much easier; sigstore’s “cosign” tool just released its version 1.0. Many organizations have recently become interested in software bill-of-materials (SBOMs), and we’ve been working on SBOMs for a long time.

If you or your organization would like to fund focused work on improving OSS security, please reach out! You can contribute to the OpenSSF (in general or as a directed fund); just contact them (e.g., Microsoft contributed to OpenSSF in December 2020). If you’d prefer, you can create a grant directly with the Linux Foundation itself — just email me at <dwheeler@linuxfoundation.org> if you have questions. For smaller amounts, say to fund a specific project, you can also consider using the LFX crowdfunding tools to fund or request funding. Many people & organizations struggle to pay individual OSS developers because of the need to handle taxes and oversight. If that’s your concern, talk to us. The LF has experience & processes to do all that, letting experts focus on getting the work done.

My sincere thanks to all the performers for their important work and to all the funders for their confidence in us!

About the author: David A. Wheeler is Director of Open Source Supply Chain Security for The Linux Foundation.