The Linux Foundation | 01 June 2016
Co-authored by Dr. David A. Wheeler
Everybody loves getting badges. Fitbit badges, Stack Overflow badges, Boy Scout merit badges, and even LEED certification are just a few examples that come to mind. A recent 538 article “Even psychologists love badges” publicized the value of a badge.
unnamed.png
GitHub now has a number of specific badges for things like test coverage and dependency management, so for many developers they’re desirable. IBM has a slew of certifications for security, analytics, cloud and mobility, Watson Health and more.
Recently The Linux Foundation joined the trend with the Core Infrastructure Initiative (CII) Best Practices Badges Program.
The free, self-service Best Practices Badges Program was designed with the open source community. It provides criteria and an automated assessment tool for open source projects to demonstrate that they are following security best practices.
It’s a perfect fit for CII, which is comprised of technology companies, security experts and developers, all of whom are committed to working collaboratively to identify and fund critical open source projects in need of assistance. The badging project is an attempt to “raise all boats” in security, by encouraging projects to follow best practices for OSS development. We believe projects that follow best practices are more likely to be healthy and produce secure software.
Here’s more background on the program and some of the questions we’ve recently been asked.
Q: Why badges?
A: We believe badges encourage projects to follow best practices, to hopefully produce better results. The badges will:
-
Help new projects learn what those practices are (think training in disguise).
-
Help users know which projects are following best practices (so users can prefer such projects).
-
Act as a signal to users. If a project has achieved a number of badges, it will inspire a certain level of confidence among users that the project is being actively maintained and is more likely to consistently produce good results.
Q: Who gets a badge? Is this for individuals, projects, sites?
A: The CII best practices badge is for a project, not for an individual. When you’re selecting OSS, you’re picking the project, knowing that some of the project members may change over time.
Q: Can you tell us a little about the “BadgeApp” web application that implements this?
A: “BadgeApp” is a simple web application that implements the criteria (fill in form). It’s OSS, with an MIT license. All the required libraries are OSS & legal to add; we check this using license_finder.
Our overall approach is that we proactively counter mistakes. Mistakes happen, so we use a variety of tools, an automated test suite, and other processes to counter them. For example, we use rubocop to lint our Ruby, and ESLint to lint our Javascript. The test suite currently has 94% statement coverage with over 3000 checked assertions, and our project has a rule that the test suite must be at least 90%.
Please contribute! See our CONTRIBUTING.md file for more.
Q: What projects already have a badge?
A: Examples of OSS projects that have achieved the badge include the Linux kernel, Curl, GitLab, OpenBlox, OpenSSL, Node.js, and Zephyr. We specifically reached out to both smaller projects, like curl, and bigger projects, like the Linux kernel, to make sure that our criteria made sense for many different kinds of projects. It’s designed to handle both front-end and infrastructure projects.
Q: Can you tell us more about the badging process itself? What does it cost?
A: It doesn’t cost anything to get a badge. Filling out the web form takes about an hour. It’s primarily self-assertion, and the advantage of self-assertion systems is that they can scale up.
There are known problems with self-assertion, and we try to counter their problems. For example, we do perform some automation, and, in some cases, the automation will override unjustified claims. Most importantly, the project’s answers and justifications are completely public, so if someone gives false information, we can fix it and thus revoke the badge.
Q: How were the criteria created?
A: We developed the criteria, and the web application that implements them, as an open source software project. The application is under the MIT license; the criteria are dual-licensed under MIT and CC-BY version 3.0 or later. David A. Wheeler is the project lead, but the work is based on comments from many people.
The criteria were primarily based on reviewing a lot of existing documents about what OSS projects should do. A good example is Karl Fogel’s book Producing Open Source Software, which has lots of good advice. We also preferred to add criteria if we could find at least one project that didn’t follow it. After all, if everyone does it without exception, it’d be a waste of time to ask if your project does it too. We also worked to make sure that our own web application would get its own badge, which helped steer us away from impractical criteria.
Q: Does the project have to be on GitHub?
A: We intentionally don’t require or forbid any particular services or programming languages. A lot of people use GitHub, and in those cases we fill in some of the form based on data we extract from GitHub, but you do not have to use GitHub.
Q: What does my project need to do to get a badge?
A: Currently there are 66 criteria, and each criterion is in one of three categories: MUST, SHOULD, or SUGGESTED. The MUST (including MUST NOTs) are required, and 42/66 criteria are MUST. The SHOULD (NOT) are sometimes valid to not do; 10/66 criteria are SHOULDs. The SUGGESTED criteria have common valid reasons to do them, but we want projects to at least consider them. 14/66 are SUGGESTED. People don’t like admitting that they don’t do something, so we think that having criteria listed as SUGGESTED are helpful because they’ll nudge people to do them.
To earn a badge, all MUST criteria must be met, all SHOULD criteria must be met OR be unmet with justification, and all SUGGESTED criteria must be explicitly marked as met OR unmet (since we want projects to at least actively consider them). You can include justification text in markdown format with almost every criterion. In a few cases, we require URLs in the justification, so that people can learn more about how the project meets the criteria.
We gamify this – as you fill in the form you can see a progress bar go from 0% to 100%. When you get to 100%, you’ve passed!
Q: What are the criteria?
A: We’ve grouped the criteria into 6 groups: basics, change control, reporting, quality, security, and analysis. Each group has a tab in the form. Here are a few examples of the criteria:
Basics
The software MUST be released as FLOSS. [floss_license]
Change Control
The project MUST have a version-controlled source repository that is publicly readable and has a URL.
Reporting
The project MUST publish the process for reporting vulnerabilities on the project site.
Quality
If the software requires building for use, the project MUST provide a working build system that can automatically rebuild the software from source code.
The project MUST have at least one automated test suite that is publicly released as FLOSS (this test suite may be maintained as a separate FLOSS project).
Security
At least one of the primary developers MUST know of common kinds of errors that lead to vulnerabilities in this kind of software, as well as at least one method to counter or mitigate each of them.
Analysis
At least one static code analysis tool MUST be applied to any proposed major production release of the software before its release, if there is at least one FLOSS tool that implements this criterion in the selected language.
Q: Are these criteria set in stone for all time?
A: The badge criteria were created as an open source process, and we expect that the list will evolve over time to include new aspects of software development. The criteria themselves are hosted on GitHub, and we actively encourage the security community to get involved in developing them. We expect that over time some of the criteria marked as SUGGESTED will become SHOULD, some SHOULDs will become MUSTs, and new criteria will be added.
Q: What is the benefit to a project for filling out the form? Is this just a paperwork exercise? Does it add any real value?
A: It’s not just a paperwork exercise; it adds value.
Project members want their project to produce good results. Following best practices can help you produce good results – but how do you know that you’re following best practices? When you’re busy getting specific tasks done, it’s easy to forget to do important things, especially if you don’t have a list to check against.
The process of filling out the form can help your project see if you’re following best practices, or forgetting to do something. We’ve had several cases during our alpha stage where projects tried to fill out the form, found they were missing something, and went back to change their project. For example, one project didn’t explain how to report vulnerabilities – but they agreed that they should. So either a project finds out that they’re following best practices – and know that they are – or will realize they’re missing something important, so the project can then fix it.
There’s also a benefit to potential users. Users want to use projects that are going to produce good work and be around for a while. Users can use badges like this “best practices” badge to help them separate well-run projects from poorly-run projects.
Q: Does the Best Practices Badge compete with existing maturity models or anything else that already exists?
A: The Best Practices Badge is the first program specifically focused on criteria for an individual OSS project. It is free and extremely easy to apply for, in part because it uses an interactive web application that tries to automatically fill in information where it can.
This is much different than maturity models, which tend to be focused on activities done by entire companies.
The BSIMM (pronounced “bee simm”) is short for Building Security In Maturity Model. It is targeted at companies, typically large ones, and not on OSS projects.
OpenSAMM, or just SAMM, is the Software Assurance Maturity Model. Like BSIMM, they’re really focusing on organizations, not specific OSS projects, and they’re focused on identifying activities that would occur within those organizations.
Q: Does the project’s websites have to support HTTPS?
A: Yes, projects have to support HTTPS to get a badge. Our criterion sites_https now says: “The project sites (website, repository, and download URLs) MUST support HTTPS using TLS. You can get free certificates from Let’s Encrypt.” HTTPS doesn’t counter all attacks, of course, but it counters a lot of them quickly, so it’s worth requiring. At one time HTTPS imposed a significant performance cost, but modern CPUs and algorithms have basically eliminated that. It’s time to use HTTPS and TLS.
Q: How do I get started or get involved?
A: If you’re involved in an OSS project, please go get your badge from here:
https://bestpractices.coreinfrastructure.org/
If you want to help improve the criteria or application, you can see our GitHub repo:
https://github.com/linuxfoundation/cii-best-practices-badge
We expect that there will need to be improvements over time, and we’d love your help.
But again, the key is, if you’re involved in an OSS project, please go get your badge:
https://bestpractices.coreinfrastructure.org/
Dr. David A. Wheeler is an expert on developing secure software and on open source software. His works include Software Inspection: An Industry Best Practice, Ada 95: The Lovelace Tutorial, Secure Programming for Linux and Unix HOWTO, Fully Countering Trusting Trust through Diverse Double-Compiling (DDC), Why Open Source Software / Free Software (OSS/FS)? Look at the Numbers!, and How to Evaluate OSS/FS Programs. Dr. Wheeler has a PhD in Information Technology, a Master’s in Computer Science, a certificate in Information Security, and a B.S. in Electronics Engineering, all from George Mason University (GMU). He lives in Northern Virginia.
Emily Ratliff is Sr. Director of Infrastructure Security at The Linux Foundation, where she sets the direction for all CII endeavors, including managing membership growth, grant proposals and funding, and CII tools and services. She brings a wealth of Linux, systems and cloud security experience, and has contributed to the first Common Criteria evaluation of Linux, gaining an in-depth understanding of the risk involved when adding an open source package to a system. She has worked with open standards groups, including the Trusted Computing Group and GlobalPlatform, and has been a Certified Information Systems Security Professional since 2004.