The Linux Foundation | 13 November 2023
.png?width=2733&height=1536&name=HPSFLogo%20(1).png)
Foundation aims to build, promote, and advance a portable core software stack for high performance computing.
SAN FRANCISCO – NOVEMBER 13, 2023 – Today, the Linux Foundation, the nonprofit organization enabling mass innovation through open source, announced the intention to form the High Performance Software Foundation (HPSF). Through a series of technical projects, HPSF aims to build, promote, and advance a portable software stack for high performance computing (HPC) by increasing adoption, lowering barriers to contribution, and supporting development efforts.
As use of HPC becomes ubiquitous in scientific computing and digital engineering, and AI use cases multiply, more and more data centers deploy GPUs and other compute accelerators. HPSF intends to leverage investments made by the United States Department of Energy's (DOE) Exascale Computing Project (ECP), the EuroHPC Joint Undertaking, and other international projects in accelerated HPC to exploit the performance of this diversifying set of architectures. As an umbrella project under the Linux Foundation, HPSF intends to provide a neutral space for pivotal projects in the high performance software ecosystem, enabling industry, academia, and government entities to collaborate together on the scientific software stack.
HPSF already benefits from strong support across the HPC landscape, including leading companies and organizations like Amazon Web Services, Argonne National Laboratory, CEA, CIQ, Hewlett Packard Enterprise, Intel, Kitware, Lawrence Berkeley National Laboratory, Lawrence Livermore National Laboratory, Los Alamos National Laboratory, NVIDIA, Oak Ridge National Laboratory, Sandia National Laboratory, and the University of Oregon. Drawing from supporting organizations and members of the community, HPSF will set up a technical advisory committee (TAC) to manage working groups tackling a variety of HPC topics, and will follow a governance model based on the Cloud Native Computing Foundation (CNCF).
The HPSF is launching with the following initial open source technical projects:
- Spack: the HPC package manager
- Kokkos: a performance-portable programming model for writing modern C++ applications in a hardware-agnostic way.
- AMReX: a performance-portable software framework designed to accelerate solving partial differential equations on block-structured, adaptively refined meshes.
- WarpX: a performance-portable Particle-in-Cell code with advanced algorithms that won the 2022 Gordon Bell Prize
- Trilinos: a collection of reusable scientific software libraries, known in particular for linear, non-linear, and transient solvers, as well as optimization and uncertainty quantification.
- Apptainer: a container system and image format specifically designed for secure high-performance computing.
- VTK-m: a toolkit of scientific visualization algorithms for accelerator architectures.
- HPCToolkit: performance measurement and analysis tools for computers ranging from laptops to the world’s largest GPU-accelerated supercomputers.
- E4S: the Extreme-scale Scientific Software Stack
- Charliecloud: HPC-tailored, lightweight, fully unprivileged container implementation.
HPSF aims to make life easier for high performance software developers through a number of focused initiatives, including:
- Continuous Integration resources tailored for HPC projects
- Continuously built, turnkey software stacks
- Architecture support
- Performance regression testing and benchmarking
HPSF representatives will be attending ACM/IEEE Supercomputing Conference (SC23) on Monday, November 13 for a kickoff presentation at 8:00pm. The presentation will happen at the DOE booth on the show floor during the opening Gala, with words from HPSF founders and supporters. HPSF representatives will also be available to talk to prospective projects and members.
The HPSF welcomes organizations from across the HPC ecosystem to get involved and help drive innovation in open source HPC solutions. To learn more about the HPSF, including how to get involved and join as a member, please visit http://hpsf.io/.
###
Quotes from Supporting Organizations
“Amazon Web Services is a significant contributor to open source communities. Spack has already demonstrated significant value for our customers through our integration with AWS ParallelCluster. We’re excited to see how this foundation can unlock further innovation and accelerate development for the broader HPC community, helping scientists and engineers solve the world’s hardest problems.”
– Ian Colle, GM, Advanced Computing and Simulation, at AWS
“Argonne has a long history of developing open-source software that is widely used by the HPC community. HPSF will provide a collaborative environment and access to resources that will enable the community to collectively develop the software needed to drive current and future generation HPC systems, and Argonne is honored to join as a founding member.”
– Rick Stevens, Associate Laboratory Director for Computing, Environment and Life Sciences at Argonne National Laboratory.
"The CEA is proud to be a significant contributor to this shared software ecosystem, which serves as a necessary catalyst for harnessing the raw power of Exascale hardware. Our belief in HPSF is underscored by our numerous free and open-source developments in the High-Performance Computing space, and especially with the recent launch of the CExA project. We are excited about the momentum that is building to ensure the sustainability of these efforts through an open-governance model, which fosters international collaboration, and assures the necessary developments for High-Performance Computing communities worldwide."
– Jacques-Charles Lafoucrière, HPC Programme Director at CEA
"The High Performance Software Foundation (HPSF) is a tremendous step in furthering both collaboration and innovation throughout all of Performance Intensive Computing and HPC. For this reason, it is with great excitement that I announce, on behalf of the Apptainer project and community, our intention to join the HPSF and directly support a broader community of performance-intensive computing needs."
– Gregory Kurtzer, CEO of CIQ and founder of open source projects like CentOS, Rocky Linux, Warewulf, and Apptainer/Singularity.
“The high performance computing community has unique requirements that require a flexible
and open environment to tackle the world’s biggest challenges. We look forward to playing a leadership role and are committed to actively participating in HPSF. Lowering the barrier to entry in HPC and broadening access to supercomputers through a vibrant open source community will accelerate innovation and future breakthroughs.”
– Trish Damkroger, Senior Vice President and Chief Product Officer, HPC, AI & Labs at Hewlett Packard Enterprise.
“Open standards are vital to accelerate innovation in the computing ecosystem. Intel is pleased to be a founding member of HPSF and we look forward to collaboration across academic, government, and corporate partners in pursuit of an open approach to advance HPC solutions.”
– Jeff McVeigh, Corporate VP and GM of the Software Engineering Group at Intel Corporation.
"Kitware is proud to be a member of the High-Performance Software Foundation. High-performance computing is in our DNA. We have been involved in advancing scientific software alongside our DOE partners for over two decades, from developing large-scale data visualization capabilities to high-performance IO and advanced software infrastructure. Kitware has been a leader and partner in significant efforts such as ParaView, VTK-m, ADIOS, and Spack, and looks forward to working with the HPSF to support and grow the scientific software ecosystem."
– Berk Geveci, Ph.D., Senior Director of Scientific Computing at Kitware.
"The formation of HPSF is an important step for the stewardship of software for large-scale systems. With a focus on advancing the capabilities of high performance computing (HPC) facilities across government, industry and academia, this partnership will benefit everyone's investments towards a common production environment for the broader HPC ecosystem. We take pride in the inclusion of LLNL-developed software among its inaugural initiatives, and eagerly anticipate active participation with the HPC community to refine a shared software stack and bolster the security of the HPC supply chain.”
– Bronis de Supinski, Chief Technology Officer for Livermore Computing at Lawrence Livermore National Laboratory (LLNL).
"We’re calling on potential members and projects to be part of this transformative journey. Your involvement can be a game-changer for the entire industry. As we’ve seen with initiatives like CNCF, collaborative efforts in open source can set new standards and directions for the technological world."
– Jim Zemlin, Executive Director at the Linux Foundation.
“Los Alamos National Laboratory is proud and excited to be a founding member of the High
Performance Software Foundation. The establishment of HPSF fulfills a longstanding need of the high performance computing community by providing a focal point for the development, curation and distribution of the software that powers scientific discovery. The Laboratory looks forward to working with our HPSF partners and the broader HPC community to drive innovation in HPC.”
– Tim Randles, Los Alamos’ Advanced Simulation and Computing Computational Systems and Software Environment Program Manager.
“Open source collaboration in a strong HPC developer ecosystem advances scientific computing and is made more important by the rapid evolution of breakthrough technologies such as generative AI, powered by a new class of supercomputing. NVIDIA joins a founding member of HPSF to help build a sustainable community of innovators that addresses some of humanity’s toughest challenges.”
– Tim Costa, Director HPC and Quantum at NVIDIA
“ORNL’s participation in the High Performance Software Foundation will help us to fulfill DOE’s mission by better enabling collaboration with our partners to develop the HPC-ready applications that have become critical in the quest for scientific breakthroughs. We are honored to have ORNL as a founding member of HPSF and bring to bear our laboratory’s depth of experience in scientific software development and high performance computing. This opportunity more effectively drives the future development of HPC applications aimed at solving the most pressing issues of our time.”
– Shaun Gleason, Interim Associate Laboratory Director for Computing and Computational Sciences, Oak Ridge National Laboratory (ORNL).
“The US Department of Energy (DOE) National Laboratory complex has led the way, for many decades, in the development, deployment and application of advanced algorithms and software on the world’s leading high performance computing systems for the express purposes of tackling problems of national and worldwide importance – from fundamental science discovery to energy and economic security and frankly overall quality of life. The more recent DOE Exascale Computing Project (ECP) is living proof of that continued leadership, witnessed in part by ECP’s Extreme Scale Scientific Software Stack (E4S) which continues to evolve and be released regularly at e4s.io. Sandia National Laboratories is just one of dozens of institutions to have proudly contributed to E4S as part of ECP and now, moving forward, is committed to contributing to and evolving HPSF. Without endeavors such as HPSF, the innovations and advances of a truly talented workforce at Sandia and the DOE overall would not be accessible or used and useful to the many thousands of developers and users who will reap scientific and engineering benefits for decades to come.”
– Doug Kothe, Associate Laboratories Director (ALD) for Advanced Science and Technology and Chief Research Officer, Sandia National Laboratories.
“As founding members, we welcome the partnership with HPSF to help create an ecosystem of HPC tools and libraries that will have a lasting and sustainable global impact on HPC. We hope to contribute tools and techniques for performance engineering and a mature software stack (E4S) to help lower the barriers for adopting HPC solutions.”
– Prof. Sameer Shende, Research Professor and Director, Performance Research Laboratory at the University of Oregon.
Quotes from Projects
"Joining the new High Performance Software Foundation is an exciting opportunity for the Lawrence Berkeley National Laboratory-led AMReX and WarpX projects, and will help sustain them, manage their growth, and embrace their international community of developers and users from laboratories, academia, and industry."
– John Bell, Project Lead for AMReX.
“I’m excited about this new collaboration, which will not only further broaden Charliecloud’s
availability and usefulness to all as a public good, but also strengthen the project’s robustness
and sustainability.”
– Reid Priedhorsky, Charliecloud Project Lead in Los Alamos’ High
Performance Computing division.
“The Exascale Computing Project focused its software libraries and tools efforts to strategically expand the cost and benefit sharing of DOE’s open-source HPC software investments. E4S is one outcome of that effort, providing a portable GPU-enabled collection of more than 115 products available with commercial support. We are excited to have E4S be part of HPSF, enabling further expansion of DOE’s abilities to engage with the HPC community.”
– Michael Heroux, Senior Scientist at Sandia National Labs, Director of ECP Software, and PI of the post-ECP project called PESO.
“The HPCToolkit team is pleased to have HPCToolkit included as a High-Performance Software Foundation (HPSF) project. Through participation in the HPSF, we look forward to expanding the user community for HPCToolkit, increasing its network of contributors, and improving the long-term sustainability of HPCToolkit.” – John Mellor-Crummey, a Professor at Rice University and the HPCToolkit Project Lead.
“Intel has been working with Spack, E4S, and the broader community ensuring oneAPI and Intel’s tools enable customers to spend time on science, not configuring software. We are proud to be a founding member of HPSF and support its mission to enable portable software stacks and open source development.” – Robert Cohn, Intel Senior Principal Engineer, Developer Software Engineering.
"As a dedicated contributor to Spack for almost ten years, our team has made significant improvements to the CI/CD infrastructure and helped adapt Spack to work with various operating systems. We look forward to continuing to contribute to the Spack project and advancing high-performance computing through the HPSF."
– Aashish Chaudhary, Assistant Director of Business Development at Kitware.
“The Trilinos Project is delighted to be part of the HPSF foundational product collection. From its inception more than 20 years ago, Trilinos has been a community of developers committed to producing a collection of open-source scientific libraries and tools that pushes the boundaries of capabilities on HPC systems for a diverse set of scientific and engineering problems. We look forward to this new and exciting way to collaborate with the HPC community.”
– Michael Heroux, Senior Scientist at Sandia National Laboratories and founder of the Trilinos Project.
"The VTK-m team is eager to participate in the High-Performance Software Foundation. Our success depends on interactions with the rest of the HPC software community, and we are eager to leverage the HPSF to continue the relationships built during ECP and help foster relationships with new collaborators and customers."
– Ken Moreland, VTK-m
"Joining the HPSF will help the WarpX user community provide simulation support to a range of international scientific experiments."
– Jean-Luc Vay, Project Lead for WarpX.
###
About The Linux Foundation
The Linux Foundation is the world’s leading home for collaboration on open source software, hardware, standards, and data. Linux Foundation projects are critical to the world’s infrastructure including Linux, Kubernetes, Node.js, ONAP, PyTorch, RISC-V, SPDX, OpenChain, and more. The Linux Foundation focuses on leveraging best practices and addressing the needs of contributors, users, and solution providers to create sustainable models for open collaboration. For more information, please visit us at linuxfoundation.org. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see its trademark usage page: www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.
Media Contact
Noah Lehman
The Linux Foundation