Mapping the Chaos of HPC Dependencies

Summary: High Performance Computing~(HPC) software stacks have become complex, with some application dependencies running into the hundreds. Packaging, distributing, and administering software stacks of this magnitude is a complex undertaking anywhere. HPC systems deal with esoteric compilers, hardware, and a host of uncommon combinations. In this article, we explore the mechanisms available for packaging software to find their own dependencies in the context of a software distribution taxonomy, and discuss their advantages and disadvantages. We discuss workarounds for some common issues caused by using these compound stacks and present Shrinkwrap: a solution for producing binaries that directly load their dependencies from specific locations and in a specific order. In addition to simplifying the use of binaries, this approach also speeds up loading by up to 7x for a large dynamically linked MPI application in our review.