spack/seps

Spack Enhancement Proposals (SEPs)

Getting Started   •   Config   •   Community   •   Contributing   •   Packaging Guide

Spack is a multi-platform package manager that builds and installs multiple versions and configurations of software. It works on Linux, macOS, Windows, and many supercomputers. Spack is non-destructive: installing a new version of a package does not break existing installations, so many configurations of the same package can coexist.

Spack offers a simple "spec" syntax that allows users to specify versions and configuration options. Package files are written in pure Python, and specs allow package authors to write a single script for many different builds of the same package. With Spack, you can build your software all the ways you want to.

See the Feature Overview for examples and highlights.

To install spack and your first package, make sure you have Python. Then:

$ git clone -c feature.manyFiles=true https://github.com/spack/spack.git
$ cd spack/bin
$ ./spack install zlib

Documentation

Full documentation is available, or run spack help or spack help --all.

For a cheat sheet on Spack syntax, run spack help --spec.

Tutorial

We maintain a hands-on tutorial. It covers basic to advanced usage, packaging, developer features, and large HPC deployments. You can do all of the exercises on your own laptop using a Docker container.

Feel free to use these materials to teach users at your organization about Spack.

Community

Spack is an open source project. Questions, discussion, and contributions are welcome. Contributions can be anything from new packages to bugfixes, documentation, or even new core features.

Resources:

• Slack workspace: spackpm.slack.com. To get an invitation, visit slack.spack.io.
• Matrix space: #spack-space:matrix.org: bridged to Slack.
• Github Discussions: for Q&A and discussions. Note the pinned discussions for announcements.
• X: @spackpm. Be sure to @mention us!
• Mailing list: groups.google.com/d/forum/spack: only for announcements. Please use other venues for discussions.

Contributing

Contributing to Spack is relatively easy. Just send us a pull request. When you send your request, make develop the destination branch on the Spack repository.

Your PR must pass Spack's unit tests and documentation tests, and must be PEP 8 compliant. We enforce these guidelines with our CI process. To run these tests locally, and for helpful tips on git, see our Contribution Guide.

Spack's develop branch has the latest contributions. Pull requests should target develop, and users who want the latest package versions, features, etc. can use develop.

Releases

For multi-user site deployments or other use cases that need very stable software installations, we recommend using Spack's stable releases.

Each Spack release series also has a corresponding branch, e.g. releases/v0.14 has 0.14.x versions of Spack, and releases/v0.13 has 0.13.x versions. We backport important bug fixes to these branches but we do not advance the package versions or make other changes that would change the way Spack concretizes dependencies within a release branch. So, you can base your Spack deployment on a release branch and git pull to get fixes, without the package churn that comes with develop.

The latest release is always available with the releases/latest tag.

See the docs on releases for more details.

Code of Conduct

Please note that Spack has a Code of Conduct. By participating in the Spack community, you agree to abide by its rules.

Authors

Many thanks go to Spack's contributors.

Spack was created by Todd Gamblin, [email protected].

Citing Spack

If you are referencing Spack in a publication, please cite the following paper:

• Todd Gamblin, Matthew P. LeGendre, Michael R. Collette, Gregory L. Lee, Adam Moody, Bronis R. de Supinski, and W. Scott Futral. The Spack Package Manager: Bringing Order to HPC Software Chaos. In Supercomputing 2015 (SC’15), Austin, Texas, November 15-20 2015. LLNL-CONF-669890.

On GitHub, you can copy this citation in APA or BibTeX format via the "Cite this repository" button. Or, see the comments in CITATION.cff for the raw BibTeX.

License

Spack is distributed under the terms of both the MIT license and the Apache License (Version 2.0). Users may choose either license, at their option.

All new contributions must be made under both the MIT and Apache-2.0 licenses.

See LICENSE-MIT, LICENSE-APACHE, COPYRIGHT, and NOTICE for details.

SPDX-License-Identifier: (Apache-2.0 OR MIT)

LLNL-CODE-811652

seps's Issues

Notes on supporting multiple build-systems using a when decorator

Below are a few notes for an alternative approach to #3 based on a class decorator. I'm posting it as an issue since there are certain semantics that need to be clarified before we can write a consistent proposal.

---

Support multiple build-systems using a when decorator

The problem

There are multiple packages that are either changing their build-system during the evolution of the project, or using different build-systems for different platforms. Spack, at the moment, provides no support to model this use case. As a result we wrote some documentation to enumerate various workarounds that people adopted - each with its own drawbacks. What we would like to have in the long term is proper support for packages that can be built using multiple build-systems.

Proposed changes

The proposal made in this SEP is to have classes that can be decorated with a when decorator similarly to what is done for methods. The basic idea is the following:

class Hdf5Base(PackageBase):
	# Common directives, attributes and methods
	depends_on('zlib')


@when('@:X.Z')
class Hdf5(AutotoolsPackage, Hdf5Base):
	depends_on('automake', type='build')

	def install(self, spec, prefix):
		pass
	
	
@when('@X.Y:')
class Hdf5(CMakePackage, Hdf5Base):
	depends_on('[email protected]:', type='build')
	
    def install(self, spec, prefix):
		pass

The same name Hdf5 is used multiple times under a decorator. The idea would be to use the class definition that "satisfies" the constraint in the when= decorator once we have a concrete spec associated with the package.

The when decorator at a class level has three different responsibilities:

1. It needs to constrain every when= parameter in every directive with its argument.
2. It needs to treat methods as if they have a when decorator on them
3. It needs to defer inheritance of the base classes to after concretization

Point 1 means that with respect to just metadata, the snippet above should be equivalent to the following:

class Hdf5(Package):
	# From the base class
	depends_on('zlib')
	# From the first decorated class
	depends_on('automake', type='build', when='@:X.Z')
	# From the second decorated class
	depends_on('[email protected]:', type='build', when='@X.Y:')

Point 2. means that methods instead should behave like:

class Hdf5(Package):
	@when('@:X.Z')
	def install(self, spec, prefix):
		# From the first decorated class
		pass
	
	@when('@X.Y:')
	def install(self, spec, prefix):
		# From the second decorated class
		pass

Finally point 3. is the more subtle and means that the class is in a sort of "undetermined" state until it is associated with a concrete spec, in which case it behaves like:

# If spec.satisfies('@:X.Z')
class Hdf5(AutotoolsPackage):
	pass

or like:

# If spec.satisfies('@X.Y:')
class Hdf5(CMakePackage):
	pass

Possible issues with the current semantics

Even though the objective here is to deal with multiple build-systems, using a class decorator is a very general mechanism that needs to have clear semantics
under all of its possible applications. Below we'll point out issues with the current idea or cases that need to be defined better.

Structure and behavior of an "undetermined" class

In the simple hdf5 example at the top it is not clear what would be the behavior for operations like:

isinstance(pkg, CMakePackage)

before a concrete spec is associated with the package. There are similar questions that can be asked with respect to the id() of the class or to the presence or absence of class attributes.

Inheritance from a when decorated class

Some packages are fork of other packages and reuse most of their implementation. These packages usually read like:

from spack.pkgkit.builtin import Hdf5

class Hdf5Fork(Hdf5):
	pass

It is not clear what would be the meaning of something like:

from spack.pkgkit.builtin import Hdf5

class Hdf5ForkBase(Hdf5):
	pass

@when('+foo')
class Hdf5Fork(Hdf5ForkBase):
	pass

@when('~foo')
class Hdf5Fork(Hdf5ForkBase):
	pass

if Hdf5 itself uses a when decorator. The problem is that we may end up with a base class that is not yet fully defined, and have on top of that another class definition that is deferred to after concretization. This makes it difficult to deal with MRO and other Python mechanism related to inheritance.

This issue may also have different variations:

from spack.pkgkit.builtin import Hdf5, Hdf5Base

class Hdf5Fork(Hdf5):
	pass

# Does this extend Hdf5 with another conditional?
@when('platform=windows')
class Hdf5Fork(Hdf5Base):
	pass

Would the above be a valid extension for an Hdf5Fork that can be used on Windows?

Multiple constraints are met

With a when decorator applied to a class we may have definitions like:

class Base(PackagesBase):
	pass

@when('+foo')
class Hdf5(Base):
	depends_on('foo')
	
	def install(self, spec, prefix)
		pass

@when('platform=darwin')
class Hdf5(Base):
	depends_on('bar')
	
	def install(self, spec, prefix)
		pass

Given the semantics outlined in the previous paragraph this means that the class above is equivalent to:

class Hdf5(Base)
	depends_on('foo', when='+foo')
	depends_on('bar', when='platform=darwin')
	
	@when('+foo')
	def install(self, spec, prefix)
		pass

	@when('platform=darwin')
	def install(self, spec, prefix)
		pass

This in turn means that if both constraints are met, the directives will be accounted for from both classes, while the install method will be from the first class only.

This behavior can be confusing to users and is complicated further if we add different base classes to the example above. Another case where the semantic is not clear is with class attributes:

class Base(PackagesBase):
	base = 0

@when('+foo')
class Hdf5(Base):
	foo = 1
	bar = 1
	
@when('platform=darwin')
class Hdf5(Base):
	foo = 2
	baz = 2

What should be the behavior of:

getattr(Hdf5, 'foo') == ?
getattr(Hdf5, 'bar') == ?
getattr(Hdf5, 'baz') == ?

in the case the final spec satisfies +foo platform=darwin? If the when decorator behaves like it does for methods, then:

getattr(Hdf5, 'foo') == 1
getattr(Hdf5, 'bar') == 1
getattr(Hdf5, 'baz') is None

and it would become unclear why eventual dependencies stemming from the second class are accounted for in the final spec. If we merge based on order we might have:

getattr(Hdf5, 'foo') == 1
getattr(Hdf5, 'bar') == 1
getattr(Hdf5, 'baz') == 2

in which case we may have an "unexpected" value for Hdf5.foo.

Selection of the build system

Since the approach is based on inheritance without changing other attributes, it's not possible to specify the build system directly. A user can try to influence the selection implicitly by specifying a build-dependency:

hdf5 ^cmake

but the flat representation of specs + clingo do not grant that cmake will be a direct dependency of hdf5 and may result in DAG which are different from what is expected (e.g. hdf5 built with autotools depending on zlib built with cmake).