opencontainers/go-digest

Common digest package used across the container ecosystem

go-digest

Common digest package used across the container ecosystem.

Please see the godoc for more information.

What is a digest?

A digest is just a hash.

The most common use case for a digest is to create a content identifier for use in Content Addressable Storage systems:

id := digest.FromBytes([]byte("my content"))

In the example above, the id can be used to uniquely identify the byte slice "my content". This allows two disparate applications to agree on a verifiable identifier without having to trust one another.

An identifying digest can be verified, as follows:

if id != digest.FromBytes([]byte("my content")) {
  return errors.New("the content has changed!")
}

A Verifier type can be used to handle cases where an io.Reader makes more sense:

rd := getContent()
verifier := id.Verifier()
io.Copy(verifier, rd)

if !verifier.Verified() {
  return errors.New("the content has changed!")
}

Using Merkle DAGs, this can power a rich, safe, content distribution system.

Usage

While the godoc is considered the best resource, a few important items need to be called out when using this package.

1. Make sure to import the hash implementations into your application or the package will panic. You should have something like the following in the main (or other entrypoint) of your application:

   import (
       _ "crypto/sha256"
       _ "crypto/sha512"
   )

   This may seem inconvenient but it allows you replace the hash implementations with others, such as https://github.com/stevvooe/resumable.

2. Even though digest.Digest may be assemblable as a string, always verify your input with digest.Parse or use Digest.Validate when accepting untrusted input. While there are measures to avoid common problems, this will ensure you have valid digests in the rest of your application.

3. While alternative encodings of hash values (digests) are possible (for example, base64), this package deals exclusively with hex-encoded digests.

Stability

The Go API, at this stage, is considered stable, unless otherwise noted.

As always, before using a package export, read the godoc.

Contributing

This package is considered fairly complete. It has been in production in thousands (millions?) of deployments and is fairly battle-hardened. New additions will be met with skepticism. If you think there is a missing feature, please file a bug clearly describing the problem and the alternatives you tried before submitting a PR.

Code of Conduct

Participation in the OpenContainers community is governed by OpenContainer's Code of Conduct.

Security

If you find an issue, please follow the security protocol to report it.

Copyright and license

Copyright © 2019, 2020 OCI Contributors Copyright © 2016 Docker, Inc. All rights reserved, except as follows. Code is released under the Apache 2.0 license. This README.md file and the CONTRIBUTING.md file are licensed under the Creative Commons Attribution 4.0 International License under the terms and conditions set forth in the file LICENSE.docs. You may obtain a duplicate copy of the same license, titled CC BY-SA 4.0, at http://creativecommons.org/licenses/by-sa/4.0/.

go-digest's Issues

Require lowercase hex hashes?

This repo currently allows case-insensitive hex, but opencontainers/image-spec#292 argued for lowercase hex to:

• Make comparison easier
• Make it easier to serve blobs out of a case-insensitive filesystem store.
• Avoid having two otherwise-identical descriptor (or descriptor-containing) blobs with different hashes because they picked different hex-casing.

Do we want to adjust DigestRegexp to make it compatible with the current image-spec requirements? Or are there good reasons to allow mixed-case hex, in which case, should image-spec be adjusted to allow them again?

Update @aaronlehmann contact

It looks like @aaronlehmann is a maintainer but the email is bad.

Are they still interested in being a maintainer and do they have a new email to use?

cc: @dmcgowan @opencontainers/go-digest-maintainers

consider using sha256-simd

Considering the impressive perf gains, thoughts about pulling this in?

https://github.com/minio/sha256-simd

Remove ParseDigest, NewDigestVerifier and DigestSha256EmptyTar

These are all vestigal from the source package.

Clarify inconsistency with standard sha256sum utility.

Hi all,

maybe is me missing something but if I run:

package main

import (
	_ "crypto/sha256"
	"fmt"
	"github.com/opencontainers/go-digest"
)

func main() {
	fmt.Println(digest.FromBytes([]byte("A")))
	// "sha256:559aead08264d5795d3909718cdd05abd49572e84fe55590eef31a88a08fdffd"
}

I get as input:
sha256:559aead08264d5795d3909718cdd05abd49572e84fe55590eef31a88a08fdffd

However if I run:

$ echo A | sha256sum 
06f961b802bc46ee168555f066d28f4f0e9afdf3f88174c1ee6f9de004fc30a0  -

Is this inconsistency wanted?

Unusual location for license defeats wwhrd and other tools

The Apache license for this project is in LICENSE.code, which means that wwhrd (and I suppose any other tool using go-license) cannot find it and so reports an unknown license (which is a failure). I've added an exception to my project but thought it worth raising since I don't like exceptions.

How would we feel about renaming LICENSE.code to just LICENSE (and leaving LICENSE.docs alone of course)? My argument for giving the code the "privileged" position is that automated tooling is going to primarily be interested in code (e.g. vendoring) while docs are not typically vendored or subject to automatic inclusion in quite the same way. Even more LICENSE.docs specifically only covers README.md and CONTRIBUTING.md which really aren't going to be vendored I think.

I can make the PR if we think this is a good idea.

As an alternative a symlink LICENSE → LICENSE.code would work too (I guess, I've not actually tried it)

There is an issue on go-license at ryanuber/go-license#16 but its from 2015 so I figured I would start here first.

Clarify MAINTAINERS post OCI move

Who will be the official list of maintainers @stevvooe ?

Failed while installing

I was trying to install using the glide. Got this error:

Fetching github.com/opencontainers/go-digest [INFO] --> Setting version for github.com/opencontainers/go-digest to a6d0ee40d4207ea02364bd3b9e8e77b9159ba1eb. [ERROR] Failed to retrieve a list of dependencies: Error resolving imports

Using multihash for digests

The descriptors spec mentions multihash+base58 as an example of an algorithm that is not registered. I'm interested to use multihashes in application/vnd.oci.image.manifest.v1+json, and to pull tags by multihash digest from distribution.

Multihashes encode the algorithm in the digest, so multihash+base58 itself isn't sufficient to generate digests (FromReader, FromBytes, FromString) or to return the size of the hash. Perhaps multihash+base58:encoded are transformed to its respective algorithm:encoded for validation / verifying purposes.

I'm curious to hear thoughts about supporting multihash (in or out of package) in a way that projects like containerd and distribution can consume them.

New release version

It's been a few years since the 1.0.0 release with many changes since. Is it possible to cut a stable release? I like the addition of the the Digester interface.

MAINTAINERS update needed

@jonboulle email incorrect in MAINTAINERS, see https://github.com/opencontainers/image-spec/blob/master/MAINTAINERS for current email

Administrative Items for move

Now that this repository has been moved to opencontainers, we'll need to update several administrative items. If there is anything missing, please edit this description or comment and I'll add it here. Once an item has a PR open, we can add it here and check it off when merged.

• Update travis CI build configuration (nothing to do here)
• Update project urls (#22)
• MAINTAINERS
  • Update maintainers with OCI project format (#19)
  • Setup maintainers group (#15)
  • PullApprove configuration (#16)
• Update files to have license header, per image-spec/runtime-spec (#28)
• Apply oci project template (#20)
• Update security statement to be in line with OCI procedures (#27)

Document Signed-off-by

I added one in #4, but grepping for Signed-off-by in this repo didn't give any hits. We probably want to land some boilerplate docs around it before this repo gets too far along.

LICENSE and LICENSE.doc causes go-license-detector to see only evidence of CC-BY-SA-4.0

When scanning Go module dependencies to create BOMs with tools like cyclonedx-gomod the setup of this repository/Go module with both LICENSE and LICENSE.doc causes the go-license-detector to see only evidence of CC-BY-SA-4.0. go-license-detector does not detect the presence of the Apache-2.0 license in LICENSE, or this evidence gets thrown under the bus in the process of creating the BOM by tools using go-license-detector.

Would it be possible to clean up the LICENSE and LICENSE.doc files with preferably only a single LICENSE file that tools like go-license-detector can correctly handle?

Add PullApprove config

We should adopt the PullApprove config shared by other OCI projects

Hardcoded assumptions that the hash will be encoded in hex

The docs for Algorithm (now algorithm) made it clear that the algorithm identifier was intended to cover both the hash and encoding algorithms. @stevvooe confirmed this interpretation in recent comments as well. The idea is that a future algorithm may chose a non-hex encoding like base 64.

The current implementation, on the other hand, bakes the hex encoding into key locations (e.g. in NewDigestFromBytes and Digest.Validate). I suggest:

• Defining an Encoding interface.
• Adding an Algorithm.Encoding() Encoding method.
• Adding an Algorithm.HashSize() int method.
• Updating go-descriptor to use those instead of the currently-hard-coded hex assumptions.

I've floated an implementation in #30 if folks want to see how that works out.

Alternative solutions include giving up on alternatives and just requiring all hashes to be hex-encoded. Or some other API for pushing encoding information into the Algorithm instances.

Thoughts?

Ship a release!

Currently there are no tags/releases :)

https://github.com/opencontainers/go-digest/releases

Rename master branch to main

This is listed under "Tier 1: Replace Immediately" in the Inclusive Naming Initiative:
https://inclusivenaming.org/word-lists/tier-1/#master

Cut new release that includes #38

Essentially this request is just for a new tag that includes #38. Go modules won't fetch the license file update because it isn't included in a tagged version.

Document digest specification?

There are already some docs for the very-closely-related OCI digests, but if this spec wants to keep mixed-case hashes around (#3), it's probably worth documenting the local choices independently. For example, is there a way to identify hex hashes vs. base64 hashes (counting characters and comparing with the algorithm's hash size?)? Are hex hashes allowed to be mixed case? We probably don't want to dig into those ambiguities in this issue, I'm just asking maintainers if they plan on providing a specification for the digests that this package will parse/generate.