palantir/pkg

A collection of stand-alone Go packages

pkg

pkg is a collection of Go packages that provide various different functionality. The packages in this project are generally independent and strive to depend on as few external packages as possible. Many of the packages in the project are also independent from each other.

An overview of some of the packages is included below. Refer to the package comments for individual packages for more information.

cli

cli provides a framework for creating CLI applications. It provides support for commands, subcommands, flags, before and after hooks, documentation, deprecation, command-line completion and other functionality.

matcher

matcher allows files to be matched based on their name or path. Supports composing and combining matchers and provides data structures that can be used as configuration to specify include and exclude rules.

objmatcher

objmatcher provides the ability to match objects based on criteria and returns a descriptive error when an object does not match its expectation. When used in combination with maps, makes it easy to perform complex matching on maps in a declarative manner (for example, requiring that some map entries match an expectation exactly while others should match a particular regular expression).

pkgpath

pkgpath provides functions for getting Go package paths. Provides functions for getting the paths to all of the packages rooted in a directory and for converting between different representations of package paths including relative, GOPATH-relative and absolute. Depends on matcher.

specdir

specdir provides the ability to define specifications for directory layouts, verify that existing directories match the specification and create new directory structures based on a specification.

License

This project is made available under the BSD 3-Clause License.

pkg's Issues

Check for nil return from custom app.ContextConfig function

There exists a very specific NPE risk where a consumer of github.com/palantir/pkg/cli both implements a custom app.ContextConfig function, and has it return nil.

This call to context.WithCancel is not nil-safe, so we should probably wrap the call in a nil check.

https://play.golang.org/p/T0OOtNMPzO

app := cli.NewApp()
app.Name = "buggy-cli"

app.ContextConfig = func(ctx context.Context) context.Context {
	return nil
}

app.Run(os.Args) // This will NPE

cli: poor messaging on misconfigured duration flag

If I have a cli with a duration flag that is not required and has no default, it will always panic with the message:

panic: time: invalid duration

this happens whether or not the user provides a value on the command line. Turns out this panic arises in the default-parsing code:

pkg/cli/flag/duration_flag.go

Lines 51 to 54 in b9b4547

	func (f DurationFlag) Default() interface{} {
	dur, err := f.Parse(f.DefaultStr())
	if err != nil {
	panic(err)

We should have better messaging to ensure developers aren't confused when the panic happens.

cli: --help should only show [subcommand] if visible subcommands exist

1. Define a CLI without any visible subcommands (assign the app's Command directly)
2. Run app --help

Expected

Help's "Usage" section should not reference subcommands

Actual

USAGE:
   app [global flags] command [command flags]

Reason

Right now, the CLI's Run function adds the _completion subcommand, and the template that determines whether subcommands are shown incorrectly considers hidden commands.

go-metrics dependency requires dep override

In #142 we switched to a fork of go-metrics; in most cases this requires consuming projects to include a dep override of the form

[[override]]
  name = "github.com/rcrowley/go-metrics"
  source = "github.com/ashrayjain/go-metrics"
  branch = "aj/rescale-on-reads"

Ideally we would not need overrides in consuming projects

metrics registry doesn't remove metric ids on unregister

The root metric registry doesn't delete metric ids from idToMetricWithTags on Unregister()

See https://github.com/palantir/pkg/blob/master/metrics/registry.go#L255

metrics: deduplicate tags on insertion into registry

This library currently uses a slice to store tags [0]. However, Witchcraft converts the tags to a map when emitting them [1]. Even though the lossy behavior of converting the tag slice to a map is well-documented [2], it makes it difficult to debug issues involving multiple instances of the same tag key getting added to the metrics registry since the internal state diverges from what is logged. Since the Witchcraft spec defines a tag as a map<string, string> [3], let's move the deduplication of tags earlier in the pipeline such that it happens on insertion into the registry.

[0] https://github.com/palantir/pkg/blob/master/metrics/tag.go#L36
[1] https://github.com/palantir/witchcraft-go-server/blob/develop/witchcraft/server_metrics.go#L42
[2] https://github.com/palantir/pkg/blob/master/metrics/tag.go#L46-L49
[3] https://github.com/palantir/witchcraft-api/blob/master/witchcraft-logging-api/src/main/conjure/witchcraft-logging-api.yml#L369-L370

retry: Default MaxBackoff is unintuitive

Issue

I would expect the following

func Test(t *testing.T) {
	err:=retry.Do(
		context.Background(),
		func() error {
			fmt.Println(time.Now())
			return errors.New("some errors")
		},
		retry.WithInitialBackoff(10*time.Second), retry.WithMultiplier(1), retry.WithMaxAttempts(5),
	)
	require.NoError(t,err)
}

To output one log line every 10 seconds. However what this actually outputs is

2019-01-16 14:51:16.025567248 -0800 PST m=+0.011095106
2019-01-16 14:51:18.092852159 -0800 PST m=+2.078377531
2019-01-16 14:51:20.357685855 -0800 PST m=+4.343208502
2019-01-16 14:51:22.459165577 -0800 PST m=+6.444685696
2019-01-16 14:51:24.425426817 -0800 PST m=+8.410944570

The reason for this output is the default for MaxBackoff is as follows:

// WithMaxBackoff sets upper limit on backoff duration.
//
// Max backoff of 0 indicates no limit.
//
// If max backoff option is not used, then default value of 2 seconds is used.

While the documentation is good, this caused us to write bugs that affected us in production as I think it's unintuitive.

Proposal

Set the default value of max backoff to 0 instead of 2 seconds, so that the max backoff will be unlimited.

I realize this is a breaking change, but I would believe that most people using this library would expect it to work this way

metrics Timer should use microseconds

Currently, the metrics.Registry interface has a Timer function that creates a timer metric. The unit for this metric is currently nanoseconds, but our tooling generally uses microseconds for durations. For consistency, the timer implementation returned by Timer should record its units in microseconds.

Note that this change will alter the semantic output of the metrics registry.

pkg/cli fails standard golint check

Implementing a cli.App.ContextConfig function causes the implementing package to fail standard golint checks:

func initContext(_ cli.Context, ctx context.Context) context.Context {
...
}

fails with:

[golint]        main.go:41:1: context.Context should be the first parameter of a function

metrics: do not mutate context when adding tags

Currently, there are scenarios in which metrics.AddTags mutates the contents of the provided context: https://github.com/palantir/pkg/blob/master/metrics/context.go#L49

This behavior is documented, but is unintuitive/quite different from the usual pattern, which generally treats contexts as immutable and makes a copy of the entire context before updating state.

I'm generally fine with changing this, but just want to take a look at the Git history of the original change, as I believe that I chose this behavior based on requests from clients/users. However, if this is not the case (or if the clients can agree to migrate to the new behavior), then I'm all for changing this.

Switch to go-metrics with decaying histograms

rcrowley/go-metrics#254

add pkg/uuid package

There is no standard package for uuid, and there are multiple open source options to choose from. It is something worth considering for better code consistency.

Recommend a better golang package for datetime

Carbon is a simple, semantic and developer-friendly golang package for datetime, it has been included by awesome-go

https://github.com/golang-module/carbon