moov-io/base

core libraries used in Moov projects

base's Issues

log: Context should accept an interface{} instead of a string

The Context interface and related methods should accept an interface{} over a string.

type Context interface {
	Context() map[string]string // -> map[string]interface{}
}

since writeValue will be called by go-kit/log to properly encode the string.

Expose BankTime as a rest endpoint

Expose the bank time functionality as a utility as part of our rest endpoint.

http package: update the logger from go-kit to base/log

refactor: method names could be improved and shortened in the log package

cc: @alovak

Split migrations by environment (prod, staging, etc.)

The database package currently supports running a different set of migrations based on the database specified in the config (sqlite or mysql currently). We should support something similar, but have one set of migrations for prod and another set for staging. You can see the sqlite vs mysql migrations at work here:

base/database/migrator.go

Line 20 in b5da069

func RunMigrations(logger log.Logger, config DatabaseConfig) error {

database/testing: add option to keep databases spun up from tests

For development, it's really helpful to inspect the database tables after a test run.

Move ParseError / ErrorList from moov-io / ach into base library for re-use

There will be a need to re-use Parse Error / ErrorList for moov-io api's, packages, etc.

Copy ParseError / ErrorList from moov-io / ach into base library for re-use.

Put in root of github.com/moov-io/base and the it can be imported from other packages: base.ErrorList

Can't use InMemory SQLite

PayGate Version: ``

What were you trying to do?

use database.InMemorySqliteConfig with database.NewAndMigrate

What did you expect to see?

connection to database created and database migrated

What did you see?

err:"no such table: items"

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Other Branches

These updates are pending. To force PRs open, click the checkbox below.

• chore(deps): update actions/checkout action to v4
• chore(deps): update actions/setup-go action to v5
• chore(deps): update github/codeql-action action to v3

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update module github.com/rickar/cal/v2 to v2.1.18
• fix(deps): update module cloud.google.com/go/spanner to v1.66.0

Detected dependencies

docker-compose

docker-compose.yml

• mysql 9-oracle

dockerfile

mysql/Dockerfile

• mysql 9.0-oracle

github-actions

.github/workflows/cgo.yml

• actions/setup-go v4
• actions/checkout v2

.github/workflows/codeql.yaml

• actions/checkout v2
• github/codeql-action v2
• github/codeql-action v2

.github/workflows/mysql-volumeless.yml

• actions/checkout v2

.github/workflows/nocgo.yml

• actions/setup-go v4
• actions/checkout v2

gomod

go.mod

• go 1.21
• cloud.google.com/go/spanner v1.65.0
• github.com/go-kit/kit v0.13.0
• github.com/go-kit/log v0.2.1
• github.com/go-sql-driver/mysql v1.8.1
• github.com/golang-migrate/migrate/v4 v4.17.1
• github.com/google/uuid v1.6.0
• github.com/googleapis/gax-go/v2 v2.13.0
• github.com/googleapis/go-sql-spanner v1.6.0
• github.com/gorilla/mux v1.8.1
• github.com/madflojo/testcerts v1.2.0
• github.com/markbates/pkger v0.17.1
• github.com/mitchellh/mapstructure v1.5.0
• github.com/prometheus/client_golang v1.19.1
• github.com/rickar/cal/v2 v2.1.17
• github.com/spf13/viper v1.19.0
• github.com/stretchr/testify v1.9.0
• go.opentelemetry.io/otel v1.28.0
• go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0
• go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.28.0
• go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.28.0
• go.opentelemetry.io/otel/sdk v1.28.0
• go.opentelemetry.io/otel/trace v1.28.0
• google.golang.org/grpc v1.65.0

---

• Check this box to trigger a request for Renovate to run again on this repository

ISO8601 time.Time JSON handling

Over in moov-io/ach#298 (and slack) we decided to have timestamp values in our API be ISO 8601. This is a common format usable by lots of languages, but Go doesn't seem to have support for it.

There are a few stale implementations laying around Github. Let's fine one, dust it off and incorporate it into our base repository.

Here's the structure: moov-io/ach#298 (comment)

See: https://github.com/moov-io/api/issues/33

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

sqlite: create helpers for migration, connections, testing

We've got a fairly good sqlite setup working for our apps and consolidating that would help keep apps uniform. I'm proposing a few new packages.

github.com/moov-io/base/database

// migrate runs our database migrations (defined at the top of this file)
// over a sqlite database it creates first.
// To configure where on disk the sqlite db is set SQLITE_DB_PATH.
//
// You use db like any other database/sql driver.
//
// https://github.com/mattn/go-sqlite3/blob/master/_example/simple/simple.go
// https://astaxie.gitbooks.io/build-web-application-with-golang/en/05.3.html
func Migrate(db *sql.DB, logger log.Logger) error {
	logger.Log("sqlite", "starting sqlite migrations...")
	for i := range migrations {
		row := migrations[i]
		res, err := db.Exec(row)
		if err != nil {
			return fmt.Errorf("migration #%d [%s...] had problem: %v", i, row[:40], err)
		}
		n, err := res.RowsAffected()
		if err == nil {
			logger.Log("sqlite", fmt.Sprintf("migration #%d [%s...] changed %d rows", i, row[:40], n))
		}
	}
	logger.Log("sqlite", "finished migrations")
	return nil
}

Perhaps one day we'll adopt flyway or a version table. For now this works.

github.com/moov-io/base/database/sqlite

func DBPath() string {
	path := os.Getenv("SQLITE_DB_PATH")
	if path == "" || strings.Contains(path, "..") {
		// set default if empty or trying to escape
		// don't filepath.ABS to avoid full-fs reads
		path = "paygate.db"
	}
	return path
}

func Connection(path string) (*sql.DB, error) {
	db, err := sql.Open("sqlite3", path)
	if err != nil {
		err = fmt.Errorf("problem opening sqlite3 file: %v", err)
		logger.Log("sqlite", err)
		return nil, err
	}
	return db, nil
}

func Version() string {
	sqliteVersion, _, _ := sqlite3.Version()
	return sqliteVersion	
}

github.com/moov-io/base/database/sqlite/testsqlite

type TestSqliteDB struct {
	db *sql.DB
	dir string // temp dir created for sqlite files
}

func (r *TestSqliteDB) Close() error {
	if err := r.db.Close(); err != nil {
		return err
	}
	return os.RemoveAll(r.dir)
}

// Connection returns a TestSqliteDB which can be used in tests
// as a clean sqlite database. All migrations are ran on the db before.
//
// Callers should call close on the returned *TestSqliteDB.
func Connection(migrator func (db *sql.DB, logger log.Logger) error) (*TestSqliteDB, error) {
	dir, err := ioutil.TempDir("", "test-sqlite")
	if err != nil {
		return nil, err
	}

	db, err := createSqliteConnection(filepath.Join(dir, "database.db"))
	if err != nil {
		return nil, err
	}

	logger := log.NewLogfmtLogger(ioutil.Discard)
	if err := migrator(db, logger); err != nil {
		return nil, err
	}
	return &TestSqliteDB{db, dir}, nil
}

http: base setup for Moov HTTP servers

We're starting to see a fair amount of drift between http.Server's used in Moov. This is intentional as overtime we've adapted servers, but we should consolidate somewhat. There's a lot of re-used features that should be in a common library.

• X-Request-Id / X-Idempotency-Key / X-User-Id extraction
• GET /ping route
• 4xx / 5xx error encoding, logging, etc
  • We can use runtime.Caller(n) here too.
• Metrics and Log timing (via http.ResponseWriter)
• CORS headers

Refactor CheckRoutingNumber from ach to base

Refactor CheckRoutingNumber validator from ach to base for reuse in wire, and icl.

calculateCheckDigit (checksum) and roundUp10 would need to be refactored in as well.

docs: Add README in log package to guide users on how to use the logger

It's not clear how base/log package should be used without looking at the implementation.

There should be a README to guide how users should use the logger.

idempotent: create redis implementation

From the X-Idempotency-Key issue (moov-io/paygate#15 (comment)) we want to implement a redis implementation of idempotent.Recorder.

There's https://github.com/rafaeljusto/redigomock which looks like it works with https://github.com/gomodule/redigo. It's GPL'd though.

@wadearnold That project is GPL v2, can we use that? Probably not?
https://github.com/rafaeljusto/redigomock/blob/master/LICENSE

See: moov-io/paygate#21

Replace megacheck with staticcheck, megacheck has been deprecated

Replace megacheck with staticcheck

http: add the ability to handle unexpected panics

Our server should be able to handle unexpected panics by:

• logging the error info and stack trace
• set the response code to 500

log: add timestamp key-value to all logs

We should add a timestamp to all logs using

logger = log.With(logger, "ts", log.DefaultTimestampUTC)

logger.Log("msg", "hello")

// Output:
// ts=2016-01-01T12:34:56Z msg=hello

ID() should be more unique

The existing implementation of ID() string only creates random hex strings. There's nothing to ensure uniqueness. We should migrate to something which does.

Currently our stack cares these strings are 40 characters long and of hex strings to strip them from Prometheus paths. (e.g. GET /users/fffffffffffffff/accounts to get-users-accounts)

Candidates

• https://github.com/gofrs/uuid

logging: add a method for logging string slices

We can do

logger.Set("orgIDs", log.String(fmt.Sprintf("[%s]",strings.Join(organizationIDs, ","))))

but a string array method would be helpful:

logger.Set("orgIDs", log.Strings(organizationIDs))

Both would output orgIDs=["id-1", "id-2", "id-3"]

---

Type signature for the valuer would be:

func Strings(vals []string) Valuer