liquibase/liquibase-package-manager

lpm - Liquibase package manager

lpm - Liquibase Package Manager

Easily manage external dependencies for Database Development. Search for, install, and uninstall liquibase drivers, extensions, and utilities.

lpm is experimental and not officially supported

lpm is an experimental project. Issues can be reported here, but there is no guarantee of support.

Installation

lpm is distributed as a single binary. Install lpm by downloading, unzipping, and moving it to a directory included in your system's PATH. Releases are available here.

Setup

lpm will make a best effort to locate the location of the liquibase lib directory. It is recommended to set the LIQUIBASE_HOME environment variable.

Examples:

export LIQUIBASE_HOME=/usr/local/opt/liquibase/libexec
echo  'export LIQUIBASE_HOME=/usr/local/opt/liquibase/libexec' >> ~/.bashrc 

Usage

lpm <command>

Available Commands

• add
• completion
• dedupe
• help
• install
• list
• remove
• search
• update
• upgrade

Autocompletion

lpm can generate shell completions for multiple shells. The following shells are available:

bash

Generate the autocompletion script for lpm for the bash shell. To load completions in your current shell session: source <(lpm completion bash)

To load completions for every new session, execute once:

• Linux:

lpm completion bash > /etc/bash_completion.d/lpm

• MacOS:

lpm completion bash > /usr/local/etc/bash_completion.d/lpm

zsh

To load completions in your current shell session: source <(lpm completion zsh)

To load completions for every new session, execute once: lpm completion zsh > "${fpath[1]}/_lpm"

fish

To load completions in your current shell session: lpm completion fish | source

To load completions for every new session, execute once: lpm completion fish > ~/.config/fish/completions/lpm.fish

You will need to start a new shell for this setup to take effect.

liquibase-package-manager's Issues

Add Update add creates multiple entries of jar files in the classpath

Describe the bug
I used lpm v0.1.7

There are multiple jars for the same plugin in the liquibase_libs directory
when there should only be one.
Liquibase will use the first jar in the list and not the second.

To Reproduce
Steps to reproduce the behavior:
Ran
lpm add hashicorp-vault
to add an extension
Realized it was an older version of the extension (version was 0.0.4)
Ran
lpm update
to update the json files lpm was using
Ran
lpm add hashicorp-vault
which ran successfully but I thought it would update the version that was already there.

Expected behavior
I would think this would either error out and ask the user to upgrade or upgrade in place

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Mac
• Browser N/A
• Version

Additional context
Add any other context about the problem here.

update aws-secrets-manager

Description
Update aws-secrets-manager in packages.json to match with aws-secrets-manager artifact name which is liquibase-aws-secrets-manager

Once open sourced update links

There are two locally hard links, one for a package and one for the package.json update URL. After open sourced, update these links.

generate autocomplete for shell

As a cli user, autocomplete prompts are very helpful.

https://github.com/spf13/cobra/blob/master/shell_completions.md

Support for Databricks Driver and Extension

lpm does not currently support the databricks Pro extension or JDBC driver. Both of these are required to connect Liquibase to Databricks. Information about the required files can be found here: https://github.com/liquibase/liquibase-databricks

Include lpm in liquibase packages (rpm, deb, brew, etc)

Problem

similar to the node/npm relationship, installing the package manager automatically will make for a better UX

Solution

add dependencies to the various packagers

"lpm add snowflake --global" does not pull down Liquibase extension

Describe the bug
Snowflake require two files: snowflake driver and liquibase-snowflake extension. LPM pulls down the driver but not the extension.
The liquibase-snowflake extension is released for each version of Liquibase and available for download.

Here is the link for version 4.8.0: https://github.com/liquibase/liquibase-snowflake/releases/download/liquibase-snowflake-4.8.0/liquibase-snowflake-4.8.0.jar

To Reproduce
No need to reproduce.
LPM is coded to only grab the snowflake driver jar file. It also needs to grab the liquibase-snowflake jar file.

Expected behavior
All files needed to work with Snowflake should be available when running "lpm add snowflake --global" command.

Additional context
With this fix, it will also fix Liquibase Docker image generated using this Dockerfile.

Add Snowflake driver

From snowflake dockerfile:

ARG SNOWFLAKE_EXTENSION_VERSION=4.4.2
ARG SNOWFLAKE_EXTENSION_SHA1=3e721e766a42fa8b3b80982e05f8bf08d5af24d8
RUN wget --no-verbose -O /liquibase/lib/snowflake-extension.jar https://repo1.maven.org/maven2/org/liquibase/ext/liquibase-snowflake/${SNOWFLAKE_EXTENSION_VERSION}/liquibase-snowflake-${SNOWFLAKE_EXTENSION_VERSION}.jar \
	&& wget --no-verbose -O wget -O /liquibase/lib/snowflake-extension.jar.asc https://repo1.maven.org/maven2/org/liquibase/ext/liquibase-snowflake/${SNOWFLAKE_EXTENSION_VERSION}/liquibase-snowflake-${SNOWFLAKE_EXTENSION_VERSION}.jar.asc \
    && gpg --auto-key-locate keyserver --keyserver keyserver.ubuntu.com --keyserver-options auto-key-retrieve --verify /liquibase/lib/snowflake-extension.jar.asc /liquibase/lib/snowflake-extension.jar \
	&& echo "$SNOWFLAKE_EXTENSION_SHA1 /liquibase/lib/snowflake-extension.jar" | sha1sum -c -

`lpm install oracle` installs MySQL connector

Describe the bug
Attempting to install Oracle Driver (ojdbc*.jar), and unable to. Get MySQL instead.

To Reproduce

Issue lpm install oracle

hecksum verified. Installing mysql-connector-java-8.0.25.jar to C:\dev\liquibase\liquibase-4.4.3/liquibase_libs/
mysql-connector-java-8.0.25.jar successfully installed in classpath.

---------- IMPORTANT ----------
Add the following JAVA_OPTS to your CLI:
export JAVA_OPTS="-cp liquibase_libs/*:C:\dev\liquibase\liquibase-4.4.3\lib\*:C:\dev\liquibase\liquibase-4.4.3\liquibase.jar"

Expected behavior
The ojdbc*.jar installed instead of mysql connector.

Desktop (please complete the following information):
Windows 10, lpm version 0.1.1

liquibase_libs is included in recent verison of liquibase

Check for liquibase version and conditionally output export command for local libs directory.

Add Oracle jdbc 8 version 21.12.0.0

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
Add Oracle jdbc 8 version 21.12.0.0. This the latest 21 version available.

Please include 21.12 for oraclepki, oracle-osdt-cert, and oracle-osdt-core also.

Describe alternatives you've considered
N/A

Additional context
N/A

Better output for ls with no liquibase_modules

lpm ls should give better output when a directory does not have a liquibase_modules directory, yet. Prompt a user to add packages or install them from liquibase.json.

Dependency tree

It's very common for one dependency to have it's own dependency. Example: the postgres extension doesn't do much without the postgres driver. lpm should be able to define and string together the entire dependency tree.

Upgrade a package to latest

add multiple packages in a single line

example: lpm add postgresql mssql

lpm creates liquibase_libs directory regardless of command issued

Describe the bug
lpm creates a liquibase_libs directory regardless of whether it is needed or not.

To Reproduce
Steps to reproduce the behavior:
Run any of the following commands and an empty liquibase_libs directory will be created in the cwd.

lpm list
lpm help
lpm search

I tested a few others that did not create it, so list may not be exhaustive.

Expected behavior

If liquibase_libs is not needed, do not create an empty directory.
Depending on where you run lpm, it will create empty directories.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: MacOS
• Browser : NA
• Version : 0.1.2

Smartphone (please complete the following information):
NA

Additional context
Add any other context about the problem here.

Add mongodb dependencies

Is your feature request related to a problem? Please describe.
When you use liquibase-mongodb, you can got parse error if the json file use /t (TAB) indentation.

Can be solved using jackson libs (https://github.com/liquibase/liquibase-mongodb#prerequisites)

Would be nice to add this dependencies on lpm add mongdb.

golint not found

Describe the bug
golint is not found on the github actions runner:

/home/runner/work/_temp/df7a5e2b-19e9-4b1c-9b18-aad5495[6](https://github.com/liquibase/liquibase-package-manager/runs/7030795143?check_suite_focus=true#step:5:7)ca94.sh: line 1: golint: command not found
Error: Process completed with exit code 12[7](https://github.com/liquibase/liquibase-package-manager/runs/7030795143?check_suite_focus=true#step:5:8).

To Reproduce
Steps to reproduce the behavior:

1. Go to any recent action
2. Click on the build log
3. See error

Expected behavior
go code should be linted

Additional context
NOTE: Golint is golang/go#38968. There's no drop-in replacement for it, but tools such as Staticcheck and go vet should be used instead.
https://github.com/golang/lint

refactor distinct install and add command

add: adds a package to use in your current package.
install: installs all the dependencies defined in a liquibase.json file.

In general, refactor to better align with yarn commands and their distinct command phases

Load dependencies from liquibase.json

For a project, have a liquibase.json file that defines dependencies, installs them locally in LIQUIBASE_PACKAGES, sets this as the class path for Liquibase. Instead of global config only.

Update to latest go to include several security fixes

Update go to version 1.20.10 or 1.21.3 to include the latest security fixes.

This way there will not be security violations included in security scans that include lpm.

enhanced support for oracle drivers

Is your feature request related to a problem? Please describe.
I cannot currently download the ojdbc11 driver with lpm and the ojdbc8 driver is not the newest.

I also need the following JARs to support Oracle ATP and ADW:
oraclepki.jar, osdt_cert.jar, and osdt_core.jar

Describe the solution you'd like
lpm install oracle8 and lpm install oracle11 (or some variance of that name) installs the oracle driver
lpm install oracle-security(?) installs the 3 security related jars

Describe alternatives you've considered
I could not use lpm and just download the jar files manually

maven central links:
ojdbc8 - https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8
NOTE: 8 has been moved to 10: https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc10
ojdbc11 - https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc11
oraclepki - https://mvnrepository.com/artifact/com.oracle.database.security/oraclepki
osdt_cert - https://mvnrepository.com/artifact/com.oracle.database.security/osdt_cert
odst_core - https://mvnrepository.com/artifact/com.oracle.database.security/osdt_core

LPM fails to follow HTTP redirects

Describe the bug
If lpm is run in an environment that it has to follow a redirect to download a package (in our case, running in a kubernetes pod with an Istio sidecar), the deployment fails with error code 1 and error:

Unable to download from <url>

To Reproduce
Steps to reproduce the behavior:

1. Create a docker image running an lpm install
   e.g.

FROM liquibase/liquibase:4.15.0

RUN lpm add mysql --global

2. Deploy this container with an Istio sidecar
3. Check container logs

Expected behavior
LPM follows the istio redirect and successfully downloads the package

Additional context
I believe the issue here is this redirect handling.

Per the go docs, if URL.Opaque is set, the url is constructed as <scheme>:<opaque>?<query>#<fragment>, instead of <scheme>://<user>@<host>/<path>?<query>#<fragment>.

So, by setting URL.Opaque to URL.Path you're dropping the host information.

For example, if attempting to download:

https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.30/mysql-connector-java-8.0.30.jar```
you'd instead attempt to GET:

https://maven2/mysql/mysql-connector-java/8.0.30/mysql-connector-java-8.0.30.jar

But because the Reidrect handler doesn't actually change the `url` variable, the error logging still would report:

Unable to download from https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.30/mysql-connector-java-8.0.30.jar

provide arm64 compatible binary

Is your feature request related to a problem? Please describe.

we run our CI system on AWS Graviton instances (arm64 cpu chipset) for better price performance, the current lpm cli is for x86_64 only.
if run on arm64, i see error: /liquibase/bin/lpm: cannot execute binary file: Exec format error

Describe the solution you'd like
we can build the binaries for both arm64 and x86_64:
https://github.com/liquibase/liquibase-package-manager/blob/master/Makefile#L18

GOOS=linux GOARCH=amd64 go build -o $(PWD)/bin/linux/lpm $(PWD)/cmd/lpm/darwin.go
GOOS=linux GOARCH=arm64 go build -o $(PWD)/bin/linux/lpm $(PWD)/cmd/lpm/darwin.go

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Add ability to install latest and/or version pin

Problem

Some users want to stay up to date on their packages

Solution

Add lpm upgrade option that will upgrade installed packages
Add version contraints ability such as liquibase-postgresql ~> 2.0

Update package.json file nightly

In a style similar to dependabot, there should be a nightly job to look from package updates and make a pull request to update the package.json file.

-g is throwing error on --upgrade

Describe the bug
Using -g for --upgrade is throwing error.

To Reproduce

C:\dev\liquibase\liquibase-4.17.2>lpm upgrade --global
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x28 pc=0x9696d9]

goroutine 1 [running]:
github.com/hashicorp/go-version.(*Version).String(0x0)
/Users/mcred/projects/go/pkg/mod/github.com/hashicorp/[email protected]/version.go:369 +0x39
github.com/hashicorp/go-version.(*Version).Compare(0xc0000d0280, 0x0)
/Users/mcred/projects/go/pkg/mod/github.com/hashicorp/[email protected]/version.go:116 +0x3b
github.com/hashicorp/go-version.(*Version).LessThan(...)
/Users/mcred/projects/go/pkg/mod/github.com/hashicorp/[email protected]/version.go:308
package-manager/internal/app/packages.Package.GetLatestVersion({{0xc0000121e0, 0x11}, {0xc000588e90, 0x9}, {0xc000066d80, 0x2a, 0x2a}}, 0xc000066d80?)
/Users/mcred/projects/liquibase/package-manager/internal/app/packages/Package.go:23 +0x1ab
package-manager/internal/app/packages.Packages.GetOutdated({0xc000076000?, 0x170f453ffff?, 0x197?}, 0x2030000?, {0xc0000d8400, 0x7, 0x8})
/Users/mcred/projects/liquibase/package-manager/internal/app/packages/Packages.go:41 +0x205
package-manager/internal/app/commands.glob..func7(0xf5f6c0?, {0xc1ce63?, 0x1?, 0x1?})
/Users/mcred/projects/liquibase/package-manager/internal/app/commands/upgrade.go:24 +0x5f
github.com/spf13/cobra.(*Command).execute(0xf5f6c0, {0xc000089200, 0x1, 0x1})
/Users/mcred/projects/go/pkg/mod/github.com/spf13/[email protected]/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0xf5ee20)
/Users/mcred/projects/go/pkg/mod/github.com/spf13/[email protected]/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
/Users/mcred/projects/go/pkg/mod/github.com/spf13/[email protected]/command.go:968
package-manager/internal/app/commands.Execute({0xc0000c8360?, 0xc0000c8330?}, {0xc1c8a4, 0x1})
/Users/mcred/projects/liquibase/package-manager/internal/app/commands/root.go:39 +0x125
main.main()
/Users/mcred/projects/liquibase/package-manager/cmd/lpm/windows.go:49 +0x2a9

C:\dev\liquibase\liquibase-4.17.2>lpm upgrade
You have no outdated packages installed.
C:\dev\liquibase\liquibase-4.17.2/liquibase_libs/

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

"lpm update" fails inside docker build with alpine-linux base image

Describe the bug
After stumbling upon this issue (liquibase/docker#171), I started working on installing lpm (v0.2.0) and liquibase (v4.19.0) while building a docker image based on alpine-linux (eclipse-temurin:8-jre-alpine) . But the docker build fails with the following error:

Step 8/20 : ARG LIQUIBASE_VERSION=4.19.0
 ---> Running in 8d950e042c3b
Removing intermediate container 8d950e042c3b
 ---> 703d3c06ac1f
Step 9/20 : ARG LPM_VERSION=0.2.0
 ---> Running in 176d13b4fc35
Removing intermediate container 176d13b4fc35
 ---> 473a81914799
Step 10/20 : ARG LB_SHA256=2ec24cacf1dc6794cde139de9778854839ee1d3fa9c134fefa92157401e57134
 ---> Running in 4931741dd23f
Removing intermediate container 4931741dd23f
 ---> f971ca606116
Step 11/20 : RUN set -x   && wget -q -O liquibase-${LIQUIBASE_VERSION}.tar.gz "https://github.com/liquibase/liquibase/releases/download/v${LIQUIBASE_VERSION}/liquibase-${LIQUIBASE_VERSION}.tar.gz"   && echo "$LB_SHA256  liquibase-${LIQUIBASE_VERSION}.tar.gz" | sha256sum -c -   && tar -xzf liquibase-${LIQUIBASE_VERSION}.tar.gz   && rm liquibase-${LIQUIBASE_VERSION}.tar.gz
 ---> Running in 03a4bdc7d61b
+ wget -q -O liquibase-4.19.0.tar.gz https://github.com/liquibase/liquibase/releases/download/v4.19.0/liquibase-4.19.0.tar.gz
+ echo '2ec24cacf1dc6794cde139de9778854839ee1d3fa9c134fefa92157401e57134  liquibase-4.19.0.tar.gz'
+ sha256sum -c -
liquibase-4.19.0.tar.gz: OK
+ tar -xzf liquibase-4.19.0.tar.gz
+ rm liquibase-4.19.0.tar.gz
Removing intermediate container 03a4bdc7d61b
 ---> 614bda2eb46a
Step 12/20 : RUN mkdir /liquibase/bin     && wget -q -O lpm.zip "https://github.com/liquibase/liquibase-package-manager/releases/download/v${LPM_VERSION}/lpm-${LPM_VERSION}-linux.zip"     && unzip lpm.zip -d bin/     && rm lpm.zip
 ---> Running in e211d9fe6d60
Archive:  lpm.zip
  inflating: bin/lpm                 
Removing intermediate container e211d9fe6d60
 ---> f483dbbd0521
Step 13/20 : ENV LIQUIBASE_HOME /liquibase
 ---> Running in 0d4c6010a863
Removing intermediate container 0d4c6010a863
 ---> 45cfedb3a773
Step 14/20 : RUN lpm update     && /liquibase/liquibase --version
 ---> Running in fdc5fd97d8f4
Unable to download from https://raw.githubusercontent.com/liquibase/liquibase-package-manager/master/internal/app/packages.json

The documentation of lpm shows an extra flag -p, whose default value is the same as given in the error. I don't know how to fix this, so any help here is highly appreciated. Thank you.

Expected behavior
The docker build should succeed

Desktop (please complete the following information):

• OS: Ubuntu 20.04
• Docker: Docker version 20.10.23, build 7155243

Multiple entries in json manifest

When running lpm add for a directory that has a liquibase.json manifest, if install was not run first, it will add the package and then add a second/third entry for the package. lpm should confirm a package needs to be installed before attempting to install it.

Auto upgrade installed extensions when liquibase is upgraded

Problem

Many extensions have a dependency on liquibase being the same version. Sometimes when liquibase is upgraded, but the extension is not, it can cause issues.

Solution

When liquibase is upgraded, auto upgrade the extensions to the matching version.

rename liquibase_modules to liquibase_libs

In anticipation of liquibase/liquibase#2015, liquibase_libs better aligns with current liquibase standards and can be looked for by the CLI runners automatically.