Add support for .tar archives as downloaded from dev.mysql.com

[utdrmac]

Add support for .tar archives as downloaded from dev.mysql.com

[utdrmac]

dbdeployer downloads add requires a local file, which means you need to download it first.

The help for add does not indicate anywhere that it must already be downloaded. As I said above, I'm expecting add to simply add it to the list and then I can download from that list later. Some additional messaging might be helpful here. I've spent the last hour trying to simply add the latest mysql to dbdeployer and it's very frustrating.

I would not expect a tool that has download abilities to require me to first manually download what I want to add before it can be added.

$ dbdeployer downloads add --help
Adds a tarball to the list

Usage:
  dbdeployer downloads add tarball_name [flags]

[datacharmer]

The file you are trying to add is an uncompressed archive containing more compressed archives.
As the error message says, dbdeployer only deals with .tar.gz or .tar.xz, not .tar
To add a new tarball, don't choose files manually, but do the following:

dbdeployer downloads add-remote mysql 8.0 Linux --minimal

Example result:

Tarball below added to $HOME/.dbdeployer/tarball-list.json
Name:          mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz
Short version: 8.0
Version:       8.0.33
Flavor:        mysql
OS:            Linux-amd64
URL:           https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz
Checksum:      MD5:377b6ae9c88a698438a19d67efa03252
Size:          60 MB
Notes:         added with version 1.72.1
Added on:      2023-06-20 15:59
Name:          mysql-8.0.33-linux-glibc2.17-aarch64-minimal.tar.xz
Short version: 8.0
Version:       8.0.33
Flavor:        mysql
OS:            Linux-aarch64
URL:           https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-aarch64-minimal.tar.xz
Checksum:      MD5:f5e01242975453ec6db02727848975c9
Size:          58 MB
Notes:         added with version 1.72.1
Added on:      2023-06-20 15:59

[datacharmer]

BTW: the problem is not "lack of support for .tar", but that the MySQL binaries are always in compressed files. Support for plain .tar would not accomplish anything

[utdrmac]

If .tar is not supported, please don't allow .tar to be added in the first place.

The file you are trying to add is an uncompressed archive containing more compressed archives.

I'm not sure how I was supposed to know that.

I originally tried as you suggested, which failed:

~/dbdeployer$ dbdeployer downloads add-remote mysql 8.0.32 linux
error getting remote tarball description: version '8.0.32' is not accepted for tarball type mysql: it must be one of [[5.7 8.0]]
~/dbdeployer$
~/dbdeployer$ dbdeployer downloads add-remote mysql 8.0 linux
error getting remote tarball description: no mysql tarballs found for 8.0
~/dbdeployer$
~/dbdeployer$ dbdeployer downloads add-remote mysql 8.0 Linux
error getting remote tarball description: no mysql tarballs found for 8.0
~/dbdeployer$
~/dbdeployer$ dbdeployer update
......  7.0 MB
1
  97 B
checksum for dbdeployer-1.72.0.linux.tar.gz matches
dbdeployer version 1.72.0
~/dbdeployer$
~/dbdeployer$ dbdeployer downloads add-remote mysql 8.0 Linux
Provided defaults are for version 1.62.0. Current version is 1.70.0
********************************************************************************
Defaults file /home/xxxxxx/.dbdeployer/config.json not validated.
Loading internal defaults
********************************************************************************

Provided defaults are for version 1.62.0. Current version is 1.70.0
error writing tarball list: [write tarball file info] tarball list check failed : tarball with OS Linux-amd64, flavor mysql, version 8.0.33, and minimal false listed more than once

[utdrmac]

Then I'm trying this:

$ dbdeployer downloads add mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz \
  --arch x86_64 \
  --OS Linux --flavor mysql \
  --minimal \
  --url https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz
file '/home/xxxxx/dbdeployer/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz' not found

Of course it is not found, it hasn't been downloaded yet. I expect dbdeployer to add it (as the command states) then I can get-unpack (?) after it is added.

[datacharmer]

dbdeployer downloads add requires a local file, which means you need to download it first.
Using dbdeployer downloads add-remote will free you from that burden.

Regarding your failure:

dbdeployer downloads add-remote mysql 8.0.32 linux

You need to use 8.0, and dbdeployer will find the newest one.

[utdrmac]

You need to use 8.0, and dbdeployer will find the newest one.

If you look at the last example in my comment above, I did try exactly that, which produced the error:

~/dbdeployer$ dbdeployer downloads add-remote mysql 8.0 Linux
Provided defaults are for version 1.62.0. Current version is 1.70.0
********************************************************************************
Defaults file /home/xxxxxx/.dbdeployer/config.json not validated.
Loading internal defaults
********************************************************************************

Provided defaults are for version 1.62.0. Current version is 1.70.0
error writing tarball list: [write tarball file info] tarball list check failed : tarball with OS Linux-amd64, flavor mysql, version 8.0.33, and minimal false listed more than once

[datacharmer]

Since you have add and add-remote, I think the meaning of add is clear.
The purpose of add is to add to the download list, using a local file, while add-remote does the same, but from remote information.

Anyway, dbdeployer can download things for you. If you want to do things with add instead of add-remote, you can download the file first

dbdeployer downloads  get https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz

and then use dbdeployer downloads add ...

[utdrmac]

How about changing the help to look like this, which would avoid this confusion:

Available Commands:
  add            Adds a local tarball to the list
  add-remote     Adds a tarball to the list, by searching MySQL downloads site

And also:
if(substring(FILENAME, -3) == ".tar") {
error(".tar is not allowed. Please provide the .tar.gz or .tar.xz")
}

[httpete]

I agree @utdrmac - I spent the last few hours fishing around trying to just add the latest version. asdf, a very popular utility relies on the fantastic dbdeployer and we are all blocked with that 8.0.33 in the list. A one command way to do the update and open a PR and we can help out.

[datacharmer]

@httpete ,
To add a new tarball, do the following:

$ dbdeployer downloads add-remote mysql 8.0 Linux --minimal
Example result:

Tarball below added to $HOME/.dbdeployer/tarball-list.json
Name:          mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz
Short version: 8.0
Version:       8.0.33
Flavor:        mysql
OS:            Linux-amd64
URL:           https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-x86_64-minimal.tar.xz
Checksum:      MD5:377b6ae9c88a698438a19d67efa03252
Size:          60 MB
Notes:         added with version 1.72.1
Added on:      2023-06-20 15:59
Name:          mysql-8.0.33-linux-glibc2.17-aarch64-minimal.tar.xz
Short version: 8.0
Version:       8.0.33
Flavor:        mysql
OS:            Linux-aarch64
URL:           https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.33-linux-glibc2.17-aarch64-minimal.tar.xz
Checksum:      MD5:f5e01242975453ec6db02727848975c9
Size:          58 MB
Notes:         added with version 1.72.1
Added on:      2023-06-20 15:59

Repeat the procedure for all flavours and operating systems:

dbdeployer downloads add-remote shell 8.0 Linux 
dbdeployer downloads add-remote cluster 8.0 Linux
dbdeployer downloads add-remote mysql 8.0 macOS
dbdeployer downloads add-remote cluster 8.0 macOS
dbdeployer downloads add-remote shell 8.0 macOS

The result will be in $HOME/.dbdeployer/tarball-list.json
That will make the downloads usable in your environment.

If you want to create a pull request for inclusion, run this command from the root of dbdeployer repo code

jq . < $HOME/.dbdeployer/tarball-list.json > downloads/tarball_list.json

Note jq is an open source JSON processor

[httpete]

done and PR updated.

[httpete]

OK something didn't work right downstream, I don't see any entry for the NON minimal mysql for linux and I Can't rerun the script. You had me use the --minimal flag for mysql and now I don't see it. Lil help?

[httpete]

@datacharmer - stuck cold. I thought overwrite flag would work but it keeps telling me it's already listed.

[dbdeployer]$ ./dbdeployer downloads add-remote mysql 8.0 Linux
error writing tarball list: [write tarball file info] tarball list check failed : tarball with OS Linux-amd64, flavor mysql, version 8.0.33, and minimal false listed more than once

[httpete]

Let's be sure for posterity to have this be the command set for updating to the latest MySql and publishing the tarballs:

First, run the installation script

dbdeployer downloads add-remote mysql 8.0 Linux
dbdeployer downloads add-remote shell 8.0 Linux 
dbdeployer downloads add-remote cluster 8.0 Linux
dbdeployer downloads add-remote mysql 8.0 macOS
dbdeployer downloads add-remote cluster 8.0 macOS
dbdeployer downloads add-remote shell 8.0 macOS

The result will be in $HOME/.dbdeployer/tarball-list.json
That will make the downloads usable in your environment.

To create a pull request for inclusion, run this command from the root of dbdeployer repo code

jq . < $HOME/.dbdeployer/tarball-list.json > downloads/tarball_list.json

[datacharmer]

List of downloads updated in release 1.73.0