trivadispf/platys

A tool for generating docker-compose environments

platys - Trivadis Platform in a Box

Copyright (c) 2018-2024 Trivadis

What is platys?

platys is a tool for generating and provisioning Modern Data Platforms based on Docker and Docker Compose.

Its main use is for small-scale Data Lab projects, Proof-of-Concepts (PoC) or Proof-of-value (PoV) projects as well as trainings.

The user of platys can choose which services to use from a list of supported services and generate a fully working docker-compose.yml file including all necessary configuration files.

This replaces our old approach, where we only had a static docker-compose.yml file with all services enabled by default. By generating the docker-compose.yml, the user has very fine-grained control on which services to include for a given platform.

How does platys work?

The following diagram shows the building blocks of platys and the basic flow when working with it.

A concrete Platform is always generated based on a given Platform Stack. A platform stack defines the set of available and usable services and has a name and a version.

The digram shows the 5 necessary steps to create a new platform and use it:

1. Initialise a new Platform context by specifying a Platform Stack. Optionally a set of services to be enabled can be specified with the init command.
2. Optionally edit the config.yml to enable services and change default values of configuration settings.
3. Generate the artefacts for the platform (mainly the docker-compose.yml but also some configuration files) by running the gen command.
4. Run docker-compose up to start your platform.
5. Use the services of your platform, once they are stared and ready to use.

Currently there is one supported Platform Stack:

• modern-data-platform - a Platform Stack for supporting a Modern (Analytical) Data Platforms

In the future, other platform stacks might be added.

Why platys?

There is no question that running workloads in containers simplifies deployment of applications. But one container is never enough, you need more than one container to implement a working solution, such as database, business logic, event broker ... and you need a Container Orchestration to manage these containers.

Today Kubernetes is the most popular container orchestrator, but it comes with a lot of complexity. For production setups, Kubernetes is definitely one way to go. But for local, development or small-scale Proof-of-Concepts, we like to use Docker Compose, a very simple approach to container orchestration. With Compose, you use a YAML file to configure your application’s services.

Especially as a consultant, coach, trainer, technology evangelist, you will be using different Compose setups for different environments.

The longer you use Docker Compose, the more of these YAML files you get and to maintain them is quite a challenge:

• But how do you easily upgrade to a new version of a container, i.e. Apache Kafka?
• Do you manually have to go through all of these Compose files, which is a lot of work and prone to errors?
• What if you want to add a new service to one environment and you know that you have used it previously in another environment?
• Do you copy-paste configs from one YAML to another?
• How do you make sure that one service from another environment will work with all the configs and port settings of your other environment?

For these and some other challenges we were looking for a better solution:

• Wouldn't it be easier to generate the Docker Compose YAML, based on a simple configuration with some ON/OFF switches of all supported services?

Enter the world of platys...

Where can I run platys?

platys is supported on Windows, macOS and 64-bit Linux.

Behind the scenes, platys runs the generator (supporting given Platform Stack) as a Docker container. Therefore you need to have Docker installed on the machine where you create a Platform. To run the Platform, you also need to have Docker Compose installed on the target machine, which can be different to the one you use for generating the platform.

See Installing platys for how to install platys and then the Getting Started with Platys and the Modern Data Platform Stack for how to use platys.

Where can I run a generated platform ?

The generated platform can be provisioned either locally or in the cloud. See Provisioning of Modern Data Platform for various versions of how to deploy the platform.

Changes

See What's new? for a detailed list of changes.

Documentation

Usage

• Installing platys
• Provisioning a platys-ready environment
• Getting Started with platys and the modern-data-platform platform stack
• Frequently Asked Questions
• Command line reference
• Glossary of Terms
• How to use a platys-generated stack without Internet

Development

• Service Design Decisions
• Creating and maintaining a Platform Stack

Platform Stacks

• Adding additional services not supported by a Platform Stack
• Upgrade to a new platform stack version

platys's Issues

Add Platys to awesome-compose once we are ready

https://github.com/docker/awesome-compose

"platys init" with the "-f" (force) flag does not work correctly

1. I'm doing a platys init --enable-services HIVE_METASTORE,MINIO,AWSCLI,PRESTO,PROVISIONING_DATA -s trivadis/platys-modern-data-platform -w 1.8.0-preview

2. a correct config.yml is generated

3. now I'm changing the --enable-servicesand use the-fflag to force overwriting config.yml:platys init --enable-services ZOOKEEPER,KAFKA --stack trivadis/platys-modern-data-platform --stack-version 1.7.0 -f

4. the generated config.yml is incorrect (from Presto====):

      # Default values for the generator
      # this file can be used as a template for a custom configuration
      # or to know about the different variables available for the generator
      platys:
          platform-name: 'default'
          platform-stack: 'trivadis/platys-modern-data-platform'
          platform-stack-version: '1.7.0'
          structure: 'flat'
      # ===== Global configuation, valid for all or a group of services ========
      # Timezone, use a Linux string such as Europe/Zurich or America/New_York
      use_timezone: 'true'
      # the name of the repository to use for private images, which are not on docker hub (currently only Oracle images)
      private_docker_repository_name: 'true'
      # ===== Apache Zookeeper ========
      ZOOKEEPER_enable: true
      # ===== Apache Kafka ========
      KAFKA_enable: true

 Presto ========
      PRESTO_enable: true
      #=====  MinIO Object Storage ========
      MINIO_enable: true
      AWSCLI_enable: true

Would be nice to have the output of "platys list_services" ordered

Currently the output of platys list_services does not seem to have any order. It would be nice to have it ordered, either:

1. in the same order as in the config.yml or
2. alphabetically order it

If possible I would prefer the first option.

how can clean / stop / restart / delete / kill all docker or specific docker?

Add Platys to awesome-compose once we are ready

https://github.com/docker/awesome-compose

We should change the name of the "config.yml" header properties to reflect correct names

Currently the header in the config.yml holds the following "header" properties.

      platys:
        platform-name: 'default'
        stack-image-name: 'trivadis/platys-modern-data-platform'
        stack-image-version: '1.5.2'
        structure: 'flat'

stack-image-name and stack-image-version are the old names, we have changed them to platform-stack and platform-stack-version in the documentation and the flags we support. Could we change it to the new names, but still support the old ones in the platys CLI (so we are backward compatible to the platform stacks). So we would support both the one above as well as the new shown below:

      platys:
        platform-name: 'default'
        platform-stack: 'trivadis/platys-modern-data-platform'
        platform-stack-version: '1.5.2'
        structure: 'flat'

Migrate command line to go rather than Python

Since there are a lot of compatibility issues when cross-compiling python the cli will be migrated to Go so that it can easily be cross-compiled

We should have some sort of profile support

When using a platform generated by platys, then some of the settings might be needed when running it one one environment but not when running it on another one.

It would be good to support some sort of conditional properties in the config.yml so that the same config can be used to generate the "same" platform for different environments.

Can we support setting the environment variables throught platys CLI?

Currently we have two environment variables DOCKER_HOST_IP and PUBLIC_IP which have to be set correctly before starting/working with the platform.
They can be added to the .env file, maybe we could add some support through the CLI to simplify handling, especially for environments where the IP is not stable.

--platform-name and --structure only work in init if also -s is used

With the latest version of init, the following command works correctly

python3 ./platys.py init --platform-name test2 --stack-version 1.5.0-preview -f -s KAFKA,PYTHON --structure subfolder

but if -s is not passed as an agrument

python3 ./platys.py init --platform-name test2 --stack-version 1.5.0-preview -f --structure subfolder

Then the config.yml is not updated.

We should remove the default from the --structure flag

Currently when doing a platys gen without specifying the --structure flag, the default is used.

This causes a warning in the log

[ZOOKEEPER_nodes] -> [1] v: [3]
[KAFKA_broker_nodes] -> [3] v: [6]
[KAFKA_SCHEMA_REGISTRY_nodes] -> [1] v: [2]
[KAFKA_CONNECT_nodes] -> [2] v: [3]
[KAFKA_KSQLDB_nodes] -> [1] v: [3]
[HADOOP_datanodes] -> [2] v: [6]
[DATASTAX_nodes] -> [3] v: [3]
[MOSQUITTO_nodes] -> [1] v: [3]
WARNING:root:The [--structure] argument is deprecated and will soon be removed please use the platys.structure variable in the config.yml file
generating stack on destination [/home/docker/platys-test/default]

and ignores the structure flag in the config.yml, which it should starting with 2.2.0.

Create a Slack channel for Platys

Create a Docker image for platys

In environments, where there is no way to download and install a CLI, a dockerized Platys would be helpful.

Error when using gen with -c option

When I'm using the -c option in the gen command to specify the local config file

platys gen -c config-dc1.yml

but also if I just do a

platys gen

I'm getting the following error

2021/12/31 03:42:13 Generating stack on [/home/docker/platys-cookbook/dc1-platform]
{"status":"Pulling from trivadis/platys-modern-data-platform","id":"1.14.0-preview"}
{"status":"Digest: sha256:a80b71eb66b27426144176577e82b090ef709273b2349b24da628bfcf90d5243"}
{"status":"Status: Image is up to date for trivadis/platys-modern-data-platform:1.14.0-preview"}
2021/12/31 03:42:15 Error response from daemon: invalid mount config for type "bind": invalid mount path: 'config-dc1.yml' mount path must be absolute

it works, if I'm using an absolute path

platys gen -c /home/docker/platys-cookbook/config-dc1.yml 

After running "platys gen", the "generated" folders are owned by root

After a platys gen, all generated folders are owned by "root:root".

4 drwxr-xr-x 57 docker docker 4096 Aug 18 05:50 ..
4 drwxr-xr-x 19 root   root   4096 Aug 18 06:00 conf
4 -rwxr-xr-x  1 docker docker 1058 Aug 18 06:00 config.yml
4 drwxr-xr-x 23 root   root   4096 Aug 18 06:00 container-volume
4 drwxr-xr-x  3 root   root   4096 Aug 18 06:00 data-transfer
8 -rw-r--r--  1 root   root   7815 Aug 18 06:00 docker-compose.yml
4 drwxr-xr-x  2 root   root   4096 Aug 18 05:57 etc
4 drwxr-xr-x  8 root   root   4096 Aug 18 06:00 init
4 drwxr-xr-x  3 root   root   4096 Aug 18 06:00 licenses
4 drwxr-xr-x 10 root   root   4096 Aug 18 06:00 plugins
4 drwxr-xr-x  4 root   root   4096 Aug 18 06:00 scripts

With the python version, platys gen ran under the current user.

Check "How to deploy on remote Docker hosts with docker-compose" for usage in platys

https://www.docker.com/blog/how-to-deploy-on-remote-docker-hosts-with-docker-compose/

Wrong values for global configuration after "platys init --enable-services"

1. I'm doing a platys init --enable-services KAFKA, PRESTO -w 1.7.0

2. a config.yml is generated with the following values:

      # Default values for the generator
      # this file can be used as a template for a custom configuration
      # or to know about the different variables available for the generator
      platys:
          platform-name: 'default'
          platform-stack: 'trivadis/platys-modern-data-platform'
          platform-stack-version: '1.7.0'
          structure: 'flat'
      # ===== Global configuation, valid for all or a group of services ========
      # Timezone, use a Linux string such as Europe/Zurich or America/New_York
      use_timezone: 'true'
      # the name of the repository to use for private images, which are not on docker hub (currently only Oracle images)
      private_docker_repository_name: 'true'
      # ===== Apache Kafka ========
      KAFKA_enable: true
      
    SPARK_enable: true
      # ===== Apache Hive Metastore ========
      HIVE_METASTORE_enable: true
      # ===== Presto ========
      PRESTO_enable: true
      #=====  MinIO Object Storage ========
      MINIO_enable: true
      AWSCLI_enable: true

The value of use_timezone and private_docker_repository_name are set to true which causes the docker-compose.yml to be invalid after a platys gen.

It would be correct to use the values from the "template" config.yml.

Provide platys installation as a docker image

In an environment where it is not possible to install platys locally, it could be run as a docker container and a volume mapping to map files into and out of the container.

An image could base on python and would need docker installation.

This article explains about how to run docker inside docker, and using the (same) Docker Daemon of the outer container: https://itnext.io/docker-in-docker-521958d34efd

"init" no longer supports setting the name of the stack and the structure

The old python CLI had support for setting the name of the stack --n and the structure --structure. It would be good to have support for them.

update to 2.4.4

root@ubuntu20dockers:/tmp# platys version
Platys - Trivadis Platform in a Box - v 2.4.3
https://github.com/trivadispf/platys
Copyright (c) 2018-2020, Trivadis AG

next

sudo curl -L "https://github.com/TrivadisPF/platys/releases/download/2.4.4/platys_2.4.4_darwin_x86_64.tar.gz" -o /tmp/platys.tar.gz

ls
platys.tar.gz
snap.lxd
systemd-private-58bcbf2707b848aba19fc8bcbffc7db6-ntp.service-k7KQ4g
systemd-private-58bcbf2707b848aba19fc8bcbffc7db6-systemd-logind.service-Yvuili
systemd-private-58bcbf2707b848aba19fc8bcbffc7db6-systemd-resolved.service-LOCJ1h
vmware-root_792-2999526369

and show me this error

root@ubuntu20dockers:/tmp# tar zvxf /tmp/platys.tar.gz

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

how solve this?

platys gen with `--config-url` does not work

When using the parameter --config-url I get the following

2020/11/09 00:32:38 Unable to continue as the file [config.yml] cannot be found

Platys on Windows is limited to Python3.7

Using Platys on Windows depends on exactly Python 3.7.x.
Working on a Windows with Python 3.8 brings the following Error :

Message as Text : The code cannot proceed because python37.dll was not found.

Reproduce the Error : (Init Platys using cmd.exe. Powershell suppresses the error and just does not do anything.)
Command used in cmd-Terminal => platys init -n platys-demo-platform -sn trivadis/platys-modern-data-platform -sv 1.2.0

Proposal for fix : Check for Python >= 3.7

Please contact me if you got questions.

Regards, Aron

"platys init" does not find the banner.txt to show next steps

When doing a platys init the following error is shown:

file written successfully
panic: open /assets/init_banner.txt: no such file or directory

init_banner.txt not available in linux dist

Will create the folder with a base config file
Traceback (most recent call last):
File "/usr/local/lib/platys.dist/platys.py", line 366, in
File "/usr/local/lib/platys.dist/click/core.py", line 829, in call
File "/usr/local/lib/platys.dist/click/core.py", line 782, in main
File "/usr/local/lib/platys.dist/click/core.py", line 1259, in invoke
File "/usr/local/lib/platys.dist/click/core.py", line 1066, in invoke
File "/usr/local/lib/platys.dist/click/core.py", line 610, in invoke
File "/usr/local/lib/platys.dist/platys.py", line 212, in init
File "/usr/local/lib/platys.dist/platys.py", line 359, in print_banner
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/platys.dist/init_banner.txt'

if there is an error in the generator (inside docker-compose-templer), it is not shown on the Platys console

After an init and gen I'm getting an error

When doing an platys init followed by a platys gen I'm getting an

2020/08/07 05:36:43 The config file is not properly formatted or missing information please ensure [platform-name], [stack-image-name] and [stack-image-version] are properly configured

error.

Here the head of the config.yml

      # Default values for the generator
      # this file can be used as a template for a custom configuration
      # or to know about the different variables available for the generator
      platys:
          platform-name: 'platys-demo-platform'
          stack-image-name: 'trivadis/platys-modern-data-platform'
          stack-image-version: '1.6.0'
          structure: 'flat'

This is also the case when I'm using an config.yml from the previous version (python CLI).

--platform-name does not end up in config.yml

Using the --platform-name flag in the init does not reflect in config.yml

      platys:
        platform-name: 'default'
        stack-image-name: 'trivadis/platys-modern-data-platform'
        stack-image-version: '1.5.0'
        structure: 'flat'

The name is still on default.

Change the way we check for maximum number of services instances

Currently the number of services to be supported is hardcoded in the platys.py script. No with the separation of platys and the platys-modern-data-platform stack into independent projects, platys should not have any knowledge of the platform stack. So we need to define a way, where platys can ask a platform stacks about limits and then apply these dynamically.

platys init creates the config.yml with no rw rights

A platys init --platform-name platys-demo-platform --stack-version 1.6.0 -f --structure flat creates the config.yml with the right owner but without any rights on it:

docker@ubuntu:~/platys-test$ ls -lsa
total 20
 4 drwxr-xr-x  2 docker docker 4096 Aug  7 05:41 .
 4 drwxr-xr-x 57 docker docker 4096 Aug  7 05:14 ..
12 ----------  1 docker docker 8691 Aug  7 05:41 config.yml

platys init removes comments from config.yml

The old version left the config.yml downloaded by a platys init as the original. I would prefer that, as the config.yml holds some important comments as well as new lines to separate stuff.

Add: Web UI for enable or disable all modern stack, something like CLOUDERA but for platys

Add: Web UI for enable or disable all modern stack, something like CLOUDERA but for platys

Avoid error when doing a gen

Currently the following error is shown when executing a platys geneven thought generation is working perfectly:

b"cp: can't stat '/opt/mdps-gen/conf/config.yml': No such file or directory"

We should remove it as it might be misleading for a new user of platys.

When running "platys gen", platys does not use the image updated in the local docker registry

1. I have created a new version of the platform-stack locally using a docker build .....
2. I have updated the platform stack version in the config.yml
3. When running platys gen I'm getting an error

panic: Error response from daemon: manifest for trivadis/platys-modern-data-platform:1.8.0-lpv not found: manifest unknown: manifest unknown

I guess that the docker command inside platys is not running under the local user but maybe under root?

Enhance --version string

Change the version to the following

Trivadis Platform in a Box - v2.2.0
https://github.com/trivadispf/platys
Copyright (c) 2018-2020, Trivadis AG

VERBOSE flag does not seem to get passed to the platform stack (docker image)

in a platys gen the VERBOSE parameter should be uppercase, guess it is mixed-case according to the go code:

		if Verbose {
			env = append(env, "Verbose=1")
		} else {
			env = append(env, "Verbose=0")
		}

Add Python 3.x and JDK to Vagrant setup

Extend documentation header in `platys -h`

Add the same header as in platys version to platys -h

$ platys -h
Platys - Trivadis Platform in a Box - v2.3.0
https://github.com/trivadispf/platys
Copyright (c) 2018-2020, Trivadis AG

Usage:
  platys [flags]
  platys [command]

Available Commands:
  clean         Cleans the contents in the $PATH/container-volume folder
  gen           Generates all the needed artifacts for the docker-based modern (data) platform
  help          Help about any command
  init          Initializes the current directory to be the root for the Modern (Data) Platform by creating an initial config file, if one does not already exists
  list_services List the services
  stacks        Lists the predefined stacks available for the init command
  version       Print the version number of platys

Flags:
  -h, --help                   help for platys
  -s, --stack string           stack version to employ (default "trivadis/platys-modern-data-platform")
  -w, --stack-version string   version of the stack to employ (default "latest")
  -v, --verbose                verbose output (default true)

Use "platys [command] --help" for more information about a command.

Add support to show the config.yml on the console without storing it on disk

I would be nice to add a command to only display the config.yml from a stack on the console (stdout) without downloading the file to the local filesystem.

platys info --stack trivadis/platys-modern-data-platform --stack-version 1.17.0

This would support the use case where a stack is generated "on the fly".

Create a checklist for release of Platys

platys command line without any arguments could show help page?

If I enter platys without any arguments, nothing is shown. Would it be possible to default to platys -h?

Missing flag for akhq

In the generated docker-compose.yml a flag for akhq is missing.

          connect:
            - name: "connect"
              url: "http://kafka-connect-1:8083"

This leads to:
Message: Error resolving field value [akhq.connections.docker-kafka-server.connect]. Property doesn't exist or cannot be converted

Generated docker-compose:

  akhq:
    image: tchiotludo/akhq:latest
    container_name: akhq
    hostname: akhq
    labels:
      com.mdps.service.webui.url: http://${PUBLIC_IP}:28107
    ports:
      - 28107:8080
    environment:
      AKHQ_CONFIGURATION: |
        akhq:
          connections:
            docker-kafka-server:
              properties:
                bootstrap.servers: "kafka-1:19092"
              schema-registry:
                url: "http://schema-registry-1:8081"
              connect:
                 url: "http://kafka-connect-1:8083"
    depends_on:
      - kafka-1
    volumes:
      - ./data-transfer:/data-transfer
    restart: unless-stopped

fixed docker-compose:

  akhq:
    image: tchiotludo/akhq:latest
    container_name: akhq
    hostname: akhq
    labels:
      com.mdps.service.webui.url: http://${PUBLIC_IP}:28107
    ports:
      - 28107:8080
    environment:
      AKHQ_CONFIGURATION: |
        akhq:
          connections:
            docker-kafka-server:
              properties:
                bootstrap.servers: "kafka-1:19092"
              schema-registry:
                url: "http://schema-registry-1:8081"
              connect:
                - name: "connect"
                  url: "http://kafka-connect-1:8083"
    depends_on:
      - kafka-1
    volumes:
      - ./data-transfer:/data-transfer
    restart: unless-stopped

Add more verbose output to `platys version`

Change the output of the platys version command to

$ platys version
Platys - Trivadis Platform in a Box - v2.3.0
https://github.com/trivadispf/platys
Copyright (c) 2018-2020, Trivadis AG

would it be possible to replace/set the version number automatically from the build?

the output of the init does not properly expand the %%path%% variable

A platys init output does not expand the %%path%% variable

docker@ubuntu:~/platys-test$ platys init --platform-name platys-demo-platform --stack-version 1.6.0 -f --structure flat
Running using config file [config.yml]
{"status":"Pulling from trivadis/platys-modern-data-platform","id":"latest"}
{"status":"Digest: sha256:c10d21ec4c10e89c27a12767843bc13034f1079f7cd87f20b6c87fc6bea5d062"}
{"status":"Status: Image is up to date for trivadis/platys-modern-data-platform:latest"}
  ______     _                 ___
 /_  __/____(_)   ______ _____/ (_)____
  / / / ___/ / | / / __ `/ __  / / ___/
 / / / /  / /| |/ / /_/ / /_/ / (__  )
/_/ /_/  /_/ |___/\__,_/\__,_/_/____/
    __  ___          __                   ____        __           ____  __      __  ____
   /  |/  /___  ____/ /__  _________     / __ \____ _/ /_____ _   / __ \/ /___ _/ /_/ __/___  _________ ___
  / /|_/ / __ \/ __  / _ \/ ___/ __ \   / / / / __ `/ __/ __ `/  / /_/ / / __ `/ __/ /_/ __ \/ ___/ __ `__ \
 / /  / / /_/ / /_/ /  __/ /  / / / /  / /_/ / /_/ / /_/ /_/ /  / ____/ / /_/ / /_/ __/ /_/ / /  / / / / / /
/_/  /_/\____/\__,_/\___/_/  /_/ /_/  /_____/\__,_/\__/\__,_/  /_/   /_/\__,_/\__/_/  \____/_/  /_/ /_/ /_/

Congratulations! Your new Platform has been initialized in %%path%%

Errors on using Platys in WSL2 using Ubuntu 18.04

Using Platys on Windows Subsystem for Linux (WSL) specifically the actual WSL2 using Ubuntu 18.04.
The errors seems not to be bound to WSL, it is more a Python-Error. Could be in the dependencies.

Error-Output :
aron@TVD-AAWELLER:/mnt/c/Users/arona$ sudo curl -L "https://github.com/TrivadisPF/platys/releases/download/2.2.0/platys.tar.gz" -o /tmp/platys.tar.gz
[sudo] password for aron:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 626 100 626 0 0 1187 0 --:--:-- --:--:-- --:--:-- 1187
100 14.6M 100 14.6M 0 0 4408k 0 0:00:03 0:00:03 --:--:-- 5244k
aron@TVD-AAWELLER:/mnt/c/Users/arona$ tar zvxf /tmp/platys.tar.gz
udo mv platys.dist/ /usr/local/lib/
sudo chown -R root:root /usr/local/lib/platys.dist/
sudo rm /tmp/platys.tar.gzplatys.dist/
platys.dist/_asyncio.so
platys.dist/_bisect.so
platys.dist/_blake2.so
....
aron@TVD-AAWELLER:/mnt/c/Users/arona$ sudo mv platys.dist/ /usr/local/lib/
aron@TVD-AAWELLER:/mnt/c/Users/arona$ sudo chown -R root:root /usr/local/lib/platys.dist/
aron@TVD-AAWELLER:/mnt/c/Users/arona$ sudo rm /tmp/platys.tar.gz
aron@TVD-AAWELLER:/mnt/c/Users/arona$ sudo ln -s /usr/local/lib/platys.dist/platys /usr/local/bin/platys
aron@TVD-AAWELLER:/mnt/c/Users/arona$ platys --version
platys - Trivadis Platform in a Box - v2.2.0
https://github.com/trivadispf/platys
Copyright (c) 2018-2020, Trivadis AG
aron@TVD-AAWELLER:/mnt/c/Users/arona$ mkdir -v platys-test
mkdir: created directory 'platys-test'
aron@TVD-AAWELLER:/mnt/c/Users/arona$ cd platys-test
aron@TVD-AAWELLER:/mnt/c/Users/arona/platys-test$ platys init -n platys-platform -sn trivadis/platys-modern-data-platform -sv 1.5.2 --structure flat
**Will create the folder with a base config file
Traceback (most recent call last):
File "/usr/local/lib/platys.dist/urllib3/connectionpool.py", line 670, in urlopen
File "/usr/local/lib/platys.dist/urllib3/connectionpool.py", line 392, in _make_request
File "/usr/local/lib/platys.dist/http/client.py", line 1254, in request
File "/usr/local/lib/platys.dist/http/client.py", line 1300, in _send_request
File "/usr/local/lib/platys.dist/http/client.py", line 1249, in endheaders
File "/usr/local/lib/platys.dist/http/client.py", line 1036, in _send_output
File "/usr/local/lib/platys.dist/http/client.py", line 974, in send
File "/usr/local/lib/platys.dist/docker/transport/unixconn.py", line 43, in connect
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/platys.dist/requests/adapters.py", line 439, in send
File "/usr/local/lib/platys.dist/urllib3/connectionpool.py", line 724, in urlopen
File "/usr/local/lib/platys.dist/urllib3/util/retry.py", line 403, in increment
File "/usr/local/lib/platys.dist/urllib3/packages/six.py", line 734, in reraise
File "/usr/local/lib/platys.dist/urllib3/connectionpool.py", line 670, in urlopen
File "/usr/local/lib/platys.dist/urllib3/connectionpool.py", line 392, in _make_request
File "/usr/local/lib/platys.dist/http/client.py", line 1254, in request
File "/usr/local/lib/platys.dist/http/client.py", line 1300, in _send_request
File "/usr/local/lib/platys.dist/http/client.py", line 1249, in endheaders
File "/usr/local/lib/platys.dist/http/client.py", line 1036, in _send_output
File "/usr/local/lib/platys.dist/http/client.py", line 974, in send
File "/usr/local/lib/platys.dist/docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/platys.dist/platys.py", line 366, in
File "/usr/local/lib/platys.dist/click/core.py", line 829, in call
File "/usr/local/lib/platys.dist/click/core.py", line 782, in main
File "/usr/local/lib/platys.dist/click/core.py", line 1259, in invoke
File "/usr/local/lib/platys.dist/click/core.py", line 1066, in invoke
File "/usr/local/lib/platys.dist/click/core.py", line 610, in invoke
File "/usr/local/lib/platys.dist/platys.py", line 173, in init
File "/usr/local/lib/platys.dist/platys.py", line 331, in pull_config
File "/usr/local/lib/platys.dist/docker/models/containers.py", line 802, in run
File "/usr/local/lib/platys.dist/docker/models/containers.py", line 861, in create
File "/usr/local/lib/platys.dist/docker/api/container.py", line 430, in create_container
File "/usr/local/lib/platys.dist/docker/api/container.py", line 440, in create_container_from_config
File "/usr/local/lib/platys.dist/docker/api/client.py", line 289, in _post_json
File "/usr/local/lib/platys.dist/docker/utils/decorators.py", line 46, in inner
File "/usr/local/lib/platys.dist/docker/api/client.py", line 226, in _post
File "/usr/local/lib/platys.dist/requests/sessions.py", line 578, in post
File "/usr/local/lib/platys.dist/requests/sessions.py", line 530, in request
File "/usr/local/lib/platys.dist/requests/sessions.py", line 643, in send
File "/usr/local/lib/platys.dist/requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))**

Proposal for fix : Seems that Python3 is needed for using Platys. A possible fix would be to use the shebang refering to python3. Alternative fix : Enable a Parameter for playts in which i can specify the Python-Version to use.

Please contact me if you got questions.

Regards,
Aron

Migrate URL handling to the cli

Currently downloading config file from URL is done in the docker container but it should be done in the cli (--config-url) is only passed to the docker image

Should platys respect docker context settings?

Docker has a concept of docker contexts and one of many context can be set as in use.

docker context use <context>

Currently platys uses either unix:///var/run/docker.sock as the default for the docker daemon or the value of DOCKER_HOST. The problem of DOCKER_HOST is, that is not linked to docker context and if it is set, then the docker context can no longer be used to switch and we get the following error: Warning: DOCKER_HOST environment variable overrides the active context. To use "orbstack", either set the global --context flag, or unset DOCKER_HOST environment variable.

The active context is stored in .docker/config.json with the meta data of each context in .docker/contexts/meta/<id>/meta.json.

Fix documenation of "platys init --help"

The documentation of platys init --help should be corrected to:

Initializes the current directory to be the root for a platys platform by creating an initial
		config file, if one does not already exists The stack to use as well as its version need to be passed by the --stack and --stack-version options.
			By default 'config.yml' is used for the name of the config file, which is created by the init

Usage:
  platys init [flags]

Flags:
  -y, ----enable-services string   List of services to enable in the config file
  -f, ----force                    If specified, this command will overwrite any existing config file
  -x, ----hw-arch string           Hardware architecture for the platform (default "x86-64")
  -e, ----seed-config string       the name of a predefined stack to base this new platform on
  -c, --config-file string         The name of the local config file (defaults to config.yml) (default "config.yml")
  -h, --help                       help for init

Global Flags:
  -s, --stack string           platform stack to provision (default "trivadis/platys-modern-data-platform")
  -w, --stack-version string   version of the platform stack to provision (default "latest")
  -v, --verbose                verbose output (default true)

Why is --stack and --stack-version a global flag?

--stackand --stack-version is not used/does not make sense in the platys gen, so I'm not sure if it is good to have it as a global flag (miss-leading). Or do you overwrite the setting in the config.yml if these are used in the platys gen?

Could we also grab other values when using the "--enable-services" in a "platys init"

When using the --enable-services of the platys init, the XXXX_enabled flags from the template config.yml are taken and written with the value of true into the generated config.yml.

It would be nice to also get the other values belonging to the service into the config.yml file, for example with a platys init --enable-services KAFKA -w 1.7.0 the following values would be added to the resulting config.yml file:

      KAFKA_enable: true
      # one of enterprise, community
      KAFKA_edition: 'community'
      KAFKA_volume_map_data: false
      KAFKA_broker_nodes: 3
      KAFKA_internal_replication_factor: 3
      KAFKA_delete_topic_enable: false
      KAFKA_auto_create_topics_enable: false

Of course for this to work, we would need to have some convention for what belongs together:

1. until the next XXXX_enable property, which for the example above would be KAFKA_SCHEMA_REGISTRY_enable (but then we would include a possible comment just before it).
2. we assume that all other property with KAFKA_xxxxx naming convention belong to the KAFKA service (which is the way it is currently done)

Think I would favor the 2nd option.