michalklempa/docker-nifi-registry

Unofficial Docker Image For NiFi Registry

docker-nifi-registry's Issues

SSH key has to be passwordless

Update README:
https://lists.apache.org/thread.html/5f21e6faea71d2f17cb597ced11f6ec1ac21501773b0b8777bcbc46a@%3Cusers.nifi.apache.org%3E

Implement templating of bootstrap.conf especially to enable Xmx and debugging

Provide a way to run with java debugging enabled and to configure memory requirements.

Consider implementing symlinks for stdout and stderr

According to https://docs.docker.com/config/containers/logging/ there are multiple possible ways to capture logfiles.

Consider implementing one of these in this image.

nginx way:

https://github.com/nginxinc/docker-nginx/blob/8921999083def7ba43a06fabd5f80e4406651353/mainline/jessie/Dockerfile#L21-L23

httpd way:

https://github.com/docker-library/httpd/blob/b13054c7de5c74bbaa6d595dbe38969e6d4f860c/2.2/Dockerfile#L72-L75

This would allow us to avoid forking tail process.

Make symlink /opt/nifi-registry/nifi-latest or provide other way for users to target /conf and /database

Without knowing the subpath (PROJECT_HOME). Users should not be forced to change their manifests and compose files when new upstream version releases.

Consider using gosu or su-exec to specify UID:GID in runtime not build time

https://github.com/tianon/gosu/blob/master/INSTALL.md

Error cloning repository with default `FLOW_PROVIDER_GIT_REMOTE_TO_PUSH` as blank

The documentation indicates that FLOW_PROVIDER_GIT_REMOTE_TO_PUSH is default to empty. This is a valid use-case in production deployment when wanting to access the read-only Git repository as the GitFlowPersistenceProvider.

Current implementation throws an error when setting FLOW_PROVIDER_GIT_REMOTE_TO_PUSH to blank due to the git clone command explicitly specifying the --origin or -o flag.

nifi-registry_1  | git clone -o $FLOW_PROVIDER_GIT_REMOTE_TO_PUSH -b $GIT_CHECKOUT_BRANCH $GIT_REMOTE_URL $FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY


nifi-registry_1  | fatal: Too many arguments.
nifi-registry_1  |
nifi-registry_1  | usage: git clone [<options>] [--] <repo> [<dir>]
nifi-registry_1  |     -o, --origin <name>   use <name> instead of 'origin' to track upstream

It may be necessary to check for empty FLOW_PROVIDER_GIT_REMOTE_TO_PUSH and exclude the entire option for -o $FLOW_PROVIDER_GIT_REMOTE_TO_PUSH from the git clone command.

Add support to template registry-aliases.xml

gitflowpersistanceprovider doesnot show the gitflow in gitlab

@michalklempa Hi , i have tried the dockerfile for gitlab persistance provider , i am running it on kubernetes

Here is the docker image i am using

FROM michalklempa/nifi-registry

ENV FLOW_PROVIDER=git \
    GIT_REMOTE_URL=https://*************/nifi-registry-git.git \
    GIT_CHECKOUT_BRANCH=  \
    FLOW_PROVIDER_GIT_REMOTE_ACCESS_USER=amx-bot \
    FLOW_PROVIDER_GIT_REMOTE_ACCESS_PASSWORD=******** \
    GIT_CONFIG_USER_NAME=amx-bot \
    GIT_CONFIG_USER_EMAIL=amx-bot@*******

i get below logs after running the dockerfile

Loading...FLOW_PROVIDER_GIT_REMOTE_ACCESS_PASSWORD is set, trying to set git credential helper for HTTPS password
git config --global credential.${GIT_REMOTE_URL}.helper '!f() { sleep 1; echo -e "username=${FLOW_PROVIDER_GIT_REMOTE_ACCESS_USER}\npassword=*****"; }; f'
Found git remote: https:/********/nifi-registry-git.git, cloning into: /opt/nifi-registry/flow-storage, with remote:  and branch: 
git clone -o $FLOW_PROVIDER_GIT_REMOTE_TO_PUSH -b $GIT_CHECKOUT_BRANCH $GIT_REMOTE_URL $FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY
Cloning into '/opt/nifi-registry/flow-storage'...
Found key: GIT_CONFIG_USER_EMAIL for git config
Setting git config: user.email=**********
git config -f ${FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY}/.git/config ${KEY} '${VALUE}'
Found key: GIT_CONFIG_USER_NAME for git config
Setting git config: user.name=amx-bot
git config -f ${FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY}/.git/config ${KEY} '${VALUE}'
NiFi-Registry running with PID 32.
tail: can't open '/opt/nifi-registry/nifi-registry-0.4.0/logs/nifi-registry-app.log': No such file or directory

Java home: /usr/lib/jvm/java-1.8-openjdk
NiFi Registry home: /opt/nifi-registry/nifi-registry-0.4.0

Bootstrap Config File: /opt/nifi-registry/nifi-registry-0.4.0/conf/bootstrap.conf

And the nifi-registry starts initializing.

When i give the http://localhost:18080 in nifi to connect to registry, it is able to see the buckets in nifi-registry , after adding the version control to the nifi i can see the commit changes in the nifi-registry but i am unable to see this changes in the gitlab repo.

it is am empty repository and even after commiting changes to nifi-regsitry it is still empty as i understand i should be able to see the commits in gitlab repo ? please let me know if i am missing anything .
Thanks

authorizers.xml always overwritten

I wanted to change the authorizers.xml based on my needs.

What I did:

1. Run docker run command

 docker run --name nifi-registry \
      -v nifi-registry-certs:/opt/certs \
      -v nifi-registry-conf:/opt/nifi-registry/nifi-registry-0.5.0/conf \
      -p 18443:18443 \
      -e 'NIFI_REGISTRY_SECURITY_KEYSTORE=/opt/certs/keystore.jks' \
      -e 'NIFI_REGISTRY_SECURITY_KEYSTOREtYPE=JKS' \
      -e 'NIFI_REGISTRY_SECURITY_KEYSTOREpASSWD=pw' \
      -e 'NIFI_REGISTRY_SECURITY_TRUSTSTORE=/opt/certs/truststore.jks' \
      -e 'NIFI_REGISTRY_SECURITY_TRUSTSTOREtYPE=JKS' \
      -e 'NIFI_REGISTRY_SECURITY_TRUSTSTOREpASSWD=pw' \
      -e 'NIFI_REGISTRY_WEB_HTTP_HOST=' \
      -e 'NIFI_REGISTRY_WEB_HTTP_PORT=' \
      -e 'NIFI_REGISTRY_WEB_HTTPS_HOST=0.0.0.0' \
      -e 'NIFI_REGISTRY_WEB_HTTPS_PORT=18443' \
      -e 'INITIAL_ADMIN_IDENTITY=CN=...' \
      -e 'NIFI_REGISTRY_SECURITY_IDENTITY_PROVIDER=ldap-identity-provider' \
      -e 'NIFI_REGISTRY_SECURITY_NEEDcLIENTaUTH=false' \
      -e LDAP_URL='ldap://...' \
      -e LDAP_MANAGER_DN='cn=...' \
      -e LDAP_MANAGER_PASSWORD='pw' \
      -e LDAP_AUTHENTICATION_STRATEGY='SIMPLE' \
      -e LDAP_USER_SEARCH_BASE='dc=nifi,dc=com' \
      -e LDAP_USER_SEARCH_FILTER='cn={0}' \
      -e 'FLOW_PROVIDER=git' \
      -e '[email protected]' \
      -e 'FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY=/opt/nifi-registry/flow-storage-git' \
      -e 'GIT_CONFIG_USER_NAME=name' \
      -e 'GIT_CONFIG_USER_EMAIL=mail' \
      -e 'SSH_PRIVATE_KEY='$(base64 -w 0 < ~/.ssh/id_rsa) \
      -e 'SSH_KNOWN_HOSTS='$(base64 -w 0 < ~/.ssh/known_hosts) \
      -e 'SSH_PRIVATE_KEY_PASSPHRASE=' \
      -d \
      michalklempa/nifi-registry:latest

2. I checked the authorizers.xml file (and figured out that I need to modify some authz stuff*)
3. Did the changes in authorizers.xml
4. Restart the container
5. Changes are gone :-(

I haven't tested all files in the conf volume, but I guess it is a general overwriting.

What I would like to set:

authorizers.xml

• ldap-user-group-provider

• <accessPolicyProvider>
    <property name="User Group Provider">composite-configurable-user-group-provider
  

• <userGroupProvider>
    <identifier>composite-configurable-user-group-provider
  

nifi-registry.properties file

• nifi.security.identity.mapping.pattern.dn=^cn=(.?),o=(.?),dc=(.?),dc=(.?)$
• nifi.security.identity.mapping.value.dn=$1

Add support for 0.5.0 release

Provide :develop image based on master branch of upstream

Provide a way to release the :develop docker tag using build
(maybe https://travis-ci.org/apache/nifi-registry/branches) of master branch in upstream.

SSH key truncated during Base64 decoding

I encountered the following error during base64 decoding: base64: truncated base64 input.
See longer output below.

My steps for testing the Docker image.

• Generate the password-less SSH key: ssh-keygen -t rsa -b 4096 -N '' -C "nifi_registry_docker_test" -f id_rsa_nifi_registry_test
• Add public key to Git repository access key.
• Use the docker-compose.github.yml as starting point.
• Use mount point binding or adjust settings for environment variables as appropriate with the following SSH variables:

    environment:
      DEBUG: 1
      # ...
      # Use of $$ allows you to refer to environment variables that you don’t want processed by Compose.
      SSH_PRIVATE_KEY: "$$(base64 -w 0 < ~/.ssh/id_rsa_nifi_registry_test)"
      SSH_KNOWN_HOSTS: "$$(base64 -w 0 < ~/.ssh/known_hosts)"
      SSH_PRIVATE_KEY_PASSPHRASE:

• Run docker-compose up

Output with error below (similar error for using SSH mount point):

nifi-registry_1  | SSH_KNOWN_HOSTS=$(base64 -w 0 < ~/.ssh/known_hosts)
nifi-registry_1  | SSH_PRIVATE_KEY=$(base64 -w 0 < ~/.ssh/id_rsa_nifi_registry_test)
nifi-registry_1  | _=/usr/bin/env
nifi-registry_1  | End of debug output
nifi-registry_1  | SSH_PRIVATE_KEY_FILE=$HOME/.ssh/id_rsa
nifi-registry_1  | SSH_KNOWN_HOSTS_FILE=$HOME/.ssh/known_hosts
nifi-registry_1  | mkdir -p $HOME/.ssh && chmod 700 $HOME/.ssh
nifi-registry_1  | echo -n "${SSH_PRIVATE_KEY}" | base64 -d > $SSH_PRIVATE_KEY_FILE && chmod 600 "${SSH_PRIVATE_KEY_FILE}"
nifi-registry_1  | base64: truncated base64 input
nifi-registry_1  | ssh-keygen ${SSH_PRIVATE_KEY_PASSPHRASE:+'-P' "${SSH_PRIVATE_KEY_PASSPHRASE}"} -y -f ${SSH_PRIVATE_KEY_FILE} > ${SSH_PRIVATE_KEY_FILE}.pub && chm
od 600 ${SSH_PRIVATE_KEY_FILE}.pub
nifi-registry_1  | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
nifi-registry_1  | @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
nifi-registry_1  | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
nifi-registry_1  | Permissions 0644 for '/home/nifi/.ssh/id_rsa' are too open.
nifi-registry_1  | It is required that your private key files are NOT accessible by others.
nifi-registry_1  | This private key will be ignored.
nifi-registry_1  | Load key "/home/nifi/.ssh/id_rsa": bad permissions
nifi-registry_1  | echo -n ${SSH_KNOWN_HOSTS} | base64 -d > $SSH_KNOWN_HOSTS_FILE && chmod 600 $SSH_KNOWN_HOSTS_FILE
nifi-registry_1  | base64: truncated base64 input
nifi-registry_1  | Found git remote: ssh://[email protected]/flow-storage.git, cloning into: /opt/nifi-registry/flow-storage, with remote: origin and
branch: master
nifi-registry_1  | git clone -o $FLOW_PROVIDER_GIT_REMOTE_TO_PUSH -b $GIT_CHECKOUT_BRANCH $GIT_REMOTE_URL $FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY
nifi-registry_1  | Cloning into '/opt/nifi-registry/nifi-flows'...
nifi-registry_1  | load pubkey "/home/nifi/.ssh/id_rsa": invalid format
nifi-registry_1  | Host key verification failed.
nifi-registry_1  | fatal: Could not read from remote repository.
nifi-registry_1  |
nifi-registry_1  | Please make sure you have the correct access rights
nifi-registry_1  | and the repository exists.

When the SSH key decoding from Base64 got truncated, it created cascading problems.

I ran a manual test in my local Cygwin environment and verified that the base64 encoding and decoding steps are working properly for my generated SSH key.

TEST_SSH_KEY=$(base64 -w 0 < ~/.ssh/id_rsa_nifi_registry_test)
cd ~/.ssh/
echo -n ${TEST_SSH_KEY} | base64 -d > test_id
diff test_id id_rsa_nifi_registry_test

The problem seems to reside with the base64 version in the Alpine base image.

How to pull the github repository automatically?

I have two different instances of Nifi Registry running. One in a dev environment and the other one in a prod environment. Both pointing to the same github repository and running on a Azure Kubernetes cluster.
When I commit my changes in a process group from dev env, a new version is created. The change is pushed and I can see it in the github repo. But the pro instance is never aware of that change, so I'm no able to upgrade that process group with the new version. Even by running a git pull in the prod container shell. When I do that I see the log of the pull, the change is there, but Nifi doesn't notice it.
Only when I restart the prod instance of Nifi Registry I am able to see that there is another available version of that process group and upgrade it.
Is there any way to automatize it so, both registry instances are always synchronized?

The version of the docker image I'm using is nifi-registry:0.8.0-default

thank you very much in advance

Provide a way to extract H2 database contents by using named volume

Where does H2 store data? Can we provide a way to not loose the data if container is destroyed?
This can be useful for Kubernetes and docker swarm environments.

Cant access to Rest-API

Hello Michal. Thank you for realy cool image.

I can't access to NiFi registry Rest Api. The base adress is /nifi-registry-api. May be you can enable this feature in image?

I saw nifi-registry-web-api-0.8.0.war in jetty folder. Any idea?

My compose file:

networks:
  nifi-network:
    driver: bridge
services:
  nifi-registry:
    image: michalklempa/nifi-registry
    networks:
      - nifi-network
    container_name: nifi-registry
    ports:
      - target: 18080
        published: 18080
        protocol: tcp
        mode: host
      - target: 18081
        published: 18081
        protocol: tcp
        mode: host
    environment:
      DEBUG: 1
      FLOW_PROVIDER: git
      GIT_REMOTE_URL:  myURL
      GIT_CHECKOUT_BRANCH: TestFlowDevelop
      FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY: /opt/nifi-registry/nifi-registry-0.8.0/flow-storage
      FLOW_PROVIDER_GIT_REMOTE_TO_PUSH: origin
      FLOW_PROVIDER_GIT_REMOTE_ACCESS_USER: i.klimenko
      FLOW_PROVIDER_GIT_REMOTE_ACCESS_PASSWORD: mypassword
      NIFI_REGISTRY_WEB_HTTPS_HOST: 18081```
  

Wrong timezone in container

When I start the container I notice that it is running in a wrong timezone - "date" says it's two hours earlier the in reality in Munich.
Is there a way to tell the container which timezone to use? E.g. in the jenkins-container I can use
-e JAVA_OPTS=-Duser.timezone=Europe/Berlin
but this doesn't work for docker-nifi-registry.

Unable to chmod temp file

Hello,

currently I am trying to deploy NiFi-Registry in OpenShift 4.1 with this Template.
Sadly I am getting this error when starting the container:
unable to chmod temp file: chmod /opt/nifi-registry/nifi-registry-0.5.0/conf/nifi-registry.properties: operation not permitted

In forehand I already had to edit the Dockerfile and add RUN chmod -R 777 ${PROJECT_BASE_DIR} to have the right permission to execute the start.sh script.

Do you have any Clue?

Thanks in advance,
Andreas

Provide providers.xml externally

Hello,

if i try to provide "providers.xml" by mounting file, i get error:

unable to chmod temp file: chmod /opt/nifi-registry/nifi-registry-0.5.0/conf/providers.xml: operation not permitte

the docs say:

providers.xml is templated iff FLOW_PROVIDER is set

but it looks like providers.xml is always templated because these environment variables are set in dockerfile:

ENV FLOW_PROVIDER file
ENV FLOW_PROVIDER_GIT_FLOW_STORAGE_DIRECTORY $PROJECT_BASE_DIR/flow-storage
ENV FLOW_PROVIDER_FILE_FLOW_STORAGE_DIRECTORY $PROJECT_BASE_DIR/flow-storage

Prepare configuration for new release 0.4.0

username and password login is asked though i dnt provide "nifi.registry.security.identity.provider"

Hi ,

I am able to get the nifi-registry working on https by following :

FROM michalklempa/nifi-registry
COPY ./certs/*.jks /opt/certs/

ENV NIFI_REGISTRY_SECURITY_KEYSTORE=/opt/certs/keystore.jks \
        NIFI_REGISTRY_SECURITY_KEYSTOREtYPE=JKS \
        NIFI_REGISTRY_SECURITY_KEYSTOREpASSWD=***********  \
        NIFI_REGISTRY_SECURITY_TRUSTSTORE=/opt/certs/truststore.jks \
        NIFI_REGISTRY_SECURITY_TRUSTSTOREtYPE=JKS
        NIFI_REGISTRY_SECURITY_TRUSTSTOREpASSWD=***** \
        NIFI_REGISTRY_SECURITY_NEEDcLIENTaUTH=false \
        NIFI_REGISTRY_WEB_HTTP_HOST= \
        NIFI_REGISTRY_WEB_HTTP_PORT= \
        NIFI_REGISTRY_WEB_HTTPS_HOST= \
        NIFI_REGISTRY_WEB_HTTPS_PORT=18443 \
        INITIAL_ADMIN_IDENTITY=CN=ArborMetrix,OU=DevOps

later this UI link https://amis.arbormetrix.com/nifi-registry/ redirects to https://amis.arbormetrix.com/nifi-registry/login/ as asks for username and password.

Not sure whats going on , I later tried providing ldap identity but it doesnot displays properties settings at the top right in the UI.

Explore ssh-agent socket option for providing SSH keys

See https://stackoverflow.com/a/36648428/3944551
Explore and possibly document this alternative of providing authentication data.

If possible, provide database flavored images

MariaDB with driver baked in (/opt/nifi-registry/libs/mariadb-java-client-2.3.0.jar):

michalklempa/nifi-registry:0.3.0-mariadb-2.3.0-XX

wait for flyway/flyway#2289 (comment) before using 2.4.1 driver.

MySQL with driver baked in (/opt/nifi-registry/libs/mysql-connector-java-5.1.47.jar):

michalklempa/nifi-registry:0.3.0-mysql-5.1.47-XX

PostgreSQL with driver baked in (/opt/nifi-registry/libs/postgresql-42.2.5.jar):

michalklempa/nifi-registry:0.3.0-postgresql-42.2.5-XX