infinispan / infinispan-images Goto Github PK
View Code? Open in Web Editor NEWInfinispan is an open source data grid platform and highly scalable NoSQL cloud data store.
Home Page: https://infinispan.org
License: Apache License 2.0
Infinispan is an open source data grid platform and highly scalable NoSQL cloud data store.
Home Page: https://infinispan.org
License: Apache License 2.0
We should provide a docker image containing a native build of Infinispan server https://github.com/infinispan/infinispan-quarkus-server. This should significantly reduce the size of the image as a JVM will no longer be required.
We should introduce a env variable, called MANAGED_ENV
, that when true indicates that the image is being executed in a controlled envrionment with an orchestrator. The most obvious example being a Kubernetes operator.
When MANAGED_ENV=true
, we should disable image usability features that are only relevant when the image is used directly by users. Features to disable.
This is a blocker in order to resolve #6
The /opt/infinispan/server/conf/*
files currently have the wrong permissions set, with r
only available for non-owners. Instead the files should have r+w
permissions to allow for the files to be overridden by users and the Infinispan containers testsuite.
Currently we create a sym link to a /opt/infinispan
when the ISPN_HOME is overridden by dg-override.yaml
, so that it's possible for users to access the server via ISPN_HOME
dir, however this causes permission denied errors when ran in more restrictive environments such as openshift. Therefore, we should remove this link as it provides minimal value and always utilise the /opt/infinispan
directory directly.
https://issues.redhat.com/browse/ISPN-11131 changed the server logging to utilise a log4j.xml file instead of logging.properties
.
The check to see if the cluster is HEALTHY in the readiness probe doesn't make sense, because if the cluster becomes DEGRADED then the server(s) would be killed and consequently date would be lost.
If the server is able to handle REST requests then it is both live & ready. Therefore, we should remove our custom probe implementations and utilise the kubernetes httpGet
probes.
livenessProbe:
httpGet:
path: /rest/v2/cache-managers/clustered/health/status
port: infinispan
readinessProbe:
httpGet:
path: /rest/v2/cache-managers/clustered/health/status
port: infinispan
@rigazilla Can you update the operator yaml to utilise httpGet
and verify that this functions at expected?
It should be possible for users to provide a complete infinispan.xml
to the server before startup.
https://infinispan.org/docs/stable/titles/rest/rest.html#rest_server_cors
Suggested format:
endpoints:
rest:
auth: true
enabled: true
cors:
- name: restrict-host1
allowCredentials: false
origins:
- http://host1
- https://host1
methods:
- GET
- name: allow all
allowCredentials: true
origins:
- '*'
methods:
- GET
- OPTIONS
- POST
- PUT
- DELETE
exposeHeaders:
- Key-Content-Type
maxAgeSeconds: 1
Currently our logging support only works for the server.log
file and console output. If org.infinispan.REST_ACCESS_LOG
is specified, then the logs will always appear in server.log
, however we should detect when this category is specified and ensure that the REST-ACCESS-FILE is used instead.
logging:
categories:
org.infinispan: info
org.infinispan.HOTROD_ACCESS_LOG: trace
org.infinispan.REST_ACCESS_LOG: trace
Should result in the server/log/rest-access.log
and server/log/hotrod-access.log
being used.
Currently the image does not explicitly define a name for the cache-container, therefore it defaults to "DefaultCacheManager". We should utilise "default" as the name as this is the same as the normal server distribution.
It should be possible for a user to mount a volume at /opt/infinispan/server/lib
, which contains a jar(s) with SerializationContextInitializer
implementations, with no additional configuration and the server should auto-configure the SerializationContext.
This requires additionall changes to the server itself: https://issues.jboss.org/browse/ISPN-10960
Currently keystore.path
is used to configure an existing keystore, however this is confusing as keystore.crtPath
expects a directory path
. To differentiate the two, we should change keystore.path
-> keystore.file
.
https://issues.redhat.com/browse/ISPN-11575 introduced encrypted security credentials so we should utilise these in the image. Instead of duplicating logic, the config-generator should just forward the parsed content of the identities.yaml to the bin/user-tool.sh
included with the server.
The image utilises the org.infinispan.images:config-generator
to generate server xml configuration from the provided yaml files. Currently this tool is written in Groovy and is executed on the JVM, however this prevents us from removing the JDK installation from the image even when utilising a native server (#37).
Unfortunately due to it's dynamic nature, Groovy is not a good fit for GraalVM and native compilation. Therefore, we should rewrite the artifact so that it can be natively compiled.
Once the artifact can be natively compiled, we should consider using the native version of the artifact in the non-native image as it removes the need to start a second jvm.
The image is based upon ubi-minimal
in order to reduce it's footprint. Consequently, the image does not have many of the tools that developers are accustomed to when debugging issues. However, additional packages can be installed at image build time.
What packages do we require for the most common debugging tasks?
ISPN-10752 enabled statistics by default in the server distribution, we should do the same for the image.
If statistics has a negative effect on performance, then currently it is necessary to utilise a custom configuration in order to disable it. A toggle to enable/disable statistics can be trivially added to the yaml config if there is demand.
It would be very helpful if we can pass in a list of pre-defined caches so that clients are always guaranteed to have them available? This is particularly useful when deploying to AWS Beanstalk in single container mode.
This requires compliments #16
We should then allow the following yaml to be configured to disable auth on specifc endpoints:
endpoints:
hotrod:
auth: true | false
enabled: true
rest:
auth: true | false
enabled: true
It should be possible for a custom user marshaller and the class white list to be configured via the config yaml file. This is necessary to facilitate storing of objects in the server as application/x-java-object
.
Possiblle yaml config:
infinispan:
marshaller: org.infinispan.commons.marshall.JavaSerializationMarshaller
whiteListRegex: ""
Currently we don't parse any args passed to server.sh
. We should forward the arguments of -b
and any -D
properties. This is needed for the Infinispan tests to execute as expected.
The config-generator should also be updated to respect the passed ${infinispan.bind.address}.
BIND_ADDRESS=
CONFIG_FILE=infinispan.xml
JAVA_OPTS=
while [ "$#" -gt 0 ]
do
case "$1" in
-b)
BIND_ADDRESS=$2
shift
;;
-c)
CONFIG_FILE=$2
shift
;;
-D*)
JAVA_OPTS="$JAVA_OPTS $1"
;;
*)
ARGUMENTS="$ARGUMENTS $1"
;;
esac
shift
done
while true; do
# Execute the process in the background in order for signals to be correctly handled
bin/server-runner \
-Dinfinispan.bind.address=${BIND_ADDRESS} \
-Dquarkus.infinispan-server.config-file=${CONFIG_FILE} \
-Dquarkus.infinispan-server.config-path=server/conf \
-Dquarkus.infinispan-server.data-path=data \
-Dquarkus.infinispan-server.server-path=${ISPN_HOME} ${JAVA_OPTS} &
ISPN_PID=$!
# Trap common signals and relay them to the server process
trap "kill -HUP $ISPN_PID" HUP
trap "kill -TERM $ISPN_PID" INT
trap "kill -QUIT $ISPN_PID" QUIT
trap "kill -PIPE $ISPN_PID" PIPE
trap "kill -TERM $ISPN_PID" TERM
if [ "x$ISPN_PIDFILE" != "x" ]; then
echo $ISPN_PID > $ISPN_PIDFILE
fi
# Wait until the background process exits
WAIT_STATUS=128
while [ "$WAIT_STATUS" -ge 128 ]; do
wait $ISPN_PID 2>/dev/null
WAIT_STATUS=$?
if [ "$WAIT_STATUS" -gt 128 ]; then
SIGNAL=`expr $WAIT_STATUS - 128`
SIGNAL_NAME=`kill -l $SIGNAL`
echo "*** Server process ($ISPN_PID) received $SIGNAL_NAME signal ***" >&2
fi
done
if [ "$WAIT_STATUS" -lt 127 ]; then
ISPN_STATUS=$WAIT_STATUS
else
ISPN_STATUS=0
fi
if [ "$ISPN_STATUS" -ne 10 ]; then
# Wait for a complete shudown
wait $ISPN_PID 2>/dev/null
fi
if [ "x$ISPN_PIDFILE" != "x" ]; then
grep "$ISPN_PID" $ISPN_PIDFILE && rm $ISPN_PIDFILE
fi
if [ "$ISPN_STATUS" -eq 10 ]; then
echo "Restarting server..."
else
exit $ISPN_STATUS
fi
done
It looks like the probe-common.sh
will always parse ${ISPN_HOME}/server/conf/infinispan.xml
even if a custom config is provided with -c
.
Anyway, when providing a custom config, the container is never ready in kubernetes, the readyness probe outputs :
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
The current image is pulling in lots of unnecessary dependencies, we should remove these at build time.
We should also consider utilising jlink to create a minimal JDK.
It should be possible to add server authorization configuration via the server.yaml
.
security:
authorization:
roles:
- name: vassal
permissions:
- READ
- WRITE
- LISTEN
- name: peasent
permissions:
- READ
Which will result in the following xml:
<authorization>
<identity-role-mapper/>
<role name="peasant" permissions="READ"/>
<role name="vassal" permissions="READ WRITE LISTEN"/>
</authorization>
In the old image it was possible to provide a single user/password combo to the image in order to quickly start the image. We should provide similar env variables to allow easy development of clients without the need to mount the yaml files required for non-dev environments.
We can then pass any args passed to docker run directly to server.sh
.
TLS is not currently working due to handshake errors, potentially related to openssl/openssl#7147
Currently we only have unit tests for the config generator. Once ISPN-11370 has been merged and released, we should create an integration testsuite for the image to ensure that created images work as expected e.g. Auth can be disabled etc.
These tests should be incorporated as part of the release pipeline. Furthermore, it should be possible to make it so that changes to the config-generator aritfacts are also built as part of the full image and executed against the testsuite.
keystore.path
is currently being ignored by the infinispan.xml
template regardless of it's state, consequently the <ssl>
elements are never added to the config. This also affects keystore.crtPath
as we internally set keystore.path
.
#35 requires that a space optimisation is removed in order for downstream specific tooling to function correctly. This tooling is not required upstream, therefore we should provide upstream, downstream and common install modules so that it's possible for some optimisations to only exist upstream or downstream. The default image.yaml
will specify the modules required for upstream, whereas the dg-override.yaml
override file will specify the downstream modules.
It should be possible for users to configure logging via the config yaml in the following format:
logging:
console:
level: trace
pattern: '%K{level}%d{HH\:mm\:ss,SSS} %-5p [%c] (%t) %s%e%n'
file:
level: trace
path: server/log
pattern: '%d{yyyy-MM-dd HH\:mm\:ss,SSS} %-5p [%c] (%t) %s%e%n'
categories:
com.arjuna: warn
org.infinispan: info
org.jgroups: warn
The jgroups discovery protocols should have enable_diagnostics=true
as the default behaviour to provide more debugging options. This should not be a security issue, as it's necessary for the port to be explicitly exposed by the container environment.
Currentlly the image uses the java-11-openjdk-headless
package which only includes a java runtime. In order to be able to effictively debug images in production we should add the -devel
packages so that we hava access to tools such as jmap
etc.
Currently the users.properties
file always stores user/passwords digested. Unfortunately this means that it's only possible to use the DIGEST-MD5 mech. Long term the plan is for the server to utilise a file-based realm as an encrypted storage to allow all mechs to work, however until then it is necessary to revert to storing user/password combos in plain text.
Hi
I am trying to perform put operation using Infinispan JS-client but I am getting following error
java.lang.SecurityException: ISPN006017: Unauthorized 'PUT' operation.
client = await infinispan.client({
port: 11222,
host: '127.0.0.1'
}, {
cacheName: 'testcahe'
});
console.log(Connected to Infinispan dashboard data
);
await client.put('test', 'hello 1');
However, the following code runs fine if Datagrid is deployed on a local machine. Only getting problem while running Infinispan as a docker container. I have tried different configurations using CORS setting and security constraint.
Am I missing any configuration
Currently we support the ASYM_ENCRYPT protocol for JGroups encryption, however this is prone to man in the middle attacks. This can be overcome by utilising the SSL_KEY_EXCHANGE
protocol, however this requires a keystore to be configured. If the user has configured a keystore, we should utilise this and automatically add the SSL_KEY_EXCHANGE
protocol to the stack if jgroups.encrypt == true
.
Current issues with ASYM_ENCRYPT
and SSL_KEY_EXCHANGE
.
<VERIFY_SUSPECT timeout="1000"/>
<gsp:scriptlet>if (jgroups?.encrypt) {</gsp:scriptlet>
<gsp:scriptlet>if (keystore?.path) {</gsp:scriptlet>
<SSL_KEY_EXCHANGE
keystore_name="${keystore.path}"
keystore_password="${keystore.password}"
keystore_type="pkcs12"
port="2157"
port_range="0"
/>
<gsp:scriptlet>}</gsp:scriptlet>
<ASYM_ENCRYPT use_external_key_exchange="${keystore?.path ? true : false}"
sym_algorithm="AES/ECB/PKCS5Padding"
asym_keylength="512"
asym_algorithm="RSA"
/>
<gsp:scriptlet>}</gsp:scriptlet>
Node 1:
17:09:53,428 DEBUG [org.jgroups.protocols.SSL_KEY_EXCHANGE] (main) 9864590834c5-46097: becoming keyserver; creating server socket
17:09:53,508 DEBUG [org.jgroups.protocols.SSL_KEY_EXCHANGE] (main) 9864590834c5-46097: SSL server socket listening on /172.17.0.2:2157
17:09:53,511 DEBUG [org.jgroups.protocols.ASYM_ENCRYPT] (main) 9864590834c5-46097: I'm the new key server
17:09:53,530 DEBUG [org.jgroups.protocols.ASYM_ENCRYPT] (main) 9864590834c5-46097: created new group key (version: DCEFD81727549FA1786B1DAF8E35DD13) because of new view [9864590834c5-46097|0] (1) [9864590834c5-46097]
17:09:53,536 INFO [org.infinispan.CLUSTER] (main) ISPN000094: Received new cluster view for channel testClusterName: [9864590834c5-46097|0] (1) [9864590834c5-46097]
17:09:53,553 INFO [org.infinispan.CLUSTER] (main) ISPN000079: Channel testClusterName local address is 9864590834c5-46097, physical addresses are [172.17.0.2:7800]
17:09:53,564 INFO [org.infinispan.CONTAINER] (main) ISPN000390: Persisted state, version=10.1.0.Beta1 timestamp=2019-12-03T17:09:53.560070Z
17:09:54,029 INFO [org.infinispan.CONTAINER] (main) ISPN000104: Using EmbeddedTransactionManager
17:09:54,275 INFO [org.infinispan.SERVER] (ForkJoinPool.commonPool-worker-3) ISPN080018: Protocol HotRod (internal)
17:09:54,357 INFO [org.infinispan.SERVER] (main) ISPN080018: Protocol REST (internal)
17:09:54,468 INFO [org.infinispan.SERVER] (ForkJoinPool.commonPool-worker-5) ISPN080004: Protocol Memcached listening on 172.17.0.2:11221
17:09:54,479 INFO [org.infinispan.SERVER] (main) ISPN080004: Protocol SINGLE_PORT listening on 172.17.0.2:11222
17:09:54,479 INFO [org.infinispan.SERVER] (main) ISPN080001: Infinispan Server 10.1.0.Beta1 started in 8683ms
17:10:03,381 INFO [org.infinispan.CLUSTER] (jgroups-5,9864590834c5-46097) ISPN000094: Received new cluster view for channel testClusterName: [9864590834c5-46097|1] (2) [9864590834c5-46097, fe47e1f1471b-64307]
17:10:03,387 INFO [org.infinispan.CLUSTER] (jgroups-5,9864590834c5-46097) ISPN100000: Node fe47e1f1471b-64307 joined the cluster
17:10:03,423 TRACE [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-5,9864590834c5-46097) 9864590834c5-46097: asking fe47e1f1471b-64307 to fetch the shared group key DCEFD81727549FA1786B1DAF8E35DD13 via an external key exchange protocol (srv=172.17.0.2:2157)
17:10:03,515 TRACE [org.jgroups.protocols.SSL_KEY_EXCHANGE] (SSL_KEY_EXCHANGE-runner-9,9864590834c5-46097) 9864590834c5-46097: failure handling client socket: Remote host terminated the handshake
17:10:03,557 TRACE [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-8,9864590834c5-46097) 9864590834c5-46097: asking fe47e1f1471b-64307 to fetch the shared group key DCEFD81727549FA1786B1DAF8E35DD13 via an external key exchange protocol (srv=172.17.0.2:2157)
17:10:03,562 TRACE [org.jgroups.protocols.SSL_KEY_EXCHANGE] (SSL_KEY_EXCHANGE-runner-9,9864590834c5-46097) 9864590834c5-46097: failure handling client socket: Remote host terminated the handshake
17:10:03,659 TRACE [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-8,9864590834c5-46097) 9864590834c5-46097: asking fe47e1f1471b-64307 to fetch the shared group key DCEFD81727549FA1786B1DAF8E35DD13 via an external key exchange protocol (srv=172.17.0.2:2157)
Node 2:
17:10:02,840 INFO [org.infinispan.CONTAINER] (main) ISPN000128: Infinispan version: Infinispan 'Chupacabra' 10.1.0.Beta1
17:10:03,012 INFO [org.infinispan.CLUSTER] (main) ISPN000078: Starting JGroups channel testClusterName
17:10:03,426 TRACE [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-5,fe47e1f1471b-64307) fe47e1f1471b-64307: fetching group key from 172.17.0.2:2157
17:10:03,430 TRACE [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-6,fe47e1f1471b-64307) fe47e1f1471b-64307: discarded mcast batch from 9864590834c5-46097 as secret key is null
17:10:03,457 TRACE [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-10,fe47e1f1471b-64307) fe47e1f1471b-64307: message from 9864590834c5-46097 (version: DCEFD81727549FA1786B1DAF8E35DD13) dropped, as a key matching that version wasn't found (current version: null)
17:10:03,508 WARN [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-5,fe47e1f1471b-64307) fe47e1f1471b-64307: failed fetching group key from 9864590834c5-46097: java.lang.IllegalStateException: failed connecting to 172.17.0.2:2157: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
17:10:03,508 TRACE [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-5,fe47e1f1471b-64307) fe47e1f1471b-64307: message from 9864590834c5-46097 (version: DCEFD81727549FA1786B1DAF8E35DD13) dropped, as a key matching that version wasn't found (current version: null)
17:10:03,557 TRACE [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-5,fe47e1f1471b-64307) fe47e1f1471b-64307: message from 9864590834c5-46097 (version: DCEFD81727549FA1786B1DAF8E35DD13) dropped, as a key matching that version wasn't found (current version: null)
17:10:03,557 TRACE [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-10,fe47e1f1471b-64307) fe47e1f1471b-64307: message from 9864590834c5-46097 (version: DCEFD81727549FA1786B1DAF8E35DD13) dropped, as a key matching that version wasn't found (current version: null)
17:10:03,558 TRACE [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-10,fe47e1f1471b-64307) fe47e1f1471b-64307: fetching group key from 172.17.0.2:2157
17:10:03,560 WARN [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-10,fe47e1f1471b-64307) fe47e1f1471b-64307: failed fetching group key from 9864590834c5-46097: java.lang.IllegalStateException: failed connecting to 172.17.0.2:2157: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
17:10:03,560 TRACE [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-10,fe47e1f1471b-64307) fe47e1f1471b-64307: message from 9864590834c5-46097 (version: DCEFD81727549FA1786B1DAF8E35DD13) dropped, as a key matching that version wasn't found (current version: null)
17:10:03,659 TRACE [org.jgroups.protocols.ASYM_ENCRYPT] (jgroups-10,fe47e1f1471b-64307) fe47e1f1471b-64307: message from 9864590834c5-46097 (version: DCEFD81727549FA1786B1DAF8E35DD13) dropped, as a key matching that version wasn't found (current version: null)
Currently we store key/values in plain text as it's necessary for our probe implementations to authenticate with the rest endpoint in order to retrieve health information. Once infinispan/infinispan#7248 has been merged, it will be possible to request this information anonymously, therefore we should no longer store credentials in plain-text.
Currently we tag our image in the following format: infinispan/server:<infinispan-version>-<image-version>
.
If a user is only interested in the most specific release of an image, it's currently not possible for the user to execute docker pull infinispan/server:10.0.0.Final
and get the latest image that utilises that artifact.
We should update our release scripts so that when an image is released, it creates the following tags:
:<infinispan-version>-<image-version>
:<infinispan-version>
With <infinispan-version>
always pointing to the latest image build that contains that Infinispan artifact.
The colours are not displayed in most cases and can cause issues in third party tooling such as Kibana.
We should add the system properties that are normally present in Infinispan xml configs so that they can be overridden via JAVA_OPTIONS
at runtime.
It should be possible for the users to provide custom jgroups stacks via a mounted volume, providing the location of the file via an entry in the config yaml. The generated infinispan.xml
file should then automatically add said stack file and utilise it for cluster communication.
The quarkus native binary does not consume a lo4j2.xml
file, instead logging must be configured via properties.
The config-generator should be updated so that it can automatically generate a file containing the required quarkus properties for the logging config specified in the server.yaml
. This can then be combined with a .env
file containing the other runtme properties, such as -Dquarkus.infinispan-server.config-file
to launch the native binary.
Requires a workaround for an underlying microdnf issue:
Policy should be something like:
if profile is developer
and no custom certificates nor service cert service are provided then let Infinispan generates self signed certificates for TLS
We should add the ASYM_ENCRYPT
protocol to the stack when the following yaml is defind:
jgroups:
encrypt: true
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.