vmware/mangle

Git Repository for the Mangle tool

Introduction

Mangle Documentation

The Mangle Documentation provides information about how to install, configure, and use Mangle™.

To navigate to the appropriate documentation, start with the Mangle GitHub IO Page.

Current Product version: 3.5.0

Demo of the Mangle Version 1.0

Intended Audiences

This information is intended for the following audiences:

Audience	Description of Role	Relevant Documentation
All Users	Understand Mangle	Overview
Mangle administrators	Deploy and configure Mangle	Mangle Deployment and Administration Guide
SRE, developers, viewers	Use Mangle to run chaos experiments and monitor progress	Mangle Users Guide
All community	Building the Mangle Codebase and creating local images of Mangle	Mangle Developers' Guide
Contributors of Mangle	Guidelines for Contributing	Contributing to Mangle

Copyright (c) 2019 VMware, Inc. All rights reserved. Copyright and trademark information. Any feedback you provide to VMware is subject to the terms at www.vmware.com/community_terms.html.

VMware, Inc.
3401 Hillview Ave.
Palo Alto, CA 94304

www.vmware.com

mangle's Issues

Suggestion for Fault injection in ECS with Mangle

Hi Team,
Could you please advise if we can perform chaos scenarios for ECS or not. If yes please guide how to proceed.

Security Vulnerabilities

Hi Mangle Team,

My team and I are trying to deploy Mangle for use on a client engagement and ran a vulnerabilities check against the dependencies and executables packages. We have found numerous critical vulnerabilites and were hoping to get some help updating the versions of these packages. Would it be possible for this to be done? Below is a list of the critical security vulnerabiltes and known fixed versions. If you need any more information please let me know.

package > current version > fixed version

apache tomcat_tomcat-embed-core   9.0.19   fixed in 9.0.31, 8.5.51, 7.0.100
ch.qos.logback_logback-core   1.1.3 fixed in 1.2.0
com.fasterxml.jackson.core_jackson-databind   2.9.8 fixed in 2.9.10.4
io.netty_netty-all   4.0.44.Final fixed in 4.1.44
io.netty_netty-codec   4.1.33.Final fixed in 4.1.46
libbz2-1.0,bzip2 1.0.6-8.1 open
libc-bin,multiarch-support,libc6 glibc 2.24-11+deb9u3 fixed in 2.24-11+deb9u4
libidn11 1.33-1 fixed in 1.33-1+deb9u1
libmagic-mgc,libmagic1,file 1:5.30-1+deb9u2 fixed in 1:5.30-1+deb9u3
libpython2.7-stdlib,libpython2.7-minimal,python2.7-minimal,python2.7 2.7.13-2+deb9u3 fixed in 2.7.13-2+deb9u4
libsystemd0,libudev1 232-25+deb9u6 fixed in 232-25+deb9u10  
org.apache.tika_tika-core 1.7 fixed in 1.14

Bad credentials after forced password change

Describe the bug
After the forced initial password change, Mangle doesn't accept either the old or the new password.

To Reproduce
Steps to reproduce the behavior:

1. Deploy Mangle 2.0
2. Load /mangle-services
3. Change the password from 'admin' to something else.
4. Attempt to login with the new password
5. UI returns "Invalid username or password"

Expected behavior
Login succeeds.

Logs

2020-01-14 20:53:19.976 [https-jsse-nio-8443-exec-6] INFO  com.vmware.mangle.services.UserService.terminateUserSession (408) - The User [email protected] not Exists/logged In
2020-01-14 20:53:20.027 [https-jsse-nio-8443-exec-6] ERROR com.vmware.mangle.services.config.MangleBasicAuthenticationEntryPoint.getMangleErrorDetails (66) - Authentication for the user [email protected] failed with the error: Bad credentials

Configuration information:

• Deployment Type: Container
• Deployment Mode: Single node
• Client OS: OS X
• Client Browser: Chrome 80

Additional context

Coming Soon in February 2021!

Mangle 3.0 will be available soon!

What can you expect in the new release?

1. Support for Azure Endpoint
2. AWS RDS Faults and EBS Volume Faults
3. DB Faults: Connection Leak, Transaction Latency, Transaction Error
4. New OOTB integrations: Prometheus, Slack
5. Support for Remote Machines as a Cluster
6. Redis Faults: Integration with RedFI
7. Network Partition Fault
8. Time Skew Fault
9. Resiliency Score - Quantifying fault tolerance using metrics
10. Real time polling for Fault Execution status
11. Modified Python based agents to reduce pre-checks and dependency for faults

CPU spike fault not injecting

Describe the bug
Mangle is deployed on OpenShift container. Trying to inject CPU spike fault to clusterK8 endpoint service.
We are not able to spike the CPU to 90%. The CPU spikes very little compared to target we want.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'Mangle UI'
2. Click on 'Fault Exception'
3. Click on 'CPU ',
4. Fill the Target cluster and JVM information, and put CPU to 100%
5. Go to Requests and Response. you will see the request is executed successfully.

Expected behavior
CPU spiked to 100% on target instance.

Screenshots
If applicable, add screenshots to help explain your problem.

Logs
If applicable, add application logs/troubleshooting bundle to help in root cause analysis.

Configuration information:

• Deployment Type: [e.g. OVA, Container] - Target API on k8s Container
• Deployment Mode: [e.g. Cluster, e] - Cluster
• Client OS: [e.g. iOS] : Open Shift
• Client Browser [e.g. chrome, safari, Swagger] - Chrome
• Version [e.g. 22] 93

Additional context
Add any other context about the problem here.

Installation in Kubernetes cluster is failling

Hi,

I am trying to deploy mangle into a Kubernetes cluster but I am not able to do so.
It fails at kubectl -n mangle apply -f .\mangle.yaml
First it complains that: error: unable to recognize ".\\mangle.yaml": no matches for kind "Deployment" in version "extensions/v1"

Than after I change magle.yaml to have apiVersion: apps/v1 it complains that:

error: error validating ".\\mangle.yaml": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false

 kubectl -n mangle apply -f .\mangle.yaml --validate=false
The Deployment "mangle" is invalid:
* spec.selector: Required value
* spec.template.metadata.labels: Invalid value: map[string]string{"app":"mangle", "environment":"test"}: `selector` does not match template `labels`

Also I noticed that in mangle.yaml there is still mangle:2.0

image: mangleuser:mangle:2.0

Thank you,
John

Filters for Network fault

Is there a way to execute Network fault with IP or Port filters? The fault is executed on the host's interface. If the deployed containers on the host use the same network interface, it would impact the traffic for all the containers. IP or port filters will help to restrict the impact to a specific container on the host

No permission to Teams chat

Describe the bug
The team chat referenced in CONTRIBUTING.md is not available to the public (no permission to the org).

To Reproduce
Steps to reproduce the behavior:

1. Go to CONTRIBUTING.md
2. Click on 'team chat'
3. Wait for MS Teams client to load
4. On "Do you want to switch to your other org?", click "Confirm"
5. See error: "You don't have access to this org in Teams. Select another org, or contact your admin for more details."

Mangle 3.5 : Update is diabled for initial password change

Describe the bug
Installed Mangle 3.5.0 both via the OVA as well as by deploying the docker containers. In both ways I could get Mangle up and running except that the update button is grayed out while changing the initial password.

To Reproduce
Steps to reproduce the behavior:
Install Mangle with the preferred way and reach the url at https://ip-address/mangle-services/
Expected behavior
The Update button should get hot when the condition for password change are met.

Screenshots

Logs

The mangle log shows no errors
. . .

2022-03-31 13:56:53.245 [main] INFO org.apache.coyote.http11.Http11NioProtocol.log (173) - Starting ProtocolHandler ["https-jsse-nio-8443"]
2022-03-31 13:56:53.298 [hz._hzInstance_1_mangle.event-4] INFO com.vmware.mangle.services.hazelcast.HazelcastQuorumListener.updateApplicationQuorumStatus (53) - Quorum has been created
2022-03-31 13:56:53.298 [hz._hzInstance_1_mangle.event-4] INFO com.vmware.mangle.services.config.MangleBootInitializer.updateClusterConfigObject (209) - Updating mangle cluster configuration
2022-03-31 13:56:53.312 [hz._hzInstance_1_mangle.event-4] INFO com.vmware.mangle.services.ClusterConfigService.updateClusterConfiguration (74) - Updating cluster config
2022-03-31 13:56:53.318 [main] INFO org.apache.coyote.http11.Http11NioProtocol.log (173) - Starting ProtocolHandler ["http-nio-8080"]
2022-03-31 13:56:53.323 [main] INFO org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/mangle-services].log (173) - Initializing Spring DispatcherServlet 'dispatcherServlet'
2022-03-31 13:56:53.323 [main] INFO org.springframework.web.servlet.DispatcherServlet.initServletBean (524) - Initializing Servlet 'dispatcherServlet'
2022-03-31 13:56:53.332 [hz._hzInstance_1_mangle.event-4] INFO com.vmware.mangle.services.config.MangleBootInitializer.initializeApplicationTasks (105) - Mangle cluster has only one node, will be triggering scheduled tasks and in-progress tasks
2022-03-31 13:56:53.338 [main] INFO org.springframework.web.servlet.DispatcherServlet.initServletBean (546) - Completed initialization in 15 ms
2022-03-31 13:56:53.340 [main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start (204) - Tomcat started on port(s): 8443 (https) 8080 (http) with context path '/mangle-services'
2022-03-31 13:56:53.340 [hz._hzInstance_1_mangle.event-4] INFO com.vmware.mangle.services.SchedulerService.getAllScheduledJobByStatus (82) - Retrieving MangleScheduled jobs with status: SCHEDULED
2022-03-31 13:56:53.343 [main] INFO com.vmware.mangle.MangleApplication.logStarted (59) - Started MangleApplication in 16.573 seconds (JVM running for 17.77)
2022-03-31 13:56:53.343 [pool-4-thread-1] INFO com.vmware.mangle.services.poll.PollingService.lambda$startPollingThread$0 (109) - Starting polling thread..
2022-03-31 13:56:53.351 [hz._hzInstance_1_mangle.event-4] INFO com.vmware.mangle.services.config.MangleBootInitializer.scheduleMisfiredJobs (118) - No jobs found to be rescheduled
2022-03-31 13:56:53.445 [main] INFO com.vmware.mangle.MangleApplication.main (37) - Application initialization completed...

. . .

Configuration information:

• Deployment Type: OVA and Container
• Deployment Mode: Single Node
• Client OS: for container centOS while the OVA runs on photon
• Client Browser Chrome and Firefox
• Version 3.5.0

Unable to run mangle jar

Describe the bug
While trying to run the mangle-services.jar after building it gives error saying:

2020-05-17 14:00:49.712 [main] ERROR org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup (62) - Error starting Tomcat context. Exception: org.springframework.beans.factory.UnsatisfiedDependencyException. Message: Error creating bean with name 'webMvcMetricsFilter' defined in class path resource [org/springframework/boot/actuate/autoconfigure/metrics/web/servlet/WebMvcMetricsAutoConfiguration.class]: Unsatisfied dependency expressed through method 'webMvcMetricsFilter' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'compositeMeterRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/metrics/CompositeMeterRegistryConfiguration.class]: Unsatisfied dependency expressed through method 'compositeMeterRegistry' parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'initializeDatadogMeterRegistry' defined in class path resource [com/vmware/mangle/services/config/MangleMicrometerConfig.class]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [com/vmware/mangle/services/hazelcast/HazelcastConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'createHazelcastCluster' threw exception; nested exception is com.vmware.mangle.utils.exceptions.MangleException: Node spin up failed, mandatory parameter validation token is not provided

To Reproduce
Steps to reproduce the behavior:

1. git clone mangle
2. run mvn clean install
3. cd mangle-service/target
4. run java –jar mangle-services-.x.x.x-jar –D......

Expected behavior
Should start server and able to run the mangle app on browser(localhost)

Logs

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.1.3.RELEASE)

2020-05-17 14:00:43.053 [main] INFO  com.vmware.mangle.MangleApplication.logStartupProfileInfo (675) - No active profile set, falling back to default profiles: default
2020-05-17 14:00:44.854 [main] INFO  org.springframework.data.repository.config.RepositoryConfigurationDelegate.registerRepositoriesIn (126) - Bootstrapping Spring Data repositories in DEFAULT mode.
2020-05-17 14:00:45.066 [main] INFO  org.springframework.data.repository.config.RepositoryConfigurationDelegate.registerRepositoriesIn (182) - Finished Spring Data repository scanning in 206ms. Found 0 repository interfaces.
2020-05-17 14:00:45.071 [main] INFO  org.springframework.data.repository.config.RepositoryConfigurationDelegate.registerRepositoriesIn (126) - Bootstrapping Spring Data repositories in DEFAULT mode.
2020-05-17 14:00:45.110 [main] INFO  org.springframework.data.repository.config.RepositoryConfigurationDelegate.registerRepositoriesIn (182) - Finished Spring Data repository scanning in 39ms. Found 18 repository interfaces.
2020-05-17 14:00:45.563 [main] INFO  org.springframework.cloud.context.scope.GenericScope.setSerializationId (294) - BeanFactory id=ce1fe525-81b7-3eee-bd34-29a5b9222500
2020-05-17 14:00:45.612 [main] INFO  org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker.postProcessAfterInitialization (330) - Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$73017990] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-05-17 14:00:45.875 [main] INFO  org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize (90) - Tomcat initialized with port(s): 8443 (https) 8080 (http)
2020-05-17 14:00:45.895 [main] INFO  org.apache.coyote.http11.Http11NioProtocol.log (173) - Initializing ProtocolHandler ["https-jsse-nio-8443"]
2020-05-17 14:00:45.905 [main] INFO  org.apache.coyote.http11.Http11NioProtocol.log (173) - Initializing ProtocolHandler ["http-nio-8080"]
2020-05-17 14:00:45.913 [main] INFO  org.apache.catalina.core.StandardService.log (173) - Starting service [Tomcat]
2020-05-17 14:00:45.914 [main] INFO  org.apache.catalina.core.StandardEngine.log (173) - Starting Servlet engine: [Apache Tomcat/9.0.19]
2020-05-17 14:00:45.926 [main] INFO  org.apache.catalina.core.AprLifecycleListener.log (173) - The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
2020-05-17 14:00:46.000 [main] INFO  org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/mangle-services].log (173) - Initializing Spring embedded WebApplicationContext
2020-05-17 14:00:46.000 [main] INFO  org.springframework.web.context.ContextLoader.prepareWebApplicationContext (296) - Root WebApplicationContext: initialization completed in 2925 ms
2020-05-17 14:00:46.091 [main] INFO  org.pf4j.DefaultPluginStatusProvider.<init> (49) - Enabled plugins: []
2020-05-17 14:00:46.092 [main] INFO  org.pf4j.DefaultPluginStatusProvider.<init> (53) - Disabled plugins: []
2020-05-17 14:00:46.099 [main] INFO  org.pf4j.DefaultPluginManager.initialize (110) - PF4J version 3.0.1 in 'deployment' mode
2020-05-17 14:00:46.103 [main] INFO  org.pf4j.AbstractPluginManager.loadPlugins (215) - No plugins
2020-05-17 14:00:46.195 [main] INFO  com.datastax.driver.core.GuavaCompatibility.selectImplementation (185) - Detected Guava >= 19 in the classpath, using modern compatibility layer
2020-05-17 14:00:46.517 [main] INFO  com.datastax.driver.core.ClockFactory.newInstance (57) - Using native clock to generate timestamps.
2020-05-17 14:00:46.721 [main] INFO  com.datastax.driver.core.logDriverVersion (261) - DataStax Java driver 3.6.0 for Apache Cassandra
2020-05-17 14:00:46.785 [main] INFO  com.datastax.driver.core.LZ4Compressor.<init> (54) - Using LZ4Factory:JavaUnsafe
2020-05-17 14:00:47.036 [main] INFO  com.datastax.driver.core.NettyUtil.<clinit> (72) - Found Netty's native epoll transport in the classpath, using it
2020-05-17 14:00:47.144 [main] DEBUG com.datastax.driver.core.Host.STATES.signalConnectionsOpening (82) - [/10.198.131.111:9042] preparing to open 1 new connections, total = 1
2020-05-17 14:00:47.567 [manglecassandracluster-nio-worker-0] DEBUG com.datastax.driver.core.Host.STATES.markInitialized (409) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[/10.198.131.111:9042-1, inFlight=0, closed=false] Transport initialized, connection ready
2020-05-17 14:00:47.747 [main] DEBUG com.datastax.driver.core.Host.STATES.setNewConnection (216) - [Control connection] established to ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042
2020-05-17 14:00:47.748 [main] INFO  com.datastax.driver.core.policies.DCAwareRoundRobinPolicy.init (110) - Using data-center name 'DC1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
2020-05-17 14:00:47.749 [main] INFO  com.datastax.driver.core.Cluster.init (1621) - New Cassandra host ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042 added
2020-05-17 14:00:47.758 [main] DEBUG com.datastax.driver.core.Host.STATES.signalConnectionsOpening (82) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] preparing to open 4 new connections, total = 5
2020-05-17 14:00:47.864 [manglecassandracluster-nio-worker-3] DEBUG com.datastax.driver.core.Host.STATES.markInitialized (409) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-4, inFlight=0, closed=false] Transport initialized, connection ready
2020-05-17 14:00:47.865 [manglecassandracluster-nio-worker-1] DEBUG com.datastax.driver.core.Host.STATES.markInitialized (409) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-2, inFlight=0, closed=false] Transport initialized, connection ready
2020-05-17 14:00:47.890 [manglecassandracluster-nio-worker-2] DEBUG com.datastax.driver.core.Host.STATES.markInitialized (409) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-3, inFlight=0, closed=false] Transport initialized, connection ready
2020-05-17 14:00:47.890 [manglecassandracluster-nio-worker-4] DEBUG com.datastax.driver.core.Host.STATES.markInitialized (409) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-5, inFlight=0, closed=false] Transport initialized, connection ready
2020-05-17 14:00:47.982 [main] DEBUG com.datastax.driver.core.Host.STATES.signalConnectionClosed (91) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-2, inFlight=0, closed=true] closed, remaining = 4
2020-05-17 14:00:47.985 [main] DEBUG com.datastax.driver.core.Host.STATES.signalConnectionClosed (91) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-3, inFlight=0, closed=true] closed, remaining = 3
2020-05-17 14:00:47.986 [main] DEBUG com.datastax.driver.core.Host.STATES.signalConnectionClosed (91) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-4, inFlight=0, closed=true] closed, remaining = 2
2020-05-17 14:00:47.988 [main] DEBUG com.datastax.driver.core.Host.STATES.signalConnectionClosed (91) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-5, inFlight=0, closed=true] closed, remaining = 1
2020-05-17 14:00:48.464 [main] DEBUG com.datastax.driver.core.Host.STATES.signalConnectionsOpening (82) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] preparing to open 4 new connections, total = 5
2020-05-17 14:00:48.577 [manglecassandracluster-nio-worker-5] DEBUG com.datastax.driver.core.Host.STATES.markInitialized (409) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-6, inFlight=0, closed=false] Transport initialized, connection ready
2020-05-17 14:00:48.601 [manglecassandracluster-nio-worker-8] DEBUG com.datastax.driver.core.Host.STATES.markInitialized (409) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-9, inFlight=0, closed=false] Transport initialized, connection ready
2020-05-17 14:00:48.604 [manglecassandracluster-nio-worker-7] DEBUG com.datastax.driver.core.Host.STATES.markInitialized (409) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-8, inFlight=0, closed=false] Transport initialized, connection ready
2020-05-17 14:00:48.604 [manglecassandracluster-nio-worker-6] DEBUG com.datastax.driver.core.Host.STATES.markInitialized (409) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-7, inFlight=0, closed=false] Transport initialized, connection ready
2020-05-17 14:00:48.607 [main] INFO  org.springframework.data.cassandra.config.CassandraSessionFactoryBean.lambda$executeScripts$0 (138) - executing raw CQL [ALTER KEYSPACE "system_auth" WITH REPLICATION = {'class':'NetworkTopologyStrategy','DC1': '1'};]
2020-05-17 14:00:48.663 [main] INFO  org.springframework.data.cassandra.config.CassandraSessionFactoryBean.lambda$executeScripts$0 (138) - executing raw CQL [CREATE KEYSPACE IF NOT EXISTS mangledb WITH replication = {'class': 'NetworkTopologyStrategy','DC1': '1'} AND DURABLE_WRITES = true;]
2020-05-17 14:00:49.374 [main] INFO  io.micrometer.datadog.DatadogMeterRegistry.start (97) - publishing metrics to datadog every 1m
2020-05-17 14:00:49.491 [main] INFO  com.vmware.mangle.services.scheduler.Scheduler.<init> (92) - Initializing Mangle Scheduler...
2020-05-17 14:00:49.498 [main] INFO  org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize (171) - Initializing ExecutorService 'Scheduler'
2020-05-17 14:00:49.499 [main] INFO  com.vmware.mangle.services.scheduler.Scheduler.<init> (94) - Initialized Mangle Scheduler...
2020-05-17 14:00:49.658 [main] INFO  org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize (171) - Initializing ExecutorService 'threadPoolTaskScheduler'
2020-05-17 14:00:49.712 [main] ERROR org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup (62) - Error starting Tomcat context. Exception: org.springframework.beans.factory.UnsatisfiedDependencyException. Message: Error creating bean with name 'webMvcMetricsFilter' defined in class path resource [org/springframework/boot/actuate/autoconfigure/metrics/web/servlet/WebMvcMetricsAutoConfiguration.class]: Unsatisfied dependency expressed through method 'webMvcMetricsFilter' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'compositeMeterRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/metrics/CompositeMeterRegistryConfiguration.class]: Unsatisfied dependency expressed through method 'compositeMeterRegistry' parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'initializeDatadogMeterRegistry' defined in class path resource [com/vmware/mangle/services/config/MangleMicrometerConfig.class]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [com/vmware/mangle/services/hazelcast/HazelcastConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'createHazelcastCluster' threw exception; nested exception is com.vmware.mangle.utils.exceptions.MangleException: Node spin up failed, mandatory parameter validation token is not provided
2020-05-17 14:00:49.750 [main] INFO  org.apache.catalina.core.StandardService.log (173) - Stopping service [Tomcat]
2020-05-17 14:00:49.756 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-connection-reaper-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.757 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-scheduled-task-worker-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.758 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-nio-worker-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 io.netty.channel.epoll.Native.epollWait0(Native Method)
 io.netty.channel.epoll.Native.epollWait(Native.java:114)
 io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
 io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
 io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.758 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-timeouter-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Thread.sleep(Native Method)
 io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:579)
 io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:478)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.759 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-nio-worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 io.netty.channel.epoll.Native.epollWait0(Native Method)
 io.netty.channel.epoll.Native.epollWait(Native.java:114)
 io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
 io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
 io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.760 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-nio-worker-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 io.netty.channel.epoll.Native.epollWait0(Native Method)
 io.netty.channel.epoll.Native.epollWait(Native.java:114)
 io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
 io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
 io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.761 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-nio-worker-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 io.netty.channel.epoll.Native.epollWait0(Native Method)
 io.netty.channel.epoll.Native.epollWait(Native.java:114)
 io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
 io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
 io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.761 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-nio-worker-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 io.netty.channel.epoll.Native.epollWait0(Native Method)
 io.netty.channel.epoll.Native.epollWait(Native.java:114)
 io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
 io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
 io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.762 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-worker-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.762 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.763 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-nio-worker-5] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 io.netty.channel.epoll.Native.epollWait0(Native Method)
 io.netty.channel.epoll.Native.epollWait(Native.java:114)
 io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
 io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
 io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.764 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-nio-worker-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 io.netty.channel.epoll.Native.epollWait0(Native Method)
 io.netty.channel.epoll.Native.epollWait(Native.java:114)
 io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
 io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
 io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.764 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-nio-worker-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 io.netty.channel.epoll.Native.epollWait0(Native Method)
 io.netty.channel.epoll.Native.epollWait(Native.java:114)
 io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
 io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
 io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.765 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-nio-worker-8] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 io.netty.channel.epoll.Native.epollWait0(Native Method)
 io.netty.channel.epoll.Native.epollWait(Native.java:114)
 io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
 io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
 io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.766 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-worker-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.766 [main] WARN  org.apache.catalina.loader.WebappClassLoaderBase.log (173) - The web application [mangle-services] appears to have started a thread named [manglecassandracluster-worker-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)
2020-05-17 14:00:49.768 [main] WARN  org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext.refresh (557) - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
2020-05-17 14:00:49.768 [main] INFO  org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.shutdown (208) - Shutting down ExecutorService 'threadPoolTaskScheduler'
2020-05-17 14:00:49.770 [main] DEBUG com.datastax.driver.core.Host.STATES.signalConnectionClosed (91) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-6, inFlight=0, closed=true] closed, remaining = 4
2020-05-17 14:00:49.770 [main] DEBUG com.datastax.driver.core.Host.STATES.signalConnectionClosed (91) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-7, inFlight=0, closed=true] closed, remaining = 3
2020-05-17 14:00:49.770 [main] DEBUG com.datastax.driver.core.Host.STATES.signalConnectionClosed (91) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-8, inFlight=0, closed=true] closed, remaining = 2
2020-05-17 14:00:49.771 [main] DEBUG com.datastax.driver.core.Host.STATES.signalConnectionClosed (91) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042-9, inFlight=0, closed=true] closed, remaining = 1
2020-05-17 14:00:49.773 [main] DEBUG com.datastax.driver.core.Host.STATES.signalConnectionClosed (91) - [ip-10-198-131-111.cloud.dev.net/10.198.131.111:9042] Connection[/10.198.131.111:9042-1, inFlight=0, closed=true] closed, remaining = 0
2020-05-17 14:00:52.002 [main] INFO  org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLoggingListener.logMessage (142) -

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-05-17 14:00:52.003 [main] ERROR com.vmware.mangle.utils.exceptions.handler.MangleUncaughtExceptionHandler.uncaughtException (67) - Uncaught exception caught in thread :main
org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:157) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543) ~[spring-context-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:142) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:316) [spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) [spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) [spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at com.vmware.mangle.MangleApplication.main(MangleApplication.java:36) [classes!/:2.0.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_251]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_251]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_251]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_251]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [mangle-services.jar:2.0.0]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [mangle-services.jar:2.0.0]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) [mangle-services.jar:2.0.0]
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52) [mangle-services.jar:2.0.0]
Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:125) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:86) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:415) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:174) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:181) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:154) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        ... 16 more
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'webMvcMetricsFilter' defined in class path resource [org/springframework/boot/actuate/autoconfigure/metrics/web/servlet/WebMvcMetricsAutoConfiguration.class]: Unsatisfied dependency expressed through method 'webMvcMetricsFilter' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'compositeMeterRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/metrics/CompositeMeterRegistryConfiguration.class]: Unsatisfied dependency expressed through method 'compositeMeterRegistry' parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'initializeDatadogMeterRegistry' defined in class path resource [com/vmware/mangle/services/config/MangleMicrometerConfig.class]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [com/vmware/mangle/services/hazelcast/HazelcastConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'createHazelcastCluster' threw exception; nested exception is com.vmware.mangle.utils.exceptions.MangleException: Node spin up failed, mandatory parameter validation token is not provided
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:769) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:509) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:204) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:235) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:226) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:101) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:88) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:261) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:234) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:54) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5139) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_251]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_251]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_251]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_251]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:423) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:932) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.startup.Tomcat.start(Tomcat.java:455) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:106) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:86) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:415) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:174) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:181) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:154) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        ... 16 more
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'compositeMeterRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/metrics/CompositeMeterRegistryConfiguration.class]: Unsatisfied dependency expressed through method 'compositeMeterRegistry' parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'initializeDatadogMeterRegistry' defined in class path resource [com/vmware/mangle/services/config/MangleMicrometerConfig.class]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [com/vmware/mangle/services/hazelcast/HazelcastConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'createHazelcastCluster' threw exception; nested exception is com.vmware.mangle.utils.exceptions.MangleException: Node spin up failed, mandatory parameter validation token is not provided
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:769) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:509) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1247) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1167) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:509) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:204) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:235) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:226) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:101) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:88) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:261) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:234) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:54) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5139) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_251]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_251]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_251]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_251]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:423) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:932) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.startup.Tomcat.start(Tomcat.java:455) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:106) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:86) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:415) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:174) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:181) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:154) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        ... 16 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'initializeDatadogMeterRegistry' defined in class path resource [com/vmware/mangle/services/config/MangleMicrometerConfig.class]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [com/vmware/mangle/services/hazelcast/HazelcastConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'createHazelcastCluster' threw exception; nested exception is com.vmware.mangle.utils.exceptions.MangleException: Node spin up failed, mandatory parameter validation token is not provided
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:601) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1463) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1427) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1205) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1167) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:509) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1247) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1167) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:509) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:204) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:235) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:226) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:101) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:88) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:261) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:234) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:54) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5139) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_251]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_251]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_251]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_251]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:423) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:932) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.startup.Tomcat.start(Tomcat.java:455) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:106) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:86) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:415) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:174) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:181) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:154) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        ... 16 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [com/vmware/mangle/services/hazelcast/HazelcastConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'createHazelcastCluster' threw exception; nested exception is com.vmware.mangle.utils.exceptions.MangleException: Node spin up failed, mandatory parameter validation token is not provided
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:627) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:456) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:307) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1470) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1427) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1273) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1205) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.resolveStream(DefaultListableBeanFactory.java:1928) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.orderedStream(DefaultListableBeanFactory.java:1922) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryConfigurer.asOrderedList(MeterRegistryConfigurer.java:86) ~[spring-boot-actuator-autoconfigure-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryConfigurer.customize(MeterRegistryConfigurer.java:71) ~[spring-boot-actuator-autoconfigure-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryConfigurer.configure(MeterRegistryConfigurer.java:60) ~[spring-boot-actuator-autoconfigure-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryPostProcessor.postProcessAfterInitialization(MeterRegistryPostProcessor.java:61) ~[spring-boot-actuator-autoconfigure-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:429) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1766) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1463) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1427) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1205) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1167) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:509) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1247) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1167) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:509) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:204) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:235) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:226) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:101) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:88) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:261) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:234) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:54) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5139) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_251]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_251]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_251]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_251]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:423) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:932) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.startup.Tomcat.start(Tomcat.java:455) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:106) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:86) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:415) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:174) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:181) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:154) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        ... 16 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'createHazelcastCluster' threw exception; nested exception is com.vmware.mangle.utils.exceptions.MangleException: Node spin up failed, mandatory parameter validation token is not provided
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:456) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:307) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1470) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1427) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1273) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1205) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.resolveStream(DefaultListableBeanFactory.java:1928) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.orderedStream(DefaultListableBeanFactory.java:1922) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryConfigurer.asOrderedList(MeterRegistryConfigurer.java:86) ~[spring-boot-actuator-autoconfigure-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryConfigurer.customize(MeterRegistryConfigurer.java:71) ~[spring-boot-actuator-autoconfigure-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryConfigurer.configure(MeterRegistryConfigurer.java:60) ~[spring-boot-actuator-autoconfigure-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryPostProcessor.postProcessAfterInitialization(MeterRegistryPostProcessor.java:61) ~[spring-boot-actuator-autoconfigure-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:429) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1766) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1463) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1427) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1205) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1167) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:509) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1247) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1167) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:509) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:204) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:235) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:226) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:101) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:88) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:261) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:234) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:54) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5139) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_251]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_251]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_251]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_251]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:423) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:932) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.startup.Tomcat.start(Tomcat.java:455) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:106) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:86) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:415) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:174) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:181) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:154) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        ... 16 more
Caused by: com.vmware.mangle.utils.exceptions.MangleException: Node spin up failed, mandatory parameter validation token is not provided
        at com.vmware.mangle.services.hazelcast.HazelcastConfig.createHazelcastCluster(HazelcastConfig.java:160) ~[classes!/:2.0.0]
        at com.vmware.mangle.services.hazelcast.HazelcastConfig$$EnhancerBySpringCGLIB$$32d44b25.CGLIB$createHazelcastCluster$2(<generated>) ~[classes!/:2.0.0]
        at com.vmware.mangle.services.hazelcast.HazelcastConfig$$EnhancerBySpringCGLIB$$32d44b25$$FastClassBySpringCGLIB$$9d2faa2c.invoke(<generated>) ~[classes!/:2.0.0]
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363) ~[spring-context-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at com.vmware.mangle.services.hazelcast.HazelcastConfig$$EnhancerBySpringCGLIB$$32d44b25.createHazelcastCluster(<generated>) ~[classes!/:2.0.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_251]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_251]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_251]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_251]
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:456) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:307) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1470) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1427) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1273) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1205) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.resolveStream(DefaultListableBeanFactory.java:1928) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.orderedStream(DefaultListableBeanFactory.java:1922) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryConfigurer.asOrderedList(MeterRegistryConfigurer.java:86) ~[spring-boot-actuator-autoconfigure-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryConfigurer.customize(MeterRegistryConfigurer.java:71) ~[spring-boot-actuator-autoconfigure-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryConfigurer.configure(MeterRegistryConfigurer.java:60) ~[spring-boot-actuator-autoconfigure-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryPostProcessor.postProcessAfterInitialization(MeterRegistryPostProcessor.java:61) ~[spring-boot-actuator-autoconfigure-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:429) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1766) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1463) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1427) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1205) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1167) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:509) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1247) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1167) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:509) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:204) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:235) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:226) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:101) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:88) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:261) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:234) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:54) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5139) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_251]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_251]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_251]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_251]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:423) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:932) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.startup.Tomcat.start(Tomcat.java:455) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:106) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:86) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:415) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:174) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:181) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:154) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        ... 16 more

Configuration information:

• Deployment Type: locally
• Deployment Mode: single node
• Client OS: Windows 10

enable running chaos tests in random order

Is your feature request related to a problem? Please describe.
Mangle doesn’t have running faults at random order feature at present. Mangle enables users to define fault schedules to run the faults at specified schedule. Schedule feature enables running of faults in a controlled manner and running of parallel faults on the same target at the same time.

Describe the solution you'd like
we could select ootb fault injections and/or the ones that we created ourselves and then we let them randomly be applied on the environment of our choice - not defined within a specific schedule but at unknown timings - log when what was injected so that we can afterwards do our RCA and check our observability and monitoring tools for lessons learned and adapt processes / workflows

Describe alternatives you've considered
none for the time being

Additional context
none for the time being

Adding Remote host Credentials with private key - Invalid Private Key

Hi Team,

I am trying to add Remote Host credentials using Private Key option and getting "Invalid Private Key" error.

I have .ppk file which I use to connect to my EC2 instance and I am able to successfully ssh to EC2 instance through that ppk file.

But when I use that same ppk file for connecting to EC2 instance through Mangle, I am getting above error.

Please help.

Mangle Version : 2.0

Thanks
Lakshmi P

Error injecting Application fault on the Target java process descriptor : kubernetes pods

Errors seen:

2021-03-12 12:44:21.452 [BytemanFaultTaskHelper-1615552998671-1] INFO com.vmware.mangle.utils.CommonUtils.delayInSeconds (67) - Sleeping for 5 seconds2021-03-12 12:44:21.452 [BytemanFaultTaskHelper-1615552998671-1] INFO com.vmware.mangle.utils.CommonUtils.delayInSeconds (67) - Sleeping for 5 seconds2021-03-12 12:44:27.616 [BytemanFaultTaskHelper-1615552998671-1] ERROR com.vmware.mangle.task.framework.helpers.CommandInfoExecutionHelper.verifyExpectedFailures (135) - Execution of Command: exec -it csp-commerce-cd77db95c-cxxhr -c csp-commerce – /bin/sh -c "export JAVA_HOME=/opt/jre && /tmp/mangle-java-agent-2.0.0/bin/bminstall.sh -p 9091 -s -b csp-commerce.jar" failed. errorCode: 0 output: Unable to use a TTY - input is not a terminal or the right kind of file/tmp/mangle-java-agent-2.0.0/bin/bminstall.sh: line 82: [: .: integer expression expected
java.lang.IllegalArgumentException: Install : invalid pid csp-commerce.jar. Found Processes: at org.jboss.byteman.agent.install.Install.attach(Install.java:557) at org.jboss.byteman.agent.install.Install.main(Install.java:60)java -Djava.library.path=/tmp/mangle-java-agent-2.0.0/bin -classpath /tmp/mangle-java-agent-2.0.0/lib/mangle-byteman-install.jar:/tmp/mangle-java-agent-2.0.0/lib/mangle-byteman.jar:/tmp/mangle-java-agent-2.0.0/lib/mangle-byteman-submit.jar:/tmp/mangle-java-agent-2.0.0/lib/tools.jar org.jboss.byteman.agent.install.Install -p 9091 -s -b csp-commerce.jar

what type of privilege do we need in order to connect to vCenter to perform chaos testing

we need to take advantage of the vCenter/adapter capability provided by Mangle, in order to perform vCenter/VM chaos testing,
when we setup an endpoint of vCenter in Mangle, we need to enter the user ID/password to connect to vCenter API, and vCenter administrator of our company is asking us what user ID/password we need? since he can not give us "admin" user privilege for sure, he is concerned about potential negative impact that our Mangle might have with "admin" privilege, which could affect hundreds of VMs managed in vCenter. so, please provide detailed guidance, since we could not find relevant info from your online documentation about this feature.

Spring Service Latency Fault throws error when providing process descriptor name in "JVM Process" field

BytemanFaultTaskHelper-1615882372248-5
 FAILED
csp-app-preview
INJECTION
Executing Fault: SPRING_SERVICE_LATENCY on endpoint: csp-app-preview. More Details: [ JVMCodeLevelFa
more
Mar 16, 2021, 1:42:44 PM
Mar 16, 2021, 1:43:49 PM
ErrorCode : FI0101, ErrorMessage : Target application is not a Java process or not yet initialized. More details available in mangle log.

Lower Level Capabilities in Network Faults

Describe the solution you'd like

1. Fault-injection on the network ports:
   At present, mangle only supports injecting the network faults such as delay, packet drop on entire network interface. It would be nice to specify the particular port(s) of an interface as well to inject these faults rather than on entire network interface.

2. Three-way Partitioning:
   Mangle supports network partitioning between 2 units - endpoint(s) and target hosts that can split the distributed systems cluster into 2 partitions. 3 partitions or partitioning between 3 entities will be useful to test the resiliency of distributed system especially with the cluster running in the AWS environment where cluster nodes could be present in the different availability zones.

Describe alternatives you've considered
For now, we are using other chaos engineering tools to inject the common network faults on the particular port(s) of a network interface.

K8s drain node attack

Can you please add K8s drain-node attack? This will help in the K8s cluster (like in Azure) where we won't have the ability to add or remove a node from the cluster on the fly.

Thx!

Connection Failure: timeout: socket is not established

Hi,
While adding Remote Machine in Endpoint I am getting an error " Connection Failure: timeout: socket is not established"
Can somebody help me on this?

Mangle Support for Amazon EKS

Mangle team, (@rpraveen-vmware @ashrimalivmware)

I communicated to you team earlier on https://github.com/vmware/mangle/issues/82 & https://github.com/vmware/mangle/issues/105 which were related to chaos setup on OpenShift cluster.

This time we are experiencing different issue. We are trying to have chaos functionality implemented in the AWS EKS environment for the GF applications and we are observing issues with kubeconfig integration. Here are the details,

Scenario

1. Mangle product is running on its own dedicated EKS cluster, lets says, EKS-Mangle
2. Target service (where chaos injection needs to happen) is running on another EKS cluster, let's say, EKS-Target
3. The kubeconfig file of EKS-Target,

EKS-Target-kubeconfig.txt

4. The required IAM policies and EKS RBAC policies have been established.

Problem Statement

While configuring & testing the endpoint connection on EKS-Mangle using the kubeconfig file of EKS-Target, we see failed connection error.
Test Connection failed for endpoint, Please reverify the credentials {0} . Reason: Test Connection failed for endpoint, Please reverify the credentials

Our Initial RCA

1. We enabled the trace level logging on the mangle UI to grep the underlying root cause.
2. We found that the issue is related to AWS authentication from mangle. Mangle package doesn't have the AWS CLI or the AWS-IAM-Authenticator required to authenticate against AWS. Please see trace log below,

exec executable aws not found

Some Useful Resources

1. https://itnext.io/how-does-client-authentication-work-on-amazon-eks-c4f2b90d943b#609a
2. https://github.com/kubernetes-sigs/aws-iam-authenticator
3. https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

From the available documentation on mangle, I do not see any specific use case support for EKS-mangle integration. Is this a limitation at this point?

Thanks
Anvesh

Cannot Add k8s Cluster

Describe the bug
Not able to add k8s cluster to mangle. Mangle is running on Openshift on the same cluster and namespace.
To Reproduce
Steps to reproduce the behavior:

1. Go to Endpoint >> Kubernetes Cluster
2. Click on + K8S Cluster
3. Add any name; add credential(Its running on same cluster); add default as namespace
4. See error

Expected behavior
Should add K8s cluster to Endpoints

Error Output

K8SServerError: Error from server (Forbidden): namespaces "default" is forbidden: User "system:serviceaccount:namespace:default" cannot get namespaces in the namespace "default": no RBAC policy matched

ErrorCode 127 Fault Injection w/ OSError: [ErrNo 22] Invalid Argument in target infra_agent.log

Hi Mangle Team,

I'm currently testing VMWare mangle in my environment without internet connectivity and I've been encountering error code 127 output: ./infra_submit: error while loading shared libraries: libz.so.1: failed to map segment from shared object when injecting CPU fault to target server from Mangle portal.

I have attached 2 log files here:

1. mangle-services-app.logfile (20220718.txt file)
2. target server infra_agent log file (infra_agent.txt) file

for your reference as I'm currently stuck on what's causing this issue. Appreciate your help on this, thank you in advance!

Xander

20220718.txt
infra_agent.log

Can't run mangle in docker

Describe the bug
I've been trying to run mangle in docker, and spring throws an exception and exits

To Reproduce
Steps to reproduce the behavior:

1. Go to 'terminal'
2. Click on '....', run docker run -ti --rm -p 8080:8080 -p 8443:8443 mangleuser/mangle:2.0
3. Scroll down to '....'
4. See error

Expected behavior
Mangle runs

Screenshots
If applicable, add screenshots to help explain your problem.

Logs

Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.TransportException: [localhost/127.0.0.1:9042] Cannot connect))
	at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:268) ~[cassandra-driver-core-3.6.0.jar!/:?]
	at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:107) ~[cassandra-driver-core-3.6.0.jar!/:?]
	at com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1652) ~[cassandra-driver-core-3.6.0.jar!/:?]
	at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1571) ~[cassandra-driver-core-3.6.0.jar!/:?]
	at com.datastax.driver.core.Cluster.init(Cluster.java:208) ~[cassandra-driver-core-3.6.0.jar!/:?]
	at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376) ~[cassandra-driver-core-3.6.0.jar!/:?]
	at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:355) ~[cassandra-driver-core-3.6.0.jar!/:?]
	at com.datastax.driver.core.Cluster.connect(Cluster.java:305) ~[cassandra-driver-core-3.6.0.jar!/:?]
	at com.vmware.mangle.services.cassandra.CassandraClusterFactoryBean.executeSpecsAndScripts(CassandraClusterFactoryBean.java:349) ~[classes!/:2.0.0]
	at com.vmware.mangle.services.cassandra.CassandraClusterFactoryBean.afterPropertiesSet(CassandraClusterFactoryBean.java:231) ~[classes!/:2.0.0]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1821) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1758) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:339) ~[spring-context-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at com.vmware.mangle.services.cassandra.CassandraConfig$$EnhancerBySpringCGLIB$$8edbffbd.cluster(<generated>) ~[classes!/:2.0.0]
	at com.vmware.mangle.services.cassandra.AbstractClusterConfiguration.getRequiredCluster(AbstractClusterConfiguration.java:58) ~[classes!/:2.0.0]
	at com.vmware.mangle.services.cassandra.CassandraConfig$$EnhancerBySpringCGLIB$$8edbffbd.CGLIB$getRequiredCluster$38(<generated>) ~[classes!/:2.0.0]
	at com.vmware.mangle.services.cassandra.CassandraConfig$$EnhancerBySpringCGLIB$$8edbffbd$$FastClassBySpringCGLIB$$c0de4937.invoke(<generated>) ~[classes!/:2.0.0]
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363) ~[spring-context-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at com.vmware.mangle.services.cassandra.CassandraConfig$$EnhancerBySpringCGLIB$$8edbffbd.getRequiredCluster(<generated>) ~[classes!/:2.0.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0-internal]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0-internal]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0-internal]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0-internal]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:456) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.resolveBeanReference(ConfigurationClassEnhancer.java:394) ~[spring-context-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:366) ~[spring-context-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at com.vmware.mangle.services.cassandra.CassandraConfig$$EnhancerBySpringCGLIB$$8edbffbd.getRequiredCluster(<generated>) ~[classes!/:2.0.0]
	at com.vmware.mangle.services.cassandra.AbstractCassandraConfiguration.session(AbstractCassandraConfiguration.java:106) ~[classes!/:2.0.0]
	at com.vmware.mangle.services.cassandra.CassandraConfig$$EnhancerBySpringCGLIB$$8edbffbd.CGLIB$session$26(<generated>) ~[classes!/:2.0.0]
	at com.vmware.mangle.services.cassandra.CassandraConfig$$EnhancerBySpringCGLIB$$8edbffbd$$FastClassBySpringCGLIB$$c0de4937.invoke(<generated>) ~[classes!/:2.0.0]
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363) ~[spring-context-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at com.vmware.mangle.services.cassandra.CassandraConfig$$EnhancerBySpringCGLIB$$8edbffbd.session(<generated>) ~[classes!/:2.0.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0-internal]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0-internal]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0-internal]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0-internal]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:622) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:456) ~[spring-beans-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]

Configuration information:

• Deployment Type: Container
• Deployment Mode: Single Node
• Client OS: iOS
• Client Browser: chrome
• Version [e.g. 22]

Additional context
NA

Doc: README file should begin

Describe the bug
Readme file for a project should begin with a brief high-level explanation of its purpose.
See https://github.com/kubernetes/kubernetes and https://github.com/grafana/grafana

Mangle Vcenter Adapter help for Kubernetes deployment of Mangle

Hello Mangle Team,

I am trying to connect Mangle to a Vcenter endpoint. After reading the documentation and watching a demo video on Youtube, I believe I need to have mangle-vc-adapter deployed in order to interact with the Vcenter. We have deployed Mangle using the Kubernetes files. I know there is a Docker command in the documentation to install the adapter, is there a similar kube command or some other way to deploy the adapter to our mangle instance? Thank you in advance for your help.

Thanks,
Tommy

unable to do a docker installation of mangle

Hi,

I am trying to deploy mangle using docker containers by executing the following commands in Windows (Docker Desktop):

run --name mangle-cassandradb -v c:/cassandra/storage/:/var/lib/cassandra -p 9042:9042 -d -e CASSANDRA_CLUSTER_NAME="manglecassandracluster" -e CASSANDRA_DC="DC1" -e CASSANDRA_RACK="rack1" -e CASSANDRA_ENDPOINT_SNITCH="GossipingPropertyFileSnitch" <internalrepo>/cassandra:3.11    


docker run --name mangle -it --rm -e DB_OPTIONS="-DcassandraContactPoints=<cassandra ip> -DcassandraSslEnabled=false -DcassandraUsername=cassandra -DcassandraPassword=cassandra" -e CLUSTER_OPTIONS="-DclusterValidationToken=mangle -DpublicAddress=127.0.0.1" -p 8080:8080 -p 8443:8443 <internalrepo>/mangleuser/mangle:3.0.0

Both containers are running but when trying in the browser with 127.0.0.1 for both 8080 and 8443 I get:

HTTP Status 404 – Not Found

Moreover I see no errors in the logs of mangle container.

Thank you,
John

Failing to inject Network Fault

Describe the issue:
I am trying mangle deployed in a openshift environment. I am trying to inject a network fault (packet delay or packet duplication) on a remote linux machine within my organizational network but the networkFault script is failing with an error as below:

com.vmware.mangle.utils.exceptions.MangleException: Precheck Failed with pre-requisites : tc is required, ip is required

The networkFault script got successfully transferred to the machine followed by successful SSH connection to execute the command. The command execution is failing.

Steps to reproduce:

1. Open mangle via browser, add a remote linux machine as End Point with its uid/pwd as credentials.
2. Go to Fault Injection --> Network Faults --> Packet Delay
3. Select remote linux machine as EndPoint, eth0 as Nic name, 1s packet delay, 3s time out
4. Run Fault.

Logs:
2020-04-24 16:25:07.640 [SystemResourceFaultTaskHelper-1587745499241] DEBUG com.vmware.mangle.task.framework.helpers.CommandInfoExecutionHelper.getAbsoluteCommand (195) - Absolute Command is /home/infra1/temp/networkFault.sh --operation=inject --faultOperation=NETWORK_DELAY_MILLISECONDS --latency=1000 --percentage=0 --nicName=eth0 --timeout=5000
2020-04-24 16:25:07.641 [SystemResourceFaultTaskHelper-1587745499241] INFO com.vmware.mangle.utils.clients.ssh.SSHUtils.runCommandReturningResult (156) - Running Command ...
2020-04-24 16:25:10.471 [SystemResourceFaultTaskHelper-1587745499241] DEBUG com.vmware.mangle.utils.clients.ssh.SSHUtils.runCommandReturningResult (165) - SSH Connected Successfully

2020-04-24 16:25:11.473 [SystemResourceFaultTaskHelper-1587745499241] DEBUG com.vmware.mangle.utils.clients.ssh.SSHUtils.runCommandReturningResult (169) - Command-output: Precheck Failed with pre-requisites : tc is required, ip is required

2020-04-24 16:25:11.474 [SystemResourceFaultTaskHelper-1587745499241] DEBUG com.vmware.mangle.utils.clients.ssh.SSHUtils.runCommandReturningResult (170) - exit-status: 127
2020-04-24 16:25:11.477 [SystemResourceFaultTaskHelper-1587745499241] ERROR com.vmware.mangle.services.tasks.executor.TaskExecutor.runTask (231) - Task Execution Failed. Reason: ErrorCode : FI0015, ErrorMessage : Execution of Command: /home/infra1/temp/networkFault.sh --operation=inject --faultOperation=NETWORK_DELAY_MILLISECONDS --latency=1000 --percentage=0 --nicName=eth0 --timeout=5000 failed. errorCode: 127 output: Precheck Failed with pre-requisites : tc is required, ip is required
.
com.vmware.mangle.utils.exceptions.MangleException: Precheck Failed with pre-requisites : tc is required, ip is required

Expected Behavior:
Packet delay should have been injected on to the machine and the device eth0

Additional Info:
I tried a similar script (networkFault.sh) on the machine directly and able to inject the fault. I used the same user credentials as entered in mangle. I also verified the tc & ip utilities are available on the machine and able to run them manually.

Mangle 3.0 Stability - Cassandra DB goes down suddenly

Environment: OpenShift v4.6.36
Kubernetes Version: v1.19.0
Mangle Version: 3.0
Issue:

1. Cassandra DB goes down with failed connections causing mangle POD to do multiple retries on the cassandra DB
2. Mangle product UI is not available for this entire duration

Interim Solution Being Followed:

1. Restart Cassandra POD
2. Restart mangle POD
3. Increase the resource limits on cassandra statefulset template as recommended by the mangle team during working session.

Previous:

 - resources:
       limits:
           cpu: '1'
           memory: 8Gi
        requests:
           cpu: '500m'
           memory: 2Gi

Current:

 - resources:
       limits:
           cpu: '2'
           memory: 8Gi
        requests:
           cpu: '1'
           memory: 4Gi

Frequency Of This Issue: Once every few weeks. Typically 7-8 weeks but it may be random too.

Logs:

1. Please find the attached logs from mangle & cassandra POD's when this issue downtime happened recently in the last week of February, 2022

cassandra_pod_failure_0227.txt
mangle_pod_failure_0227.txt

Deployment Templates:

1. Please find the attached cassandra statefulset & mangle deployment template resource
   cassandra_statefulset_template.txt
   mangle_deployment_template.txt

Unable to Run Mangle Jar

Describe the bug
While trying to run the mangle-services.jar after building it gives error saying:

org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is java.lang.IllegalStateException: Unable to create the directory [C:\home\mangle\var\opt\mangle-tomcat] to use as the base directory

The user doesn't actually have access to the /home so its throwing the error, is there a way I can change the path of base dir for mangle?

To Reproduce
Steps to reproduce the behavior:

1. git clone mangle
2. run mvn clean install
3. cd mangle-service/target
4. run java –jar mangle-services-.x.x.x-jar –D......

Expected behavior
Should start server and able to run the mangle app on browser(localhost)

Logs

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-05-04 19:04:32.577 [main] ERROR com.vmware.mangle.utils.exceptions.handler.MangleUncaughtExceptionHandler.uncaughtException (67) - Uncaught exception caught in thread :main
org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is java.lang.IllegalStateException: Unable to create the directory [C:\home\mangle\var\opt\mangle-tomcat] to use as the base directory
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:157) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543) ~[spring-context-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:142) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:316) [spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) [spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) [spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at com.vmware.mangle.MangleApplication.main(MangleApplication.java:36) [classes!/:2.0.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_74]
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_74]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_74]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_74]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [mangle-services.jar:2.0.0]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [mangle-services.jar:2.0.0]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) [mangle-services.jar:2.0.0]
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52) [mangle-services.jar:2.0.0]
Caused by: java.lang.IllegalStateException: Unable to create the directory [C:\home\mangle\var\opt\mangle-tomcat] to use as the base directory
        at org.apache.catalina.startup.Tomcat.initBaseDir(Tomcat.java:815) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.startup.Tomcat.getServer(Tomcat.java:624) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.apache.catalina.startup.Tomcat.getService(Tomcat.java:557) ~[tomcat-embed-core-9.0.19.jar!/:9.0.19]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:165) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:181) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:154) ~[spring-boot-2.1.3.RELEASE.jar!/:2.1.3.RELEASE]
        ... 16 more
2020-05-04 19:04:32.694 [main] ERROR com.vmware.mangle.utils.exceptions.handler.MangleUncaughtExceptionHandler.printFullCoreDump (91) - \home\mangle\var\opt\mangle-tomcat\logs\mangle-thread-dump-2020-05-04-19-04-32-602.log

Configuration information:

• Deployment Type: locally
• Deployment Mode: single node
• Client OS: Windows 10

Build/Debug Python fault scripts - requirements.txt to build infra_agent.tar.gz

I'm attempting to use Mangle to inject faults into a BOSH managed VM and I need to debug and modify the Python scripts that are running on the remote VM. The Python fault injection scripts that are dumped onto the VM look to be compiled so I've attempted to clone the repo to the target VM and modify them from there, however that raises the following issues:

1. The infra_agent.py script gets an error importing the mangleinfraagent module unless I move infra_agent.py to the parent folder so that mangleinfraagent is a subfolder relative to the script.
2. There's no pip requirements.txt or dependency docs so cloning the repo and attempting to manually run the scripts requires a lot of fail, pip install, try again.
3. How do I build a new infra_agent.tar.gz? I don't see a build script or manual docs on how to do this.

These appear to be the dependencies required to get the infra_agent to start.

• psutil
• distro
• psycopg2
• pymongo
• cassandra-driver

Dynatrace support for Resiliency Score - Quantifying fault tolerance using metrics

Dynatrace support for Resiliency Score reporting.

Service Unavailable - Random injection does not work

Describe the bug
Service Unavailable fault in Infrastructure Faults for K8S - random injection 'true' does not do randomly, instead it works similar to random injection 'false'. For example, if there 2 PODs running and Service Unavailable with Random Injection 'true', injects faults on both running PODs.

To Reproduce
Steps to reproduce the behavior:

1. Infrastructure - K8S - Service Unavailable
2. Select endpoint for K8S
3. Select Service Labels and add Key/Value
4. Random Injection 'True'
5. Run Fault
6. Execute

Expected behavior
Expected to see the requests to one of the PODs fail with service unavailable issue. Requests to both PODs fail with service unavailable

Screenshots
If applicable, add screenshots to help explain your problem.

Logs
If applicable, add application logs/troubleshooting bundle to help in root cause analysis.

Configuration information:
Environment: OpenShift v4.6.36
Kubernetes Version: v1.19.0
Mangle Version: 3.5

Additional context
Add any other context about the problem here.

Add support for elastic kibana integration with mangle to view the fault metrics on kibana

This is a feature request to support new metric monitoring tool.

ERR_CONNECTION_CLOSED accessing Mangle Web

Hi Mangle Team,

Need assistance as Mangle WebUI is not accessible via browser and always says the error connection closed.

Have verified the ports and firewall rules and tnc and curl/telnet are working.

docker containers are running and up:

When I curl -kv https:///mangle-services from photon-os server:

When I curl -kv https:///mangle-services from remote:

Have tried restarting the containers 1 by 1 mangleDB first, then mangleWeb, it's still the same issue. Please note that the browsers cache have been cleared as well and it's still the same issue. There's no notable errors also in the mangleDB and mangleWeb docker containers as the errors shown have the same error logs of our other running instance of mangle. Please kindly help asap as I'm currently stuck on what's causing this issue.. I have tried for the last 2 days. Thank you in advance!

Official Slackclient available

Thanks for the OOTB Slack integration feature.
A small request related to the slackclient being used: The slackclient is from an old git repo without many recent updates. Can you please consider upgrading it to the official Slackclient provided/maintained by Slack team - https://slack.dev/java-slack-sdk/guides/web-api-basics.

Please let me know if I can send a PR for the same?

Thanks!

Unable to connect docker which is hosted in the remote machine.

I am trying to connect docker from mangle and i am end up with this error message. Please help me out and refer the attachment.

Container deployment issue

Hi Team,
We are trying to deploy Mangle3.0 container images in ECS Fargate. We have not done any changes to images so far. while deploying we are getting the below error for Cassandra:
Cassandra is unavailable-sleeping
Connection error:('Unable to connect to any servers'{'127.0.0.1':error(111,"Tried connecting to [(127.0.0.1,9042)].Last error:Connection refused")})

while trying from EC2 we got the same error but we can connect to Cassandra through cqlsh.
Verified the TaskDefinition with AWS support team already.

Thanks!!

What kind of credential is needed for infrastructure faults?

What kind of credential and permission is required by mangle to run injections on remote endpoints to execute infrastructure faults? Im testing in our environment and using a credential with local admin rights but still facing permission denied issues.

./infra_submit --operation inject --faultname cpuFault --load 85 --timeout 3 --faultId cpuFault
errorCode 126 output: bash: ./infra_submit: Permission denied

Mangle keeps posing for password change very frequently, not sure if this is expected behaviour.

Describe the bug
Mangle keeps posing for password change very frequently, not sure if this is expected behaviour. The user id I use is "admin"

To Reproduce
Steps to reproduce the behavior:

1. Open url https://manglenamespace/mangle-services/
2. It poses password change

Expected behavior
Should login the user and not pose change password screen so frequently

Build fails due to dependencies from VMware Eng repo

Describe the bug

To Reproduce
Steps to reproduce the behavior:
module : om.vmware.mangle:mangle-models
mvn install

Logs
Could not transfer artifact com.fasterxml.jackson.datatype:jackson-datatype-joda:pom:2.11 from/to build-artifactory.eng.vmware.com (https://build-artifactory.eng.vmware.com/artifactory/fiaasco-release): build-artifactory.eng.vmware.com: nodename nor servname provided, or not known

Monitoring Tool -Integration Requirement

Navigation: Settings-> Integration-> Metric-Providers
Similar to Datadog, and WaveFront we would like to have Applicare monitoring tool integrated with the Mangle.
Please do suggest and advice on feasibility of Applicare integration.

Unable to add vCenter adapter

Unable to add vCenter adapter in Mangle 3.5 version: As per document I have followed troubleshooting steps.

1. Verified vCenter adapter Swagger URL working

2. curl -k https://192.168.225.240:8443/mangle-vc-adapter/application/health - working

{"status":"UP"}r

Timeout Milliseconds in Under Fault Injection Tab

We have couple of request on the timeout milliseconds

1. Naming convention change request-Timeout milliseconds to Duration.
2. Dropdown request for selecting the time frequency (eg: ms, seconds, minutes & hr)

Can't connect to Mangle UI

I am getting the following error:

Bad Request
This combination of host and port requires TLS.

Java Method Latency Not Working

Describe the bug
When I execute Java Method Latency fault with 20,000ms latency at specific method and class from mangle application, it is showing in logs that "Completed Fault Injection". However when I make a manual API call to the service, I do not observe any latency close to 20 seconds added to service response. The response time of this API without Network latency is typically ~5 seconds and are able to verify in Application logs that it invokes the target Java Class and Method. I was able to successfully execute other Application Fault – Java - Burn CPU on target JVM process running on K8s POD and able to observe spike in JVM process CPU utilization. However I am unable to execute Application Fault - Java method latency successfully.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'Mangle Application -> Login -> Fault Execution -> Application Faults -> Java Method Latency'
2. Enter All the details - K8s endpoint, Class, Method, Latency (20,000 ms), Rule Event, K8s arguments and JVM Properties of a springboot service
3. Click on 'Run Fault'
4. Go to Requests and Reports - 'Wait for the Fault to Complete by clicking the refresh button'
5. Now Test the service(Which will hit the class and method mentioned in step 2) by making manual API call using postman or curl command and check its response time.

Expected behavior
The response of the service should be > 20,000ms.

Screenshots
If applicable, add screenshots to help explain your problem.

Logs
If applicable, add application logs/troubleshooting bundle to help in root cause analysis.
2020-12-22 17:01:15.009 [BytemanFaultTaskHelper-1608656456217-1] INFO com.vmware.mangle.task.framework.helpers.AbstractCommandExecutionTaskHelper.triggerFaultInjectionStage (165) - Completed Fault Injection

Configuration information:

• Deployment Type: Container
• Deployment Mode: Single Node
• Client OS: Windows
• Client Browser: Chrome
• Version: 84.0.4147.125

Additional context
Add any other context about the problem here.
We tried to debug this by reviewing the Mangle agent processes created on POD and observed multiple parent and child processes spawned by mangle on target POD. We also notice that all these processes are terminated in ~5 seconds of Chaos test execution.

jboss 4104 0 5 06:20 ? 00:00:00 tar -xmf - -C /tmp/mangle-java-agent-2.0.0
jboss 4118 0 0 06:20 ? 00:00:00 /bin/sh -c export JAVA_HOME= && /tmp/mangle-java-agent-2.0.0/bin/bminstall.sh -p 9091 -s -b 639
jboss 4125 4118 0 06:20 ? 00:00:00 /bin/bash /tmp/mangle-java-agent-2.0.0/bin/bminstall.sh -p 9091 -s -b 639
jboss 4126 4125 0 06:20 ? 00:00:00 /bin/bash /tmp/mangle-java-agent-2.0.0/bin/bminstall.sh -p 9091 -s -b 639
jboss 4213 0 0 06:20 ? 00:00:00 /bin/sh /tmp/mangle-java-agent-2.0.0/bin/bmsubmit.sh -p 9091 /tmp/mangle-java-agent-2.0.0/70bb3609-d61f-49fb-b025-3f7064c9854d.btm
jboss 4221 4213 0 06:20 ? 00:00:00 /bin/java -classpath /tmp/mangle-java-agent-2.0.0/lib/mangle-byteman.jar:/tmp/mangle-java-agent-2.0.0/lib/mangle-byteman-submit.jar org.jboss.byteman.agent.submit.Submit -p 9091 /tmp/mangle-java-agent-2.0.0/70bb3609-d61f-49fb-b025-3f7064c9854d.btm

Since we did not notice any latency induced, we tried to manually re-execute the command “/bin/sh -c export JAVA_HOME=/pkgs/openjdk8 && /tmp/mangle-java-agent-2.0.0/bin/bminstall.sh -p 9091 -s -b 639” on target POD and noticed that listener was running with an error “Agent is already running on requested process”.

Below is the snippet of output of the command.

/tmp/mangle-java-agent-2.0.0/lib/mangle-byteman.jar
org.jboss.byteman.agent.FiaascoTransformListener- I am here on Pid: 639

java.lang.IllegalStateException: Agent is already running on requested process
at org.jboss.byteman.agent.install.Install.notifyRetriedInstallation(Install.java:98)
at org.jboss.byteman.agent.install.Install.isPortAvailable(Install.java:75)
at org.jboss.byteman.agent.install.Install.injectAgent(Install.java:613)
at org.jboss.byteman.agent.install.Install.main(Install.java:61)

We tried to execute bmsubmit.sh with few options to list the already executed faults or currently running faults, which returned empty list.

Below is the logs.
sh-4.2$ /tmp/mangle-java-agent-2.0.0/bin/bmsubmit.sh -laf
{}

sh-4.2$ /tmp/mangle-java-agent-2.0.0/bin/bmsubmit.sh -llf
{}

sh-4.2$ /tmp/mangle-java-agent-2.0.0/bin/bmsubmit.sh -gaf
{}

sh-4.2$ pwd
/tmp/mangle-java-agent-2.0.0
sh-4.2$ ls
537eff34-2f04-446c-b7d0-e624b99251e4.btm bin de583ce6-27cf-400e-9dfa-4df65cc8f0dc.btm mangle-java-agent-2.0.0
70bb3609-d61f-49fb-b025-3f7064c9854d.btm c4a0a9e7-e6a6-421e-a68c-8fe8dab1c8ac.btm fdd9ac8f-2260-4e9b-b3d0-f350b2799a32.btm
a2761878-dd89-48b8-96c4-9ac855f99db3.btm ddd7ccaf-c062-41a8-9813-02a972a0ebfe.btm lib
sh-4.2$

mangleWEB unable to start

Fresh OVA deployment.. have to login to the appliance to request DHCP address, otherwise no IP on first boot. mangleDB container seems to run fine however mangleWEB fails with the following error..

vcenter 6.7 build 11727113
vsphere 6.7 build 10302608
virtual-wire used for vnic

root@photon-machine [ ~ ]# docker logs mangleWEB | more | grep ERROR

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /home/mangle/var/opt/vmware/mangle/cert/server.jks -destkeystore /home/mangle/var/opt/vmware/mangle/cert/server.jks -deststoretype pkcs12".
2019-11-28 22:09:39.951 [main] ERROR org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup (62) - Error starting Tomcat context. Exception: org.springframework.beans.factory.UnsatisfiedDependencyException. Message: Error creating bean with name 'webMvcMetricsFilter' defined in class path resource [org/springframework/boot/actuate/autoconfigure/metrics/web/servlet/WebMvcMetricsAutoConfiguration.class]: Unsatisfied dependency expressed through method 'webMvcMetricsFilter' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'compositeMeterRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/metrics/CompositeMeterRegistryConfiguration.class]: Unsatisfied dependency expressed through method 'compositeMeterRegistry' parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'initializeDatadogMeterRegistry' defined in class path resource [com/vmware/mangle/services/config/MangleMicrometerConfig.class]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [com/vmware/mangle/services/hazelcast/HazelcastConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'createHazelcastCluster' threw exception; nested exception is com.vmware.mangle.utils.exceptions.MangleException: Node spin up failed, mandatory parameter public address for the mangle instance is not provided

API auth type?

Describe the bug
Looking to use the APIs. Found the Swagger spec, but unclear what auth type is used.

Tried using basic auth, but that doesn't appear to be it. Not finding any OAuth-related APIs either, so unable to retrieve a token (or create one via the UI).

"timestamp": 1579295166847,
"code": "FI0132",
"description": "Invalid username or password",
"details": "/mangle-services/rest/api/v1/metric-provider-controller/"

To Reproduce
Steps to reproduce the behavior:

1. Load any API, i.e. https://host/mangle-services/rest/api/v1/metric-provider-controller/?metricProviderByStatus=ALL

Expected behavior
An API response is returned.

Public roadmap?

In reading the other issue, @aswathy-ramabhadran mentioned a January 2021 release.
This got me thinking that it would be nice if the Mangle roadmap was published (i.e. in GitHub) where we could all access it.

Is this something that we could consider doing?

And related to this, do you see any other opportunities for us to stay in lockstep with each other? I cannot find any recent commits on the repo, so I'm assuming that work is being done elsewhere?

Network Partition Fault: Not working as expected

Describe the bug
I am trying to inject network partition fault in a 4 Node cluster with EC2 instances where 2 nodes are specified as an endpoint group and other 2 nodes are defined as hosts which need to be partitioned from given endpoint group. I see that only 1 out of 2 hosts is partitioned from the network and remaining nodes still remain and form a cluster of 3 nodes that includes one of the hosts.

To Reproduce
Steps to reproduce the behavior:

1. Create a Cluster of 4 nodes with Aerospike database Server (https://github.com/aerospike/aerospike-server.git) in AWS EC2 instances.
2. Add 2 of the nodes from cluster as an remote endpoint in Mangle and create and endpoint group from these 2 nodes
3. Inject Network partition fault from Mangle UI with other 2 nodes as the hosts.
4. Fault is successfully injected in Mangle. However, I observed that only 1 of the given 2 hosts was removed from the cluster of 4 nodes.

Please let me know if the more details are required on any step to reproduce the issue.

Expected behavior
Both of the given hosts should be the removed from the cluster and only 2 nodes (which were defined as endpoints) should be the part of the remaining cluster.

Screenshots
N/A

Logs
If applicable, add application logs/troubleshooting bundle to help in root cause analysis.

Configuration information:

• Deployment Type: Container
• Deployment Mode: Single Node
• Client OS: Ubuntu 20.04
• Client Browser : Firefox
• Version 94.0

Additional context
Add any other context about the problem here.

Unable to add Prometheus configurations as metrics provider

Describe the bug
I am unable to add Prometheus configurations as a metrics provider. The UI is not giving me any option to edit or delete the same, is there any documentation on how to integrate prometheus with Mangle ? If so please share.

To Reproduce
Deployed Mangle over EKS.
Steps to reproduce the behavior:

1. Go to 'Settings -> Integrations -> Metrics Providers -> Prometheus (Predefined)'
2. Refer to attached snippet

Expected behavior
Similar to configurations available while trying to add Wavefront/Datadog/Dynatrace as metrics provider, it should take config details for prometheus as well. But currently its just showing it in UI with no editing allowed.

Screenshots
Attached screenshot for reference.

Configuration information:

• Deployment Type: Container
• Deployment Mode: Single Node
• Client Browser: Chrome
• Version: 3.5.0

MySQL support

Hi Mangle team!

Since adopting Mangle for use within our company we've had to make a number of changes.
Aside from addressing vulnerabilities this primarily required swapping out Cassandra for MySQL (we could've gone with Aerospike, but for other reasons decided to go with MySQL).

I could either publish these changes through a public fork (since they're not backwards compatible), or we could send you a PR with the changes and leave you to make that determination how best to incorporate these changes.

Is this something you would be interested in? If so, how would you like to receive the changes?

Thanks!