My purpose in life is to cleanup stale apps and services on a Cloud Foundry foundation. I can be configured to report on and remove orphaned services and stopped app instances older than a configurable duration. I do many other useful things too.
License: Apache License 2.0
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot]")
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
The design of ResourceNotificationPolicyExecutorTask and ResourceNotificationPolicy only really allow for existence matching. Label values in ResourceEmailMetadata are just email accounts which have an email domain appended.
What if we wanted to apply multiple labels to a resource and then find all resources matching a selector (which could be a combination of label key-value pair conditions), then notify an owner/designee? See selector requirement examples in table here.
The current implementation does not support this use case.
What we really want to be able to do is to notify someone (a notifyee) of a resource matching a selector.
So, maybe, if we consistently applied a label key notifyee where the value is always an email account or distribution-list alias stripped of its domain along with any other combo of label key-value pairs to a supported ResourceType, we could use that to send notifications. Perhaps we make a notifyee label-key configurable (per policy definition)?
it is my understanding that currently cfbutler operates at an org level. is there any possibility to have it work at space level. our current desired use would be to exclude specific applications from cleanup. based on current documentation this does not seem possible.
Unfortunately, cf-butler makes use of some Java 11 features. Turns out there's an open issue logged here. As soon as this is resolved, and we have a new release of GraalVM, I'll give this a try.
We're getting an exception on execution of the policy.
INFO [reactor-http-nio-3] i.p.c.t.StackChangeAppInstancesPolicyExecutorTask: AppDetail(pk=28, organization=zoo-labs, space=dev, appId=6bee0960-e60c-4ebe-b437-499aee922e2b, appName=cloud-native-spring, buildpack=java, image=, stack=cflinuxfs2, runningInstances=1, totalInstances=1, memoryUsage=258104599, diskUsage=174395392, urls=[cloud-native-spring-hilarious-quokka.apps.dev.cloudmonk.me], lastPushed=2019-07-12T19:32:02, lastEvent=audit.app.ssh-authorized, [email protected], lastEventTime=2019-07-12T19:34:01, requestedState=started) is a candidate for stack change using policy ApplicationPolicy(pk=3, id=675cc5f5e2b9aa7ef1069989c6876e0532d8413a, operation=change-stack, description=Change the stack for applications restricted to whitelisted organizations, state=started, options={stack-from=cflinuxfs2, stack-to=cflinuxfs3}, organizationWhiteList=[zoo-labs, jujubees, full-beaker]).
2019-07-12T19:44:54,902 INFO [cloudfoundry-client-nio-3] i.p.c.t.StackChangeAppInstancesPolicyExecutorTask: Attempting to stage package Relationship{id=https://api.sys.dev.cloudmonk.me/v3/packages/62c24176-b5a0-4abd-8669-0aa1b2bc8fae}
2019-07-12T19:44:54,908 INFO [cloudfoundry-client-nio-4] i.p.c.t.StackChangeAppInstancesPolicyExecutorTask: Attempting to stage package Relationship{id=https://api.sys.dev.cloudmonk.me/v3/packages/62c24176-b5a0-4abd-8669-0aa1b2bc8fae}
2019-07-12T19:44:55,131 ERROR [cloudfoundry-client-nio-6] r.u.Loggers$Slf4JLogger: Operator called default onErrorDropped
org.cloudfoundry.client.v3.ClientV3Exception: CF-UnprocessableEntity(10008): Unable to use package. Ensure that the package exists and you have access to it.
at org.cloudfoundry.reactor.util.ErrorPayloadMapper.lambda$null$2(ErrorPayloadMapper.java:55)
(This is just an idea. I realize it's probably way more work / complexity than the value it might bring. Just puttin' it out there, though)
As a PCF operator who shares responsibilities with others, in order to change the set of policies used by cf-butler, I could see value in being able to store policies in plaintext within a git repo, and have cf-butler update the policies it uses directly from the repository.
This allows traceability (who set this policy and why?), makes it easier to submit changes (just set a PR and when it's merged to master it's slurped up by cf-butler) and version-controlling of the policies used.
Cf-butler already provides endpoints for capturing product and release data from both the Pivotal Network and an Operations Manager. Why not combine this data into a useable report for operators?
Something like
from Operations Manager
from Pivotal Network
For each product calculate the # of days between the installed version and the latest available version.
This would provide operations teams an at-a-glance view of how fresh each installed product is. It might help drive a service level objective on product freshness. E.g., the platform team will ensure through automation or other means that installed product versions available on each managed foundation is only ever N-2 versions behind.
In order to view logging on actions cf-butler took, GET /reports can be used. However it's unclear if there is an intent to use a persistent data service (mysql) to store those reports for longer than the life of the instance.
Disclaimer: I didn't look too deep in the source code for it, but figure it might be worth pointing out that it's something I looked for in the docs.
Currently policy execution is limited to auditing actions taken as defined by each policy. Users must swallow all historical records from the /policies/report endpoint. But what if a user is only interested in a subset of the results after one or more policy executions?
Let's make cf-butler even more useful by adding a key-value pair to the options property-map of each policy. Key-value pair might be:
"options": {
...
"recipients": [
"[email protected], [email protected]"
]
}
Immediately after policy execution and on detection of the recipients property an email would be sent to all recipients. The email might look like:
To: recipient-1
From: [email protected]
Subject: yyyy-mm-dd HH:mm:ss ***PolicyExecutionTask results
Please be informed that *** PolicyExecutionTask was executed on yyyy-mm-dd HH:mm:ss. The results are herewith attached for your consideration.
Kindly,
The Platform Team
The attachment would contain just the subset of historical records (i.e., the audit trail) that were produced upon successful completion of policy execution.
Consider externalizing configuration for: from, subject, template of message body. Opt for notifications generated from Java Mail or Send Grid.
Seen when running cf-butler on foundations hosting 100s to 1000s of application and/or service instances...
2019-10-24T08:19:23.019-07:00 [APP/PROC/WEB/0] [OUT] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
2019-10-24T08:19:23.019-07:00 [APP/PROC/WEB/0] [OUT] at java.base/java.lang.Thread.run(Unknown Source)
2019-10-24T08:19:23.023-07:00 [APP/PROC/WEB/0] [OUT] [30m2019-10-24T15:19:22,962[m [1;31mERROR [m[[1;34mparallel-3[m] [1;33mi.p.c.t.AppDetailTask[m: AppDetailTask terminated with error
2019-10-24T08:19:23.023-07:00 [APP/PROC/WEB/0] [OUT] reactor.core.Exceptions$CompositeException: Multiple exceptions
2019-10-24T08:19:23.023-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.Exceptions.multiple(Exceptions.java:107)
2019-10-24T08:19:23.023-07:00 [APP/PROC/WEB/0] [OUT] Suppressed: reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms
2019-10-24T08:19:23.023-07:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:317)
2019-10-24T08:19:23.023-07:00 [APP/PROC/WEB/0] [OUT] Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
2019-10-24T08:19:23.023-07:00 [APP/PROC/WEB/0] [OUT] Assembly trace from producer [reactor.core.publisher.MonoFlatMap] :
We have a wealth of data collected in SnapshotDetail.
While cf-hoover-ui exists to present and filter multiple foundations' data by application and service instance, it might be interesting to facilitate filtering and push notifications of ad-hoc subsets of this data on one foundation.
Envisioning a new SnapshotQueryPolicy (-QP suffixed) like
{
"description":"Query snapshot detail data for all running Docker containers",
"query": "select * from app_detail where image is not null and requested_state = 'started'"
}
Note query above is just plain old SQL. We would just emit each row in the resultset as a comma-separated value list and add as an attachment.
There are some drawbacks from a data sensitivity standpoint, but this opens up all manner of possible filter and join options for data collected.
We shouldn't leave the endpoints open. Too much potential for malicious activity (particularly if no whitelist is defined per policy or if a user specified alternative instance states, like running).
First define enums for app and service instances states. Then add Spring Security to the party, see https://www.baeldung.com/spring-security-5-reactive for some examples of how to get this done.
Seems like we should leave open the GET endpoints and secure the POST and DELETE endpoints.
Checks for not null managers when saving developers.
With the acquisition of Pivotal by VMWare, and the recent announcements regarding the Tanzu portfolio the README should be updated to reflect these updates. Some hyperlinks are now broken and will need to be fixed.
Hello @pacphi , we are trying to put in place some policies , to stop old applications (which have not been pushed since a long period) .
I took as a base example the policy file cf-butler-sample-config/blob/master/stop-applications-policy-sample-AP.json .
Mine is the following :
{ "description":"Stop applications that were last_pushed 1 months earlier than now", "operation":"stop", "state":"started", "options":{ "from-duration":"P30D" } }
But the task always tell me that 0 applications have been stopped despite that i see at least one app eligible to be stopped :
"AOS","AJSpace","ca3f26b0-2c64-4b0b-b346-0a24e3decf0b","springmusic-test-appmetrics","java","v4.36",,"cflinuxfs3","1","1","0.3059712","0.16539648","springtmusic-appmetrics.shared-ee-qvi.af-klm.com","2021-05-04T12:58:15","audit.app.start","62c1f89e-ab7b-4d87-86ce-e5f280f88b40","2021-09-02T12:02:31","started","Minor Release","2021-07-27T00:00","4.41","https://github.com/cloudfoundry/java-buildpack/releases/tag/v4.41"
Any idea what i did wrong ?
I looked at the code , but difficult to see where the test to the "Last Pushed Date" date is made as a filter .
Thanks again if you can help ,
Eric
Hello @pacphi , we 've found an issue with the buildpack version shown in the detail/Ai report .
This version is the version of the current buildpack provided by the platform .
But the real buildpack version used by the application is inside the current droplet object and can be different if the application has not been restaged since a Platform Buildpack upgrade .
There's a cf plugin (cf report-buildpack )(https://github.com/govau/cf-report-buildpacks/blob/master/cmd/report-buildpacks/report-buildpacks.go) which gives the right info .
We wanted to use this detail/AI report to identify outdated buildpacks usage by running applications , to enforce them to restage .
Thanks again if you can have a look , knowing that this is a big change in the way to gather informations from the application .
Best regards
Eric
The cf v3 API has support for assigning metadata (e.g., labels and annotations) to apps and (possibly in the future) service instances.
An interesting use case from a policy enforcement perspective is to have developers be required to label apps in conjunction with cf push. The cf CLI nor the cf-java-client currently have support so we'd have to build out something perhaps with a Spring WebFlux.
If an operator could apply and search for labels on apps (not unlike applying and filtering on tagged compute resources in AWS, Azure, or GCP), then there is another lever to: a) manage finite compute capacity (what cf-butler is arguably good at), b) establish compliance. Case b) is super-interesting because it can take many forms (regulation, cost control).
So, what we want to do here is offer yet another way to define a policy. If an app does not have 1-n labels defined then go ahead and stop and delete the app (and optionally any bound services).
Cloud Foundry for Kubernetes? Yup, see cf-for-k8s.
If we attempted to deploy cf-butler, we'd...
The above is by no means an exhaustive list of concerns.
What I'd like to do for MVP purposes is...
Aspiring in a second pass to...
With the above in mind, I might want to entertain re-organizing the structure of this repo to support e.g., common, cf-bosh and cf-for-k8s modules, resulting Impact:
Hi @pacphi,
Upon trying to integrate cf-butler using git set as the Policy Provider, it appears that there is not a way to pass Git credentials when deploying cf-butler to Pivotal Application Service.
I tried URL encoding my Git username/password and passing the credentials via the Git URI, but it fails with the following error:
2020-03-02T18:34:34.078-05:00 [APP/PROC/WEB/0] [OUT] [30m2020-03-02T23:34:34,073[m [33mWARN [m[[1;34mmain[m] [1;33mi.p.c.c.GitClient[m: Cannot clone Git repository at https://<username>:<password>@gituri.com/repo/repo-policies.git
2020-03-02T18:34:34.078-05:00 [APP/PROC/WEB/0] [OUT] org.eclipse.jgit.api.errors.TransportException: https://<username>:<password>@@gituri.com/repo/repo-policies.git: Authentication is required but no CredentialsProvider has been registered
Unfortunately, we won't be able to make this repository public, as it is internal.
Would it be possible to add functionality to support passing credentials in the same manner as the Git URI? I suspect it'd look something like this:
{
"CF_POLICIES_PROVIDER": "git",
"CF_POLICIES_URI": "https://mygit.uri.com",
"CF_POLICIES_GIT_USERNAME": "username",
"CF_POLICIES_GIT_PASSWORD": "password",
"CF_POLICIES_FILE-PATHS": [
"policy1-AP.json",
"policy2-SP.json"
]
}
Let me know your thoughts
When i enable the ops manger endpoints usage and access them i see in the logs that its running into cert issue. I checked the https://github.com/pacphi/cf-butler/blob/master/src/main/java/io/pivotal/cfapp/config/ButlerCfEnvProcessor.java but seems like there is no mapping for skipping ssl validation for OM api calls.
The token will expire. Need to keep it alive.
See https://spring.io/blog/2019/03/20/spring-tips-reactive-mysql-support-with-jasync-sql-and-r2dbc.
Should just mean...
So there's this: https://community.pivotal.io/s/article/how-to-migrate-from-one-cloud-foundry-stack-to-another-stack.
It talks about the Stack Auditor plugin, but we're constrained to a single application per invocation of cf change-stack. As of the 0.0.4 release we even have support for zero-downtime deployment. But what if we want to update the stack for 100s to 1000s of applications? We need some automation.
Looking at the changer.go source it is making some calls that do not (yet) appear to be supported by cf-java-client (particularly the zdt-deploy call) << we might consider employing Spring WebFlux's WebClient.
Like issue #51, we'd want to define a sub-class of ApplicationPolicy, and maybe marshall/unmarshall to/from JSON something like
Change stack (proposed)
{
"application-policies": [
{
"operation": "change-stack",
"description": "Update stack applications restricted to whitelisted organizations",
"options": {
"stack-from": "cflinuxfs2,
"stack-to": "cflinuxfs3"
},
"organization-whitelist": [ "zoo-labs" ]
}
]
}
We'd want to validate both stack-from and stack-to are valid (i.e., installed) stacks. Other than that we're looking to replicate functionality in go-cli implementation.
I am using the CF admin account, but I see in the logs saying it can't get a token from UAA
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] �[30m2019-06-03T13:03:11,037�[m �[1;31mERROR �[m[�[1;34mreactor-tcp-epoll-1�[m] �[1;33mo.s.c.l.CompositeLog�[m: [08a2e87f] 500 Server Error for HTTP GET "/snapshot/spaces/users"
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] java.lang.IllegalArgumentException: Cannot decode value of type java.sql.Timestamp
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.r2dbc.postgresql.codec.DefaultCodecs.decode(DefaultCodecs.java:97)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.r2dbc.postgresql.PostgresqlRow.get(PostgresqlRow.java:88)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.pivotal.cfapp.repository.R2dbcTkRepository.fromRow(R2dbcTkRepository.java:45)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.pivotal.cfapp.repository.R2dbcTkRepository.lambda$findOne$0(R2dbcTkRepository.java:40)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.r2dbc.postgresql.PostgresqlResult.lambda$map$0(PostgresqlResult.java:71)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:100)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxZip$ZipCoordinator.drain(FluxZip.java:735)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxZip$ZipInner.onNext(FluxZip.java:894)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxRepeatPredicate$RepeatPredicateSubscriber.onNext(FluxRepeatPredicate.java:78)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoProcessor.onNext(MonoProcessor.java:389)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:171)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onComplete(FluxFilterFuseable.java:166)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxReplay$UnboundedReplayBuffer.replayNormal(FluxReplay.java:550)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxReplay$UnboundedReplayBuffer.replay(FluxReplay.java:653)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxReplay$ReplaySubscriber.onComplete(FluxReplay.java:1188)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxTake$TakeSubscriber.onComplete(FluxTake.java:147)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxTake$TakeSubscriber.onNext(FluxTake.java:127)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:422)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.EmitterProcessor.onNext(EmitterProcessor.java:264)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxHide$HideSubscriber.onNext(FluxHide.java:74)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:180)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxWindowPredicate$WindowFlux.drainRegular(FluxWindowPredicate.java:636)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxWindowPredicate$WindowFlux.drain(FluxWindowPredicate.java:714)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxWindowPredicate$WindowFlux.onNext(FluxWindowPredicate.java:756)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxWindowPredicate$WindowPredicateMain.onNext(FluxWindowPredicate.java:227)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:238)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxWindowPredicate$WindowPredicateMain.onNext(FluxWindowPredicate.java:248)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:113)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxHandle$HandleConditionalSubscriber.onNext(FluxHandle.java:320)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onNext(FluxPeekFuseable.java:826)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:275)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:849)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxGenerate$GenerateSubscription.next(FluxGenerate.java:169)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.r2dbc.postgresql.message.backend.BackendMessageDecoder.lambda$decode$1(BackendMessageDecoder.java:82)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxGenerate$GenerateSubscription.slowPath(FluxGenerate.java:262)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxGenerate$GenerateSubscription.request(FluxGenerate.java:204)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:1874)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:1748)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxGenerate.subscribe(FluxGenerate.java:83)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.Flux.subscribe(Flux.java:7799)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:442)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:244)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:206)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:322)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:335)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:91)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:432)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:333)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2019-06-03T09:03:11.038-04:00 [APP/PROC/WEB/0] [OUT] at java.base/java.lang.Thread.run(Unknown Source)
2019-06-03T09:03:11.043-04:00 [RTR/1] [OUT] cf-butler-responsible-genet.dev.dev.east.paas.geointservices.io - [2019-06-03T13:03:11.031+0000] "GET /snapshot/spaces/users HTTP/1.1" 500 0 180 "-" "PostmanRuntime/7.13.0" "10.104.96.7:28910" "10.104.98.222:61022" x_forwarded_for:"70.167.241.178, 10.104.96.7" x_forwarded_proto:"https" vcap_request_id:"d9b11113-fd21-4db4-57b2-188e2f25bcc5" response_time:0.011171557 app_id:"5719baea-86b4-4046-801c-5b26b51d954f" app_index:"0" nga:"-" pcf:"-" x_b3_traceid:"cd9bb8ec138d6598" x_b3_spanid:"cd9bb8ec138d6598" x_b3_parentspanid:"-"
2019-06-03T09:03:35.745-04:00 [RTR/2] [OUT] cf-butler-responsible-genet.dev.dev.east.paas.geointservices.io - [2019-06-03T13:03:35.739+0000] "OPTIONS /cloudfoundryapplication/health HTTP/1.1" 200 0 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" "10.104.96.7:1764" "10.104.98.222:61022" x_forwarded_for:"70.167.241.178, 10.104.96.7" x_forwarded_proto:"https" vcap_request_id:"90d6527c-cb81-4d59-4ccd-87c5a08ad1e9" response_time:0.005906697 app_id:"5719baea-86b4-4046-801c-5b26b51d954f" app_index:"0" nga:"-" pcf:"-" x_b3_traceid:"962bddeb40bf456a" x_b3_spanid:"962bddeb40bf456a" x_b3_parentspanid:"-"
2019-06-03T09:03:35.745-04:00 [RTR/0] [OUT] cf-butler-responsible-genet.dev.dev.east.paas.geointservices.io - [2019-06-03T13:03:35.740+0000] "OPTIONS /cloudfoundryapplication/info HTTP/1.1" 200 0 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" "10.104.96.7:61626" "10.104.98.222:61022" x_forwarded_for:"70.167.241.178, 10.104.96.7" x_forwarded_proto:"https" vcap_request_id:"41d39fdc-d155-40fa-5e92-7c45db09cac8" response_time:0.004362136 app_id:"5719baea-86b4-4046-801c-5b26b51d954f" app_index:"0" nga:"-" pcf:"-" x_b3_traceid:"782cda88b7439b1b" x_b3_spanid:"782cda88b7439b1b" x_b3_parentspanid:"-"
2019-06-03T09:03:35.747-04:00 [RTR/1] [OUT] cf-butler-responsible-genet.dev.dev.east.paas.geointservices.io - [2019-06-03T13:03:35.742+0000] "OPTIONS /cloudfoundryapplication/mappings HTTP/1.1" 200 0 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" "10.104.96.7:28030" "10.104.98.222:61022" x_forwarded_for:"70.167.241.178, 10.104.96.7" x_forwarded_proto:"https" vcap_request_id:"6c8669b4-fab6-4811-77b2-ecf0a76ca472" response_time:0.004907145 app_id:"5719baea-86b4-4046-801c-5b26b51d954f" app_index:"0" nga:"-" pcf:"-" x_b3_traceid:"1cc6ab469a11e3d8" x_b3_spanid:"1cc6ab469a11e3d8" x_b3_parentspanid:"-"
2019-06-03T09:03:35.792-04:00 [RTR/0] [OUT] cf-butler-responsible-genet.dev.dev.east.paas.geointservices.io - [2019-06-03T13:03:35.761+0000] "GET /cloudfoundryapplication/mappings HTTP/1.1" 200 0 50798 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" "10.104.96.7:61626" "10.104.98.222:61022" x_forwarded_for:"70.167.241.178, 10.104.96.7" x_forwarded_proto:"https" vcap_request_id:"6aabf72e-d2f2-4894-6241-143cca85a089" response_time:0.030318053 app_id:"5719baea-86b4-4046-801c-5b26b51d954f" app_index:"0" nga:"-" pcf:"-" x_b3_traceid:"e5a035caf7bc11f5" x_b3_spanid:"e5a035caf7bc11f5" x_b3_parentspanid:"-"
2019-06-03T09:03:35.796-04:00 [RTR/2] [OUT] cf-butler-responsible-genet.dev.dev.east.paas.geointservices.io - [2019-06-03T13:03:35.766+0000] "GET /cloudfoundryapplication/info HTTP/1.1" 200 0 84 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" "10.104.96.7:1764" "10.104.98.222:61022" x_forwarded_for:"70.167.241.178, 10.104.96.7" x_forwarded_proto:"https" vcap_request_id:"f7ca66ba-980e-4757-7ae2-7bdd02261fc4" response_time:0.030357352 app_id:"5719baea-86b4-4046-801c-5b26b51d954f" app_index:"0" nga:"-" pcf:"-" x_b3_traceid:"dc9b54afe6faf9e2" x_b3_spanid:"dc9b54afe6faf9e2" x_b3_parentspanid:"-"
2019-06-03T09:03:35.815-04:00 [RTR/1] [OUT] cf-butler-responsible-genet.dev.dev.east.paas.geointservices.io - [2019-06-03T13:03:35.772+0000] "GET /cloudfoundryapplication/health HTTP/1.1" 200 0 192 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" "10.104.96.7:28958" "10.104.98.222:61022" x_forwarded_for:"70.167.241.178, 10.104.96.7" x_forwarded_proto:"https" vcap_request_id:"1fc9e94a-b5a5-42f4-773f-f257c6d30861" response_time:0.043300277 app_id:"5719baea-86b4-4046-801c-5b26b51d954f" app_index:"0" nga:"-" pcf:"-" x_b3_traceid:"9c10dec6c3d7408d" x_b3_spanid:"9c10dec6c3d7408d" x_b3_parentspanid:"-"
and from the api it says
{
"timestamp": "2019-05-30T20:46:44.390+0000",
"path": "/snapshot/spaces/users",
"status": 500,
"error": "Internal Server Error",
"message": "Cannot decode value of type java.sql.Timestamp"
}
In order to better evaluate/consume cf-butler, consider including a few examples of policy payloads and a description of what they do.
As the moment, it's hard for me to tell what the total set of policy configuration options are available. Perhaps I am looking in the wrong place, though-- I assume it's intended that they're deduced from the application policy and serviceinstancepolicy classes.
But even still, I'm unsure of what things like from-duration vs. from-datetime entails. Is that the difference of "how old the instance has been around" vs. "is the instance older than "? These kinds of explanations on the policies themselves would be helpful. (Again, maybe I'm totally looking in the wrong place though!)
At some point could you add SSO based authentication to the endpoint so the information is not exposed to the outside world
Deploying the latest from master (f573774) to Cloud Foundry results in a startup issue.
I've looked into ButlerConfig.java, application.yml, and my own copy of secrets.json for anything amiss (e.g., spurious bad characters, tabs instead of spaces), but nothing stands out as problematic on first-glance.
Here's what I see when consulting cf logs cf-butler --recent
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT 2020-03-07T17:19:24,020 ERROR [main] o.s.b.SpringApplication: Application run failed
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT java.lang.IllegalStateException: Error processing condition on io.pivotal.cfapp.config.ButlerConfig.tokenProvider
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:60)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForBeanMethod(ConfigurationClassBeanDefinitionReader.java:184)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:144)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:120)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:331)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:236)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:275)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:95)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:706)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.refresh(ReactiveWebServerApplicationContext.java:66)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at io.pivotal.cfapp.AppInit.main(AppInit.java:18)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at java.base/java.lang.reflect.Method.invoke(Unknown Source)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'vcap.services.cf-butler-secrets.credentials.CF_TOKEN_PROVIDER' in value "${vcap.services.cf-butler-secrets.credentials.CF_TOKEN_PROVIDER}"
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.util.PropertyPlaceholderHelper.replacePlaceholders(PropertyPlaceholderHelper.java:124)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.core.env.AbstractPropertyResolver.resolveRequiredPlaceholders(AbstractPropertyResolver.java:210)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.core.env.PropertySourcesPropertyResolver.getProperty(PropertySourcesPropertyResolver.java:88)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.core.env.PropertySourcesPropertyResolver.getProperty(PropertySourcesPropertyResolver.java:62)
2020-03-07T09:19:24.02-0800 [APP/PROC/WEB/0] OUT at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:47)
2020-03-07T09:19:24.08-0800 [CELL/SSHD/0] OUT Exit status 0
The service instance count in the snapshot/summary provides different results at different times.
Example:
curl -s https://cf-butler-<domain.com>/snapshot/summary | jq
When running 1st time provides the below result.
"service-instance-counts": {
"by-organization": {
“ORG-1”: 9,
"ORG-2": 1,
"ORG-3": 10,
"ORG-4": 4,
"ORG-5": 5,
"ORG-6": 71,
"ORG-7": 12,
"ORG-8": 1,
"ORG-9": 19
},
When running second time, provides different values..
"service-instance-counts": {
"by-organization": {
"ORG-1": 2,
"ORG-2": 20,
"ORG-3": 10,
"ORG-4": 57,
"ORG-5": 5,
"ORG-6": 266,
"ORG-7": 8,
"ORG-8": 1,
"ORG-9": 58
}
Using the below configurations.
{
"CF_TOKEN-PROVIDER": "userpass",
"CF_API-HOST": "my-domain-name",
"CF_USERNAME": "admin",
"CF_PASSWORD": "<i-wont-share-the-password-;-)>",
"CF_ORGANIZATION-BLACK-LIST": [
"a",
"b",
"c",
"d",
"e",
"f"
],
"CF_ACCOUNT-REGEX": "^[_A-Za-z0-9-\+]+(\.[_A-Za-z0-9-]+)@[A-Za-z0-9-]+(\.[A-Za-z0-9]+)(\.[A-Za-z]{2,})$",
"CRON_COLLECTION": "* */10 * * * ",
"CRON_EXECUTION": "" */5 * * * *",
"EXPOSED_ACTUATOR_ENDPOINTS": "beans,env,info,health,metrics,scheduledtasks,loggers,mappings,prometheus"
}
Did the basic setup required not sure if I need to do anything else apart from the above to get this working..
After the app built successfully, it returns error on r2db related message.
Report on average # of app instances per Diego Cell.
We could consume the total-running-application-instances metric from the /snapshot/summary endpoint.
The other info we need is the counts from the Resource Config tile section of the number of Diego Cells for each of PAS, and PAS for Windows installed tiles. (Note some foundations may have isolation segments so multiples of each tile would need to be summed).
To arrive at the answer we'd just divide the app instances into the total # of diego cells.
For the latter need to do some research on how to obtain.
Although cf-butler currently honors a blacklist, its house-keeping responsibility is perhaps a bit too wide when one contemplates a foundation with dozens of organizations with 2-3 spaces each and an average of 5-10 app and service instances per space.
With that scenario in mind, it might be worthwhile to implement a whitelist. To maintain backward compatibility a policy without a whitelist will apply to all orgs except those in the blacklist. If a whitelist is defined, then only execute the policy for those orgs (again excluding overlapping orgs in the blacklist).
This filter logic would apply to application and service-instance policy execution.
Add the ability to define a policy that when executed scans for apps with one or more app instances that have been in an "idle" state (e.g., idle could mean some minimum threshold of CPU or RAM consumption or HTTP traffic) over a configurable duration (e.g., idle for 24 hours).
Large scale enterprise development is messy,... folks forget they’ve left sh*t running and this capability would offer another way for operators to get control of finite compute capacity. You might be thinking why not employ an auto scaler and policy? Thing is it’s not always consistently set up and bound to applications. And what if you truly want to scale to 0? In this case we're favoring availability of "thickly provisioned" compute.
Hi @pacphi,
I've been having issues getting the MySQL implementation to work with r2dbc implementation.
The first error I was seeing was:
2020-07-06T21:55:57.15-0400 [APP/PROC/WEB/0] OUT org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dbmsSettings' defined in file [/home/vcap/app/BOOT-INF/classes/io/pivotal/cfapp/config/DbmsSettings.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'connectionFactory' defined in class path resource [org/springframework/boot/autoconfigure/r2dbc/ConnectionFactoryConfigurations$Pool.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.r2dbc.pool.ConnectionPool]: Factory method 'connectionFactory' threw exception; nested exception is java.lang.IllegalArgumentException: URL xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx does not start with the r2dbc scheme
After banging my head for a few hours, I realized there is a mistake/typo in the Property Value for the R2DBC_USERNAME that's being set to the URL instead of the USERNAME:
When I realized this, I updated my config with the following and re-deployed Butler.:
addOrUpdatePropertyValue("spring.r2dbc.username", "R2DBC_USERNAME", cfCredentials, properties);
However, I still ran into the same error message stating that the URL was not in the correct r2dbc format:
2020-07-06T22:13:33.442-04:00 [APP/PROC/WEB/0] [OUT] Caused by: java.lang.IllegalArgumentException: URL rdbc:mysql://q-n3s3y1.q-g57257.bosh:3306/service_instance_db does not start with the r2dbc scheme
I specified the R2DBC properties just as you have shown in your example secrets file, but then I discovered that there is a Typo in your example files.
The change I made to fix the issue was I had to replace:
"R2DBC_URL": "rdbc:mysql://q-n3s3y1.q-g57257.bosh:3306/service_instance_db"
with:
"R2DBC_URL": "r2dbc:mysql://q-n3s3y1.q-g57257.bosh:3306/service_instance_db"
I hope this helps others! I can also make a PR of this with the corrected changes if you'd like.
Thanks!
So there's this https://github.com/maruTA-bis5/mattermost4j.
Maybe we can look at enhancing the existing query policy feature by offering an alternative configuration to target a Mattermost instance?
Cf-butler exposes quite a bit of data via /snapshot, /products, /accounting endpoints.
Wouldn't it be cool if we could take JSON emitted from these endpoints (even the variants where we may have to supply params or path variables) and convert to CSV?
I'm thinking we could piggy-back off of the existing Query allow the query to specify an endpoint call instead of sql. And we could use Jackson to map JSON to CSV, see https://www.baeldung.com/java-converting-json-to-csv. We'd have to enhance the QueryService to delegate to an alternate implementation.
One example of where this might have value would be where you'd want to ask for usage data by organization and create a query policy to ship the results (apps, tasks, and services) as email attachments to interested recipients.
We swallow all historical records by visiting /policies/report. Maybe add another endpoint to filter between two LocalDate (start and end).
When I try to push the app after building, i get a "No compatible attachment provider is available" error message. It looks like the app is trying to start with JDK and not JRE. In the newer buildpacks, the JDK has been removed. Is there any tricks available to run this on JRE instead of JDK?
2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT 2019-12-11T20:52:49,852 ERROR [main] o.s.b.SpringApplication: Application run failed 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT java.lang.IllegalStateException: No compatible attachment provider is available 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:389) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:374) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:342) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at reactor.tools.shaded.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:328) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at reactor.tools.agent.ReactorDebugAgent.init(ReactorDebugAgent.java:41) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.boot.reactor.DebugAgentEnvironmentPostProcessor.postProcessEnvironment(DebugAgentEnvironmentPostProcessor.java:49) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEnvironmentPreparedEvent(ConfigFileApplicationListener.java:188) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEvent(ConfigFileApplicationListener.java:176) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:76) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:53) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:345) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at io.pivotal.cfapp.AppInit.main(AppInit.java:17) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at java.base/java.lang.reflect.Method.invoke(Unknown Source) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) 2019-12-11T14:52:49.86-0600 [APP/PROC/WEB/0] OUT at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)
When collecting users across foundations we realized that capitalization was affecting uniqueness.
"[email protected]" and "[email protected]" are being counted as 2 users
Modifying cf-mgmt doesn't affect the users already created in UAA.
I am getting this error with a crunchy postgresql bound and unbound
2019-03-29T13:24:31.206-04:00 [APP/PROC/WEB/0] [OUT] 2019-03-29 17:24:31.206 INFO 22 --- [ main] io.pivotal.cfapp.AppInit : The following profiles are active: secrets,cloud
2019-03-29T13:24:31.797-04:00 [APP/PROC/WEB/0] [OUT] 2019-03-29 17:24:31.796 WARN 22 --- [ main] onfigReactiveWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanDefinitionStoreException: Failed to process import candidates for configuration class [io.pivotal.cfapp.AppInit]; nested exception is java.lang.IllegalStateException: Error processing condition on org.springframework.boot.autoconfigure.r2dbc.ConnectionFactoryAutoConfiguration$ConnectionFactoryConfiguration
2019-03-29T13:24:31.824-04:00 [APP/PROC/WEB/0] [OUT] 2019-03-29 17:24:31.824 ERROR 22 --- [ main] o.s.boot.SpringApplication : Application run failed
2019-03-29T13:24:31.824-04:00 [APP/PROC/WEB/0] [OUT] org.springframework.beans.factory.BeanDefinitionStoreException: Failed to process import candidates for configuration class [io.pivotal.cfapp.AppInit]; nested exception is java.lang.IllegalStateException: Error processing condition on org.springframework.boot.autoconfigure.r2dbc.ConnectionFactoryAutoConfiguration$ConnectionFactoryConfiguration
I get this error after cloning the repo and then running ./gradlew build
cf-butler]$ ./gradlew build
Task :compileJava FAILED
FAILURE: Build failed with an exception.
Could not resolve all files for configuration ':compileClasspath'.
Could not find org.springframework.boot.experimental:spring-boot-starter-data-r2dbc:0.1.0.BUILD-20190507.184709-3.
Required by:
project :
Could not find org.springframework.boot.experimental:spring-boot-autoconfigure-r2dbc:0.1.0.BUILD-20190507.184651-3.
Required by:
project :
Could not find org.springframework.data:spring-data-r2dbc:1.0.0.BUILD-20190508.084637-64.
Required by:
project :
Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
Get more help at https://help.gradle.org
BUILD FAILED in 3s
1 actionable task: 1 executed
As of 2019-07-05, Spring Boot 2.2 M4 and Spring Cloud Hoxton M1 are available. Early builds of cf-butler based on this combo may be beneficial in light of hack in place to pin the Spring Data dependencies. Until Spring Cloud Services releases client libraries compatible with the aforementioned combo this is really just an experiment. For now this issue is here to track and motivate an eventual upgrade.
Hello @pacphi , we are are very happy to use your applications , but found some issue on several endpoints .
One of them is /stemcell/associations which fails .
I look at the code , and i guess this is due to the om version check which misses some token .
Thanks in advance if you can have a look .
Best regards
Eric
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT Caused by: java.lang.IllegalStateException: Unable to create a ConnectionFactory for 'ConnectionFactoryOptions{options={database=, password=REDACTED, user=, host=, port=3306, driver=mysql}}'. Available drivers: [ h2, postgresql ]
when I build the app I used the ./gradlew -b build.w-mysql.gradle but it looks like the mysql driver is missing
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'connectionFactory' defined in class path resource [io/pivotal/cfapp/config/CloudConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.r2dbc.spi.ConnectionFactory]: Factory method 'connectionFactory' threw exception; nested exception is java.lang.IllegalStateException: Unable to create a ConnectionFactory for 'ConnectionFactoryOptions{options={database=my_848658d67c874, password=REDACTED, user=my_aab977c7ca364, host=cf-848658d6-7c87-4e33-9526-8c0d116a0cdb.c26jraliffm9.us-east-1.rds.amazonaws.com, port=3306, driver=mysql}}'. Available drivers: [ h2, postgresql ]
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:638)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:468)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1326)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1165)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1276)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1196)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:868)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:771)
2020-02-11T15:30:15.64-0500 [APP/PROC/WEB/0] OUT ... 27 more
Currently PoliciesService is a facade for ApplicationPolicy and ServiceInstancePolicy and only allows for findAll, deleteAll, and save (where save can contain one or more of either or both new ApplicationPolicy, ServiceInstancePolicy).
Let's allow users to:
ApplicationPolicy/(ies)ServiceInstancePolicy/(ies)Pretty straight-forward feature request here. Let's make it easy on Windows folks.
Background:
Add capability that lets an operator see what applications and/or service instances would be affected when a policy is executed. Endpoint would generate output like the historical /report endpoint. (Neither applications nor service instances would be harmed on such requests).
While we benefit from knowing the buildpack and stack of an application via the /snapshot/detail endpoint, we'd also like to know the version of the buildpack.
This shouldn't be all that hard to add. And then of course we'd want to surface the same downstream in cf-hoover and cf-hoover-ui.
Hello there!
My name is Ana. I noted that you use the mutation testing tool in the project.
I am a postdoctoral researcher at the University of Seville (Spain), and my colleagues and I are studying how mutation testing tools are used in practice. With this aim in mind, we have analysed over 3,500 public GitHub repositories using mutation testing tools, including yours! This work has recently been published in a journal paper available at https://link.springer.com/content/pdf/10.1007/s10664-022-10177-8.pdf.
To complete this study, we are asking for your help to understand better how mutation testing is used in practice, please! We would be extremely grateful if you could contribute to this study by answering a brief survey of 21 simple questions (no more than 6 minutes). This is the link to the questionnaire https://forms.gle/FvXNrimWAsJYC1zB9.
Drop me an e-mail if you have any questions or comments ([email protected]). Thank you very much in advance!!
Hello , we would like to use the OPsmanager endpoints to get overview of all the product version installed , but we are facing an issue to connect the Opsmanager with user/pwd .
This is due to the integration of a Saml Identity provider (https://docs.pivotal.io/ops-manager/2-10/opsguide/config-rbac.html#enable-saml) .
With such integration , only client/secret can be used to authenticate .
Is it planned to add such a feature to cf-butler (in addition to user/pwd) ?
Thanks in advance
Eric
Hi @pacphi,
I am getting the following errors on the bootBuildImage step when I try to compile your latest code from master:
> Task :bootBuildImage FAILED
Building image 'docker.io/library/cf-butler:0.1-SNAPSHOT'
> Pulling builder image 'docker.io/cloudfoundry/cnb:0.0.53-bionic' ........................................ ..........
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':bootBuildImage'.
> Docker API call to 'docker://localhost/v1.24/images/create?fromImage=docker.io%2Fcloudfoundry%2Fcnb%3A0.0. 53-bionic' failed with status code 500 "com.sun.jna.LastErrorException: [2] No such file or directory"
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log outpu t. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.1.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 5m 1s
18 actionable tasks: 18 executed
It looks like it's trying to pull this image, docker.io/cloudfoundry/cnb:0.0.53-bionic. I checked on Docker Hub and I don't see this image anywhere https://hub.docker.com/r/cloudfoundry/cnb/tags
Is there an alternative image that should be used?
Ok. We have cf-cli's cf scale. We also have the app auto-scale cli.
These are great tools for manually and automatically scaling a single application. But what if we want to set a policy org-wide or perhaps system-wide?
Seems like we could enhance the existing ApplicationPolicy with additional properties. Introduce the notion of an operation (perhaps w/ an enum like ApplicationOperation). So we already support delete,... let's add scale. Then we might take a look at creating sub-classes of ApplicationPolicy, because who knows what else we may want to define. Maybe we wind up w/ DeleteApplicationPolicy and ScaleApplicationPolicy?
Let's look at examples of how we'd marshall/unmarshall to/from JSON
Delete (current, adjusted)
{
"application-policies": [
{
"operation": "delete",
"description": "Remove stopped applications that are older than some duration from now and restricted to whitelisted organizations",
"state": "stopped",
"from-duration": "PT30S",
"delete-services": true,
"organization-whitelist": [ "zoo-labs" ]
}
]
}
Scale (proposed)
{
"application-policies": [
{
"operation": "scale",
"description": "Scale applications restricted to whitelisted organizations",
"state": "started",
"options": {
"instances-from": 1,
"instances-to": 2
},
"organization-whitelist": [ "zoo-labs" ]
}
]
}
In both policies above we added an operation property to discriminate between them.
For the proposed scale policy, we added an options section. (Acknowledging that scaling by other factors like disk and memory might be "a thing"). We'd want to add some validation on the options like: instances-from > 0 and instances-to > 0.
So what would happen above is that the policy would execute and look for applications where the # of instances running is 1 and then scale up to 2 instances. (We could scale down too). We're also being explicit about where we apply scaling because we first filter using a combo of state, organization-whitelist and instances-from.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.