spring-cloud/spring-cloud-deployer-cloudfoundry

The Spring Cloud Deployer implementation for Cloud Foundry

Spring Cloud Connectors

Spring Cloud Connectors provides a simple abstraction that JVM-based applications can use to discover information about the cloud environment on which they are running, connect to services, and have discovered services registered as Spring beans. It provides out-of-the-box support for discovering common services on Heroku and Cloud Foundry cloud platforms, and it supports custom service definitions through Java Service Provider Interfaces (SPI).

Note	This project is in maintenance mode, in favor of the newer Java CFEnv project. We will continue to release security-related updates but will not address enhancement requests.

Learn more

See the project site and current documentation.

Build

The project is built with Gradle. The Gradle wrapper allows you to build the project on multiple platforms and even if you do not have Gradle installed; run it in place of the gradle command (as ./gradlew) from the root of the main project directory.

To compile the project and run tests

./gradlew build

To build a JAR

./gradlew jar

To generate Javadoc API documentation

./gradlew api

To list all available tasks

./gradlew tasks

Contributing

Spring Cloud is released under the non-restrictive Apache 2.0 license, and follows a very standard Github development process, using Github tracker for issues and merging pull requests into master. If you want to contribute even something trivial please do not hesitate, but follow the guidelines below.

Sign the Contributor License Agreement

Before we accept a non-trivial patch or pull request we will need you to sign the Contributor License Agreement. Signing the contributor’s agreement does not grant anyone commit rights to the main repository, but it does mean that we can accept your contributions, and you will get an author credit if we do. Active contributors might be asked to join the core team, and given the ability to merge pull requests.

Code of Conduct

This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to [email protected].

Code Conventions and Housekeeping

None of these is essential for a pull request, but they will all help. They can also be added after the original pull request but before a merge.

• Use the Spring Framework code format conventions. If you use Eclipse you can import formatter settings using the eclipse-code-formatter.xml file from the Spring Cloud Build project. If using IntelliJ, you can use the Eclipse Code Formatter Plugin to import the same file.

• Make sure all new .java files to have a simple Javadoc class comment with at least an @author tag identifying you, and preferably at least a paragraph on what the class is for.

• Add the ASF license header comment to all new .java files (copy from existing files in the project)

• Add yourself as an @author to the .java files that you modify substantially (more than cosmetic changes).

• Add some Javadocs and, if you change the namespace, some XSD doc elements.

• A few unit tests would help a lot as well — someone has to do it.

• If no-one else is using your branch, please rebase it against the current master (or other target branch in the main project).

• When writing a commit message please follow these conventions, if you are fixing an existing issue please add Fixes gh-XXXX at the end of the commit message (where XXXX is the issue number).

spring-cloud-deployer-cloudfoundry's Issues

Implement attributes retrieval

In the status() call, attributes population has not been implemented.

Add a new deployment property for Task specific service bindings

As a developer, I'd like to break the SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_SERVICES property into two parts, so I can pass stream and task specific service bindings separately.

Acceptance:

• SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_SERVICES is split into SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_SERVICES and SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_TASK_SERVICES
• Passing services using the respective property only applies to either stream or tasks and not both at any given time (for example: rabbit will be usually only used for stream apps and mysql for task apps)
• Unit tests adapted to new property

Fix typo of "Sucessfully"

Undeploy has a typo

Update to latest java-client buildpack

Update cf-java-client and reactor versions

As a developer, I'd like to upgrade to the latest stable releases of cf-java-client and reactor versions.

Upgrade reactor and cf-java-client release versions

Rename repository to spring-cloud-deployer-cloudfoundry.

Also update the configuration property prefix in CloudFoundryDeployerProperties from

@ConfigurationProperties(prefix = "spring.cloud.cloudfoundry.deployer")
to
@ConfigurationProperties(prefix = "spring.cloud.deplyer.cloudfoundry")

Getting started docs that reference these properties needs to be updated.

Group should be optional

Or maybe not even explicit (should be the concern of the caller).

Remove existing TaskLauncher APIs

Adapt to TCK wrt count, instanceIndex, and deploymentProperties

Apply CloudFoundryDeploymentProperties to tasks

Currently, the only bit of information carried by CloudFoundryDeploymentProperties that applies to task deployments is bound services. There is an opportunity to also apply:

• memory
• disk
• buildpack
• healthcheck
• instances?

Research and apply as necessary, and maybe split that task into a common superclass that only contains what is applicable to both streams and tasks

Upgrade to M10 release of cf-java-client

As a developer, I'd like to upgrade to M10 release of cf-java-client, so I can redeploy apps as many times as I want. See this issue for background.

Adapt to new Tasks TCK tests

As a user, I want to define custom routes

It uses the default route (application name). It would be nice to offer an override setting to supply a list of routes.

Allow deployed apps to specify a command

This would be useful for buildpacks that require one (such as python).

Should pass the full apps TCK

In particular, state should be reported as failed when appropriate

Deployer is not sharing the database

As a user, I'm using the deployer in SCDF-CF-Server with a commonly bound mysql service with the hope to reuse the database for stream/task metadata and task/job executions. With this setup, I'm unable to reuse the same datasource. The apps launched by the deployer are bound to a complete different database instance.

As a user, I want the option to deploy without using a container health check

This is the equivalent of "cf push -u none".

It's critical if you enable DEBUG with the buildpack, and need the app to suspend itself on startup, but still let you create the SSH tunnel to attach a remote debugger.

See https://github.com/cloudfoundry/java-buildpack/blob/master/docs/framework-debug.md for more details.

Adapt to new Tasks TCK tests

Passing in server.port for http source results in app in 'crashed' state on PCF

Bad behavior when deploying http source to PCF and specifying a port.

Improve mapping for apps that are effectively in 'crashed' state

The default retry attempts for

case "CRASHED":
  return DeploymentState.failed;

is very long, certainly more than 3 hrs and heard it maybe weeks, so it is really never reached in practice for a dataflow deployment. We should introduce our own logic to keep track of how many deployment attempts were made and then decide to fail the deployment state after our own threshold is reached. This is done in the kubernetes deployer.

Rename to spring-cloud-deployer-cloudfoundry

Rename repository, artifactId, references to such..

Launching task throws java.lang.IndexOutOfBoundsException

When launching task on CF, below exception is thrown:

It looks like ListSpaceResponse has couple of SpaceResources when MonoSingle subscriber expects one item.
When debugging I see the following response:

ListSpacesResponse{nextUrl=null, previousUrl=null, resources=[SpaceResource{entity=SpaceEntity{allowSsh=true, applicationEventsUrl=/v2/spaces/f7005c0f-d261-49a0-b446-3c9484eb4d43/app_events, applicationsUrl=/v2/spaces/f7005c0f-d261-49a0-b446-3c9484eb4d43/apps, auditorsUrl=/v2/spaces/f7005c0f-d261-49a0-b446-3c9484eb4d43/auditors, developersUrl=/v2/spaces/f7005c0f-d261-49a0-b446-3c9484eb4d43/developers, domainsUrl=/v2/spaces/f7005c0f-d261-49a0-b446-3c9484eb4d43/domains, eventsUrl=/v2/spaces/f7005c0f-d261-49a0-b446-3c9484eb4d43/events, managersUrl=/v2/spaces/f7005c0f-d261-49a0-b446-3c9484eb4d43/managers, name=development, organizationId=3f308b42-a407-4aea-bf77-d064149339ad, organizationUrl=/v2/organizations/3f308b42-a407-4aea-bf77-d064149339ad, routesUrl=/v2/spaces/f7005c0f-d261-49a0-b446-3c9484eb4d43/routes, securityGroupsUrl=/v2/spaces/f7005c0f-d261-49a0-b446-3c9484eb4d43/security_groups, serviceInstancesUrl=/v2/spaces/f7005c0f-d261-49a0-b446-3c9484eb4d43/service_instances, spaceQuotaDefinitionId=null}, metadata=Metadata{createdAt=2014-08-28T17:17:26Z, id=f7005c0f-d261-49a0-b446-3c9484eb4d43, updatedAt=null, url=/v2/spaces/f7005c0f-d261-49a0-b446-3c9484eb4d43}}, SpaceResource{entity=SpaceEntity{allowSsh=true, applicationEventsUrl=/v2/spaces/37fea686-46f1-40f6-b556-7bc261378669/app_events, applicationsUrl=/v2/spaces/37fea686-46f1-40f6-b556-7bc261378669/apps, auditorsUrl=/v2/spaces/37fea686-46f1-40f6-b556-7bc261378669/auditors, developersUrl=/v2/spaces/37fea686-46f1-40f6-b556-7bc261378669/developers, domainsUrl=/v2/spaces/37fea686-46f1-40f6-b556-7bc261378669/domains, eventsUrl=/v2/spaces/37fea686-46f1-40f6-b556-7bc261378669/events, managersUrl=/v2/spaces/37fea686-46f1-40f6-b556-7bc261378669/managers, name=development, organizationId=b5e2992c-a7ba-4fc9-811b-b38b0a42b4a1, organizationUrl=/v2/organizations/b5e2992c-a7ba-4fc9-811b-b38b0a42b4a1, routesUrl=/v2/spaces/37fea686-46f1-40f6-b556-7bc261378669/routes, securityGroupsUrl=/v2/spaces/37fea686-46f1-40f6-b556-7bc261378669/security_groups, serviceInstancesUrl=/v2/spaces/37fea686-46f1-40f6-b556-7bc261378669/service_instances, spaceQuotaDefinitionId=null}, metadata=Metadata{createdAt=2016-08-08T21:13:12Z, id=37fea686-46f1-40f6-b556-7bc261378669, updatedAt=null, url=/v2/spaces/37fea686-46f1-40f6-b556-7bc261378669}}], totalPages=1, totalResults=2}

java.lang.IndexOutOfBoundsException: Source emitted more than one item
    at reactor.core.publisher.MonoSingle$SingleSubscriber.onNext(MonoSingle.java:130) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:636) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.drain(FluxFlatMap.java:511) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxFlatMap$FlatMapInner.onSubscribe(FluxFlatMap.java:994) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:95) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:59) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:360) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.MonoFlatMap$FlattenSubscriber$InnerSubscriber.onNext(MonoFlatMap.java:195) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.drainAsync(FluxFlattenIterable.java:362) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.drain(FluxFlattenIterable.java:595) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.onNext(FluxFlattenIterable.java:220) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxBuffer$BufferExactSubscriber.onComplete(FluxBuffer.java:182) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:177) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxConcatArray.subscribe(FluxConcatArray.java:96) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxBuffer.subscribe(FluxBuffer.java:75) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxFlattenIterable.subscribe(FluxFlattenIterable.java:103) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.MonoFlatMap$FlattenSubscriber.onNext(MonoFlatMap.java:158) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxResume$ResumeSubscriber.onNext(FluxResume.java:75) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:173) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:131) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.Operators$DeferredScalarSubscriber.complete(Operators.java:797) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.MonoThenApply$MonoThenApplyManager$SecondSubscriber.onNext(MonoThenApply.java:204) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:125) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxUsing$UsingFuseableSubscriber.onNext(FluxUsing.java:320) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:115) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onNext(FluxPeekFuseable.java:390) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.Operators$DeferredScalarSubscriber.complete(Operators.java:797) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.MonoReduce$ReduceSubscriber.onComplete(MonoReduce.java:152) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:157) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxSubscribeOn$SubscribeOnPipeline.onComplete(FluxSubscribeOn.java:137) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.ipc.netty.common.NettyChannelHandler$InboundSink.drain(NettyChannelHandler.java:696) ~[reactor-netty-0.5.0.M2.jar!/:na]
    at reactor.ipc.netty.common.NettyChannelHandler$InboundSink.request(NettyChannelHandler.java:772) ~[reactor-netty-0.5.0.M2.jar!/:na]
    at reactor.core.publisher.Operators$DeferredSubscription.set(Operators.java:598) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxSubscribeOn$SubscribeOnPipeline.onSubscribe(FluxSubscribeOn.java:117) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.ipc.netty.common.NettyChannelHandler.subscribe(NettyChannelHandler.java:543) ~[reactor-netty-0.5.0.M2.jar!/:na]
    at reactor.core.publisher.FluxSource.subscribe(FluxSource.java:59) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.publisher.FluxSubscribeOn$SourceSubscribeTask.run(FluxSubscribeOn.java:235) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at reactor.core.scheduler.ExecutorScheduler$ExecutorTrackedRunnable.run(ExecutorScheduler.java:134) ~[reactor-core-3.0.0.RC1.jar!/:na]
    at io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451) ~[netty-all-4.1.3.Final.jar!/:4.1.3.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418) ~[netty-all-4.1.3.Final.jar!/:4.1.3.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401) ~[netty-all-4.1.3.Final.jar!/:4.1.3.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877) ~[netty-all-4.1.3.Final.jar!/:4.1.3.Final]
    at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_20]

Upgrade to SCDe 1.0.3

As a developer, I'd like to upgrade to 1.0.3.RELEASE of Spring Cloud Deployer.

Add support for command-line arguments

As a developer, I want to be able to configure command line args as a separate set of properties to be passed to an app when it's deployed.

Similar to the Local SPI (via: spring-cloud/spring-cloud-dataflow#597), we would need this supported at the CF deployer SPI.

Please make CloudFoundryClient @ConditionalOnMissingBean

So users can override the defaults implementation.

Project does not compile in Eclipse

It could be a JDT bug, but it would be nice to either get that fixed or refactor to make the compiler happy.

maven remote repositories settings

Hey there. So, in working on trying to externalize configuration info in a properties file, and then use Spring Cloud Config Server to provide these environment settings at installation time, I'm got a question about some of the values I would normally put a certain way in the YML manifest.

First, in a YML manifest, I might define them this way:

JAVA_OPTS: -Dhttp.keepAlive=false
MAVEN_REMOTE_REPOSITORIES_SNAPSHOTS_URL: <nexus url>
MAVEN_REMOTE_REPOSITORIES_RELEASES_URL: <another nexus url>

So how would I put these into a properties file? This is my guess:

java.opts=-Dhttp.keepAlive=false
maven.remote.repositories.snapshots.url=<nexus url>
maven.remote.repositories.releases.url=<another nexus url>

Explore options to deploy python apps

As a developer, I'd like to explore the option to deploy non-Boot apps; specifically, python to begin with.

@ConfigurationProperties metadata not being published in jar

So users that consume this library don't get auto-complete tooling (for instance). We just need to add the config processor at compile time.

Revisit auto-generation of routes and the character limit

As a developer, I'd like to revisit the current algorithm used to auto-generate routes in the context of character limits.

Harmonise @ConfigurationProperties with spring-cloud-cloudfoundry

E.g. "organization"->"org", "api"->"url".

Implement AppDeployer using the v2 CF API

As a developer I would like to implement the AppDeployer SPI using the v2 CF library in an 'operator'/'higher level' API style.

Acceptance criteria are unit tests that has coverage similar to the current v1 API. A manual test of deploying time | log is also required.

Do not version .idea files

Enable Task-based unit testing

Task-based unit tests were disabled since the Task V3 APIs are still in flux. All the work invested in that test case is kept, but disabled since it doesn't yet work. When the API is stabilized, reenable that test suite and finish it.

Upgrade cf-java-client and reactor releases

As a developer, I'd like to update to latest cf-java-client and reactor releases.

Following are the tracked releases:

<reactor-core.version>3.0.2.RELEASE</reactor-core.version>
<reactor-netty.version>0.5.1.RELEASE</reactor-netty.version>

Move Spring related environment variables into SPRING_APPLICATION_JSON

Certain apps have very complex property settings which are confused by Cloud Foundry's handling of environment variables.

Include a flag that instructs the deployer to first serialize the environment variables into a JSON structure and then write it into SPRING_APPLICATION_JSON instead of separate variables.

Allow the request to override buildpack on deployment

Buildpack, mem, and disk are only set via fixed configuration properties. Overriding on a per request basis is critical to usage in a multithreaded system.

Implement TaskLauncher SPI

As a developer, I'd like to implement TaskLauncher SPI, so I can launch short-lived apps in Cloud Foundry.

CF Deployer should support commandLineArgs for apps

Add ability to override deploy timeouts

Surface cloudfoundry/cf-java-client@1bbf9d1 as configuration properties

Support special chars in cmdLineArgs

The current TCK test for cmdLineArgs is @Ignored because some chars need special handling and this has proven difficult in the past. Revisit

In CloudFoundryAppDeployer, replace CloudFoundryClient usage with -operations API call

Look for TODO comment.

Apps should be able to use a their own prefix for connection props

Allows users to set a prefix for the connection (deployment) properties

Rename environmentProperties to deploymentProperties

As a developer, I'd like to adapt to deploymentProperties changes inherited from the core deployer SPI [spring-cloud/spring-cloud-deployer#73].

Adapt to new deploymentProperties contract from s-c-deployer

Deployment properties should not be blindly passed to app

Allow MavenProperties to work in TCK tests

Provide a temp fix until spring-cloud/spring-cloud-deployer#102 is reolved.

Add the ability to delete v3 App

As a developer, I'd like to add support to delete a v3 app.

Upgrade to 2.0.0.RC1 release of cf-java-client

As a developer, I'd like to upgrade to 2.0.0.RC1 release of cf-java-client.

Removing defaults from CF specific property keys

The deployment property names such as spring.cloud.deployer.cloudfoundry.defaults.memory in CloudFoundryAppDeployer should have the word defaults to be consistent with other deployer implementations.