spring-attic / spring-cloud-gcp Goto Github PK
View Code? Open in Web Editor NEWIntegration for Google Cloud Platform APIs with Spring
License: Apache License 2.0
Integration for Google Cloud Platform APIs with Spring
License: Apache License 2.0
Create inbound/outbound gateways for Storage. Inbound should be based on AbstractRemoteFileStreamingMessageSource
Currently, PubSubTemplate.send()
returns a String with PubsubMessage
's ID. We block the call to publish() and, ideally, we'd return a future instead.
We should implement a PubSub ApiFuture
adapter to Spring's ListenableFuture
, so the user doesn't have to use anything PubSub.
Right now dependency management is very chaotic. Depending on the google-cloud-java
is a problem for modules that really do not depend on all the libs.
Just to make the project build, right now all projects depend on google-cloud-java
which is an abuse of dependencies.
Header would have precedence over PubSubTemplate
's topic
field.
Some good examples are the Kafka and Kinesis adapters:
https://github.com/spring-projects/spring-integration-aws/blob/master/src/main/java/org/springframework/integration/aws/outbound/KinesisMessageHandler.java#L93
https://github.com/spring-projects/spring-integration-kafka/blob/master/src/main/java/org/springframework/integration/kafka/outbound/KafkaProducerMessageHandler.java#L147
More discussion here.
There are a few packages that aren't used currently and could cause user confusion.
org.springframework.cloud.gcp.pubsub.converters.* (except SimpleMessageConverter)
org.springframework.cloud.gcp.pubsub.converters.support
org.springframework.cloud.gcp.pubsub.*
We should remove them before launching anything to the public.
They cover areas that we already have issues for: (a)sync mode (#51) and topic/message conversion (#54). We should wait until those issues are worked on in order to provide those features. Until then, it's better to remove the classes so they don't confuse users.
@artembilan wdyt?
Still thinking what this will entail.
There have been suggestions to chain MessageConverters in PubSubTemplate, e.g., through the use of a CompositeMessageConverter.
Need to look more deeply into this, including how it's done in other parts of Spring.
Right now we have a code:
return GoogleCredentials.fromStream(
new FileInputStream(resourceLoader.getResource(
this.environment.getProperty(AUTH_LOCATION_KEY)).getFile()));
Where it doesn't not make sense to try to get file and wrap it into FileInputStream
when we can use Resource.getInputStream()
directly.
OTOH not all Resource
implementations provide getFile()
. I can use something like ByteArrayResource
to load credentials from my local DB, for example.
We've been using self hosted Jenkins in other neighbour projects and I think it's been good so far.
Based on discussions with the Cloud Pub/Sub team, I submitted an issue on googleapis to have the PubSub proto refactored to PubSub instead of Pubsub. That probably isn't going to happen anytime soon due to all the dependencies, given this code was out there for 2 years now. However we can stop the pattern propagating and follow the style guide for the correct camel case.
note: this would be essentially the reversion of #56
We're currently working with the Pub/Sub team to fix the metrics backend to allow us to track Pub/Sub API usage by Spring integrations.
There are other APIs where we should also track usage. Right now, we're using:
The Cloud SQL Admin API's SQLAdmin.Builder
provides an applicationName
parameter that we could populate. It's not clear that the Storage API provides any such method for usage tracking.
Starter project to deal with MessaConverters, ResourceManagement, MessagingTemplate, MessageContainers.
Making the bridge between spring-messaging and pubsub
setCredentialsProvider
is deprecated on google-cloud-java 0.20.1-alpha and there is a new way to provide credentials.
Right now we get a new Subscriber
in the PubSubInboundChannelAdapter.start()
each time we stop()/start()
the channel adapter.
I'm pretty fine to have it as a single instance during the whole PubSubInboundChannelAdapter
lifecycle.
Therefore Subscriber.defaultBuilder()
should be done from the onInit()
.
Of course, that should be done only if Subscriber
can be stopped and started again at run time.
The AbstractRemoteFileStreamingMessageSource
follows a polling mechanism to fetch changed data on the remote storage.
Google Storage supports notifications via PubSub. We should provide a event driven approach that can listen for messages on the specified topic and then fetch the changed files
Right now we set the callback for the user, PubSubInboundChannelAdapter.receiveMessage()
.
We should add in a functional interface in the constructor to allow users to specify a callback.
There were also talks of specifying a MessageListenerContainer (?), but I haven't looked into this.
It should probably also be renamed to something else, since JSON isn't the only format supported by Google's OAuth2 credentials.
Now that we have the project base and some code already in, we should look into missing unit tests and add them as soon as possible.
We should refactor the project to allow addition of the spring-integration-gcp module as well as the starters.
We will probably end with:
[Library modules]
[Integration module]
[Starters modules]
The SI module is currently a child of the parent spring-cloud pom. We need to look in a way to make it more consistent with other SI modules and probably let it live on it's own pom.
Here's some things to keep in mind:
There is an example on KafkaMessageProducerHandler.
I noticed this happens when terminating my toy app. I suspect the delays are taking place at resource cleanup, more specifically, terminating connections to Google Cloud Pub/Sub.
Just leaving it here for future reference. We would want to get to the bottom of it before any alpha release.
On a Spring Boot scenario, it would be nice to have things like the GCP project ID injected as a String
bean, as well as a potential GcpProject
or even GcpConfiguration
@ConfigurationProperties
object.
This would be useful even in non Spring Boot scenarios, like SI or even plain Spring Framework.
Much like we have today with RabbitMQ, it would be nice to extract all the admin stuff from pubsub into a separate class.
One of the main reasons, besides good separation of concerns, is that we could use it to create topics from the storage module when users decide to enable notification from a bucket.
Currently if I connect a PubsubInboundChannelAdapter to a non-existent subscription, it silently fails.
We could change this behaviour by changing this line:
protected void doStart() {
super.doStart();
this.subscriber = this.subscriberFactory.getSubscriber(this.subscriptionName, this::receiveMessage);
this.subscriber.startAsync(); // <---- this fails silently.
}
We could add an .awaitRunning()
to ensure that when the subscriber started that it is actually running.
Get the basics for our plugin to generate the website
Spring Messaging in general has the notion of a MessageListenerContainer, which allows configuration of different settings such as ACK mode, concurrent listeners.
We need to add same behavior here not only to be consistent with the Spring ecosystem, but to allow our users to control those parameters.
As usual, let's use the spring-kafka project (https://github.com/spring-projects/spring-kafka/tree/master/spring-kafka/src/main/java/org/springframework/kafka/listener) as source of inspiration, since it's a new take on a well stablished concept
Currently we have GcpHeaders
, GCPProperties
, GCPContextAutoConfiguration
. The latter two need to be renamed.
The following line in org.springframework.integration.gcp.outbound.PubsubMessageHandler should be a blocking call to be able to determine whether the returned listenableFuture failed or succeeded:
protected void handleMessageInternal(Message<?> message) throws Exception {
this.pubsubTemplate.send(this.topic, message); // this fails silently if e.g. a topic is not defined
}
This could be just a simple .get()
call, or we can even have a debug log which would print out the result in case of success.
Optionally we could even take into account timeout configuration?
In #50, some suggested polishing changes surfaced a bigger issue. We might want to either use the default ExecutorProvider
(with 4 threads by default) or, instead, allow the user to specify their own ExecutorProvider
implementation.
We should look into injecting the ExecutorProvider
into the PubSubTemplate
which, right now, isn't possible in SI, since PubSubMessageHandler
uses new PubSubTemplate()
instead of injecting it.
Another issue is that re-using the same ExecutorProvider
does not guarantee that only one Executor
is generated, thereby reducing the number of created threads, which was the motivation behind the reuse.
Right now, we're sending PubSub's AckReplyConsumer
back to the user so they can ack/nack the message.
We should provide the option to ack messages automatically, and use a sensible default.
Maybe Storage Resource Abstraction as well.
Example: PubsubTemplate
rather than PubSubTemplate
.
The idea is to provide a default header in our PubSubMessageTemplate that could be use to track utilization. Something on the same aspects as the Agent
header of HTTP.
Let's take this as a Spike to verify first the feasibility of this approach
Currently, the SI-generated headers, like ID and timestamp, are being preserved on message conversion (Spring to PubsubMessage).
We should remove those messages on conversion, as well as any GcpHeaders, while at the same time, providing some means of preserving those headers if the users wants.
More details here.
The basis for any interaction with google libs is it's credentials object being available. There are a handful of ways to bootstrap it.
This module should provide a boot starter that will bootstrap a bean of type Credential
and will use Conditinal logic to determine what type of Credential
it will create
The blob
property must be volatile
.
The resolve()
method should not be synchronized
from the first row.
We can use " Double-Checked Locking" pattern do not block all the threads if blob
is already initialized:
class Foo {
private volatile Helper helper = null;
public Helper getHelper() {
if (helper == null) {
synchronized(this) {
if (helper == null)
helper = new Helper();
}
}
return helper;
}
}
https://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html
This is more of a placeholder for whomever is going to work in this story. Just make sure we have a good implementation with reasonable defaults for users to springify their pubsub experience ;)
To put on our page, as per #48
Project ID may be obtained from the environment on GCP using GOOGLE_CLOUD_PROJECT
or the metadata server.
This may require introduction of GcpProjectIdProvider
.
The spring-cloud-gcp-starter-storage
already supports loading gs://...
resources using DefaultResourceLoader
, but direct injection using @Value
does not work.
@Value("gs://<bucket>/<object>")
private Resource gcsResource;
Currently, for some reason it tries to resolve the resource via ServletContext.
java.io.FileNotFoundException: Could not open ServletContext resource [/gs://<bucket>/<object>]
at org.springframework.web.context.support.ServletContextResource.getInputStream(ServletContextResource.java:141) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
at com.example.HelloController.readGcs(BootDemoApplication.java:164) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_73]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_73]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_73]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_73]
NestedRuntimeException
and make sure we are using it across the PubSubTemplate, and the Message channel AdaptersProvide a module to interact with Google Storage
The minimum requirement is to provide a Resource
resolver to spring, so users could inject resources using something like : gs://<bucket_name>/<object_name>
The spring-cloud-gcp-starter-pubsub 1.0.0.BUILD-SNAPSHOT artifact resolves differently in Maven and Gradle (works well in Gradle and breaks in Maven).
The correct version currently seems to be 0.7.0 of google-library-credentials:
<dependency>
<groupId>com.google.auth</groupId>
<artifactId>google-auth-library-credentials</artifactId>
<version>0.7.0</version>
</dependency>
The maven and gradle dependency resolvers resolve the tree in different order and when the maven one finds the 0.4.0 version of google-authlibrary-credentials (pulled in by io.grpc:grpc-auth:jar:1.4.0:compile). 0.4.0 causes java.lang.ClassNotFoundException: com.google.auth.ServiceAccountSigner
as mentioned here
See the partial dependency tree dumps below (left is maven right is gradle):
[INFO] +- org.springframework.cloud:spring-cloud-gcp-starter-pubsub:jar:1.0.0.BUILD-SNAPSHOT:compile +--- org.springframework.cloud:spring-cloud-gcp-starter-pubsub:1.0.0.BUILD-SNAPSHOT
[INFO] | +- org.springframework.cloud:spring-cloud-gcp-starter-core:jar:1.0.0.BUILD-SNAPSHOT:compile | +--- org.springframework.cloud:spring-cloud-gcp-starter-core:1.0.0.BUILD-SNAPSHOT
[INFO] | | \- org.springframework.cloud:spring-cloud-gcp-core:jar:1.0.0.BUILD-SNAPSHOT:compile | | +--- org.springframework.cloud:spring-cloud-gcp-core:1.0.0.BUILD-SNAPSHOT
[INFO] | +- org.springframework.cloud:spring-cloud-gcp-pubsub:jar:1.0.0.BUILD-SNAPSHOT:compile | | | +--- com.google.cloud:google-cloud-core:1.2.3
[INFO] | | \- org.springframework:spring-messaging:jar:4.3.8.RELEASE:compile | | | | +--- com.google.guava:guava:20.0
[INFO] | +- com.google.cloud:google-cloud-pubsub:jar:0.20.1-beta:compile | | | | +--- joda-time:joda-time:2.9.2 -> 2.9.9
[INFO] | | +- io.netty:netty-tcnative-boringssl-static:jar:2.0.3.Final:compile | | | | +--- org.json:json:20160810 -> 20140107
[INFO] | | +- com.google.cloud:google-cloud-core:jar:1.2.1:compile | | | | +--- com.google.api:api-common:1.1.0
[INFO] | | | +- com.google.guava:guava:jar:20.0:compile | | | | | +--- com.google.code.findbugs:jsr305:3.0.0
[INFO] | | | +- joda-time:joda-time:jar:2.9.9:compile | | | | | \--- com.google.guava:guava:19.0 -> 20.0
[INFO] | | | +- org.json:json:jar:20140107:compile | | | | +--- com.google.api:gax:1.4.2
[INFO] | | | +- com.google.api:api-common:jar:1.1.0:compile | | | | | +--- com.google.auto.value:auto-value:1.2
[INFO] | | | | \- com.google.code.findbugs:jsr305:jar:3.0.0:compile | | | | | +--- com.google.guava:guava:20.0
[INFO] | | | +- com.google.api:gax:jar:1.4.1:compile | | | | | +--- com.google.auth:google-auth-library-oauth2-http:0.7.0
[INFO] | | | | +- com.google.auto.value:auto-value:jar:1.2:compile | | | | | | +--- com.google.auth:google-auth-library-credentials:0.7.0
[INFO] | | | | \- org.threeten:threetenbp:jar:1.3.3:compile | | | | | | +--- com.google.http-client:google-http-client:1.19.0 -> 1.21.0
[INFO] | | | +- com.google.protobuf:protobuf-java-util:jar:3.3.0:compile | | | | | | | +--- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.0
[INFO] | | | | \- com.google.code.gson:gson:jar:2.8.0:compile | | | | | | | \--- org.apache.httpcomponents:httpclient:4.0.1 -> 4.5.3
[INFO] | | | +- com.google.api.grpc:proto-google-common-protos:jar:0.1.12:compile | | | | | | | +--- org.apache.httpcomponents:httpcore:4.4.6
[INFO] | | | \- com.google.api.grpc:proto-google-iam-v1:jar:0.1.12:compile | | | | | | | +--- commons-logging:commons-logging:1.2
[INFO] | | +- com.google.cloud:google-cloud-core-grpc:jar:1.2.1:compile | | | | | | | \--- commons-codec:commons-codec:1.9 -> 1.10
[INFO] | | | +- com.google.protobuf:protobuf-java:jar:3.3.0:compile | | | | | | +--- com.google.http-client:google-http-client-jackson2:1.19.0 -> 1.21.0
[INFO] | | | +- io.grpc:grpc-protobuf:jar:1.4.0:compile | | | | | | | +--- com.google.http-client:google-http-client:1.21.0 (*)
[INFO] | | | | \- io.grpc:grpc-protobuf-lite:jar:1.4.0:compile | | | | | | | \--- com.fasterxml.jackson.core:jackson-core:2.1.3 -> 2.8.8
[INFO] | | | +- io.grpc:grpc-context:jar:1.4.0:compile | | | | | | \--- com.google.guava:guava:19.0 -> 20.0
[INFO] | | | \- com.google.api:gax-grpc:jar:0.21.1:compile | | | | | +--- com.google.api:api-common:1.1.0 (*)
[INFO] | | +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.12:compile | | | | | +--- org.threeten:threetenbp:1.3.3
[INFO] | | +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.12:compile | | | | | \--- com.google.code.findbugs:jsr305:3.0.0
[INFO] | | +- io.grpc:grpc-netty:jar:1.4.0:compile | | | | +--- com.google.protobuf:protobuf-java-util:3.3.0 -> 3.3.1
[INFO] | | | +- io.grpc:grpc-core:jar:1.4.0:compile (version selected from constraint [1.4.0,1.4.0]) | | | | | +--- com.google.protobuf:protobuf-java:3.3.1
[INFO] | | | | +- com.google.errorprone:error_prone_annotations:jar:2.0.19:compile | | | | | +--- com.google.guava:guava:19.0 -> 20.0
[INFO] | | | | \- com.google.instrumentation:instrumentation-api:jar:0.4.2:compile | | | | | \--- com.google.code.gson:gson:2.7 -> 2.8.0
[INFO] | | | +- io.netty:netty-codec-http2:jar:4.1.11.Final:compile (version selected from constraint [4.1.11.Final,4.1.11.Final]) | | | | +--- com.google.api.grpc:proto-google-common-protos:0.1.13
[INFO] | | | | +- io.netty:netty-codec-http:jar:4.1.11.Final:compile | | | | | +--- com.google.protobuf:protobuf-java:3.3.0 -> 3.3.1
[INFO] | | | | | \- io.netty:netty-codec:jar:4.1.11.Final:compile | | | | | \--- com.google.api:api-common:1.1.0 (*)
[INFO] | | | | \- io.netty:netty-handler:jar:4.1.11.Final:compile | | | | \--- com.google.api.grpc:proto-google-iam-v1:0.1.13
[INFO] | | | | \- io.netty:netty-buffer:jar:4.1.11.Final:compile | | | | +--- com.google.protobuf:protobuf-java:3.3.0 -> 3.3.1
[INFO] | | | | \- io.netty:netty-common:jar:4.1.11.Final:compile | | | | +--- com.google.api:api-common:1.1.0 (*)
[INFO] | | | \- io.netty:netty-handler-proxy:jar:4.1.11.Final:compile | | | | \--- com.google.api.grpc:proto-google-common-protos:0.1.13 (*)
[INFO] | | | +- io.netty:netty-transport:jar:4.1.11.Final:compile | | | \--- org.slf4j:slf4j-api:1.7.25
[INFO] | | | | \- io.netty:netty-resolver:jar:4.1.11.Final:compile | | +--- com.google.cloud:google-cloud-pubsub:0.20.1-beta
[INFO] | | | \- io.netty:netty-codec-socks:jar:4.1.11.Final:compile | | | +--- io.netty:netty-tcnative-boringssl-static:2.0.3.Final
[INFO] | | +- io.grpc:grpc-stub:jar:1.4.0:compile | | | +--- com.google.cloud:google-cloud-core:1.2.1 -> 1.2.3 (*)
[INFO] | | \- io.grpc:grpc-auth:jar:1.4.0:compile | | | +--- com.google.cloud:google-cloud-core-grpc:1.2.1
[INFO] | | \- com.google.auth:google-auth-library-credentials:jar:0.4.0:compile | | | | +--- com.google.cloud:google-cloud-core:1.2.1 -> 1.2.3 (*)
[INFO] | +- com.google.cloud:google-cloud-storage:jar:1.2.1:compile | | | | +--- com.google.guava:guava:20.0
[INFO] | | +- com.google.cloud:google-cloud-core-http:jar:1.2.1:compile | | | | +--- com.google.protobuf:protobuf-java:3.3.0 -> 3.3.1
[INFO] | | | +- com.google.auth:google-auth-library-oauth2-http:jar:0.7.0:compile | | | | +--- com.google.protobuf:protobuf-java-util:3.3.0 -> 3.3.1 (*)
[INFO] | | | | \- com.google.http-client:google-http-client-jackson2:jar:1.19.0:compile | | | | +--- io.grpc:grpc-protobuf:1.4.0
[INFO] | | | +- com.google.http-client:google-http-client:jar:1.21.0:compile | | | | | +--- io.grpc:grpc-core:1.4.0
[INFO] | | | | \- org.apache.httpcomponents:httpclient:jar:4.5.3:compile | | | | | | +--- com.google.guava:guava:19.0 -> 20.0
[INFO] | | | | +- org.apache.httpcomponents:httpcore:jar:4.4.6:compile | | | | | | +--- com.google.errorprone:error_prone_annotations:2.0.19
[INFO] | | | | \- commons-codec:commons-codec:jar:1.10:compile | | | | | | +--- com.google.code.findbugs:jsr305:3.0.0
[INFO] | | | +- com.google.oauth-client:google-oauth-client:jar:1.21.0:compile | | | | | | +--- io.grpc:grpc-context:1.4.0
[INFO] | | | +- com.google.api-client:google-api-client:jar:1.21.0:compile | | | | | | \--- com.google.instrumentation:instrumentation-api:0.4.2
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.