Giter Site home page Giter Site logo

spring-guides / gs-integration Goto Github PK

View Code? Open in Web Editor NEW
12.0 25.0 43.0 945 KB

Integrating Data :: Learn how to build an application that uses Spring Integration to fetch data, process it, and write it to a file.

Home Page: http://spring.io/guides/gs/integration/

License: Apache License 2.0

Java 100.00%
spring-integration

gs-integration's Introduction

This guide walks you through the process of using Spring Integration to create a simple application that retrieves data from an RSS Feed (Spring Blog), manipulates the data, and then writes it to a file. This guide uses traditional Spring Integration XML configuration. Other guides show how to use Java Configuration and DSL with and without Lambda expressions.

What You Will Build

You will create a flow with Spring Integration by using traditional XML configuration.

Starting with Spring Initializr

You can use this pre-initialized project and click Generate to download a ZIP file. This project is configured to fit the examples in this tutorial.

To manually initialize the project:

  1. Navigate to https://start.spring.io. This service pulls in all the dependencies you need for an application and does most of the setup for you.

  2. Choose either Gradle or Maven and the language you want to use. This guide assumes that you chose Java.

  3. Click Dependencies and select Spring Integration.

  4. Click Generate.

  5. Download the resulting ZIP file, which is an archive of a web application that is configured with your choices.

Note
If your IDE has the Spring Initializr integration, you can complete this process from your IDE.
Note
You can also fork the project from Github and open it in your IDE or other editor.

Add to the Build Files

For this example, you need to add two dependencies:

  • spring-integration-feed

  • spring-integration-file

The following listing shows the final pom.xml file:

link:complete/pom.xml[role=include]

The following listing shows the final build.gradle file:

link:complete/build.gradle[role=include]

Define an Integration Flow

For this guide’s sample application, you will define a Spring Integration flow that:

  • Reads blog posts from the RSS feed at spring.io.

  • Transforms them into an easily readable String consisting of the post title and the URL for the post.

  • Appends that String to the end of a file (/tmp/si/SpringBlog).

To define an integration flow, you can create a Spring XML configuration with a handful of elements from Spring Integration’s XML namespaces. Specifically, for the desired integration flow, you work with elements from these Spring Integration namespaces: core, feed, and file. (Getting the last two is why we had to modify the build files provided by the Spring Initializr.)

The following XML configuration file (from src/main/resources/integration/integration.xml) defines the integration flow:

link:complete/src/main/resources/integration/integration.xml[role=include]

Three integration elements are in play here:

  • <feed:inbound-channel-adapter>: An inbound adapter that retrieves the posts, one per poll. As configured here, it polls every five seconds. The posts are placed into a channel named news (corresponding to the adapter’s ID).

  • <int:transformer>: Transforms entries (com.rometools.rome.feed.synd.SyndEntry) in the news channel, extracting the entry’s title (payload.title) and link (payload.link) and concatenating them into a readable String (and adding a newline). The String is then sent to the output channel named file.

  • <file:outbound-channel-adapter>: An outbound channel adapter that writes content from its channel (named file) to a file. Specifically, as configured here, it appends anything in the file channel to a file at /tmp/si/SpringBlog.

The following image shows this simple flow:

A flow that reads RSS feed entries

Ignore the auto-startup attribute for now. We revisit that later when we discuss testing. For now, notice that it is, by default, true, which means the posts are fetched when the application starts. Also note the property placeholder in the filename-generator-expression. It means that the default is SpringBlog but can be overridden with a property.

Make the Application Executable

Although it is common to configure a Spring Integration flow within a larger application (perhaps even a web application), there is no reason that it cannot be defined in a simpler standalone application. That is what you will do next: Create a main class that kicks off the integration flow and that declares a handful of beans to support the integration flow. You will also build the application into a standalone executable JAR file. We use Spring Boot’s @SpringBootApplication annotation to create the application context. Since this guide uses the XML namespace for the integration flow, you must use the @ImportResource annotation to load it into the application context. The following listing (from src/main/java/com/example/integration/IntegrationApplication.java) shows the application file:

link:complete/src/main/java/com/example/integration/IntegrationApplication.java[role=include]

Run the application

Now you can run the application from the jar by running the following command:

java -jar build/libs/{project_id}-0.1.0.jar

... app starts up ...

Once the application starts, it connects to the RSS feed and starts fetching blog posts. The application processes those posts through the integration flow you defined, ultimately appending the post information to a file at /tmp/si/SpringBlog.

After the application has been running for awhile, you should be able to view the file at /tmp/si/SpringBlog to see the data from a handful of posts. On a UNIX-based operating system, you can also tail the file to see the results, as they are written, by running the following command:

tail -f /tmp/si/SpringBlog

You should see something like the following sample output (though the actual news will differ):

Spring Integration Java DSL 1.0 GA Released @ https://spring.io/blog/2014/11/24/spring-integration-java-dsl-1-0-ga-released
This Week in Spring - November 25th, 2014 @ https://spring.io/blog/2014/11/25/this-week-in-spring-november-25th-2014
Spring Integration Java DSL: Line by line tutorial @ https://spring.io/blog/2014/11/25/spring-integration-java-dsl-line-by-line-tutorial
Spring for Apache Hadoop 2.1.0.M2 Released @ https://spring.io/blog/2014/11/14/spring-for-apache-hadoop-2-1-0-m2-released

Testing

Examine the complete project and you will see a test case, in src/test/java/com/example/integration/FlowTests.java:

link:complete/src/test/java/com/example/integration/FlowTests.java[role=include]

This test uses Spring Boot’s test support to set a property named auto.startup to false. It is generally not a good idea to rely on a network connection for tests, especially in a CI environment. Instead, we prevent the feed adapter from starting and inject a SyndEntry into the news channel for processing by the rest of the flow. The test also sets the feed.file.name so that the test writes to a different file. Then it:

  • Verifies that the adapter is stopped.

  • Creates a test SyndEntry.

  • Deletes the test output file (if it is present).

  • Sends the message.

  • Verifies that the file exists.

  • Reads the file and verifies that the data is as expected.

Summary

Congratulations! You have developed a simple application that uses Spring Integration to fetch blog posts from spring.io, process them, and write them to a file.

gs-integration's People

Contributors

andirdju avatar artembilan avatar bclozel avatar btalbott avatar buzzardo avatar cbeams avatar garyrussell avatar gregturn avatar habuma avatar nhumblot avatar robertmcnees avatar royclarkson avatar sdeleuze avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gs-integration's Issues

SAXParseException when running the project with Spring Boot 1.4.0

When running the project from the commandline, I encounter a SAXParseException. If I downgrade Spring Boot from 1.4.0 to 1.3.7, the problem goes away.

Steps to reproduce:
git clone https://github.com/spring-guides/gs-integration.git
cd gs-integration/complete
gradle wrapper
./gradlew build
java -jar build/libs/gs-integration-0.1.0.jar

Encounter Exception:

org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 25 in XML document from class path resource [hello/integration.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 25; columnNumber: 59; cvc-complex-type.3.2.2: Attribute 'mode' is not allowed to appear in element 'file:outbound-channel-adapter'.
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:399) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.boot.BeanDefinitionLoader.load(BeanDefinitionLoader.java:180) ~[spring-boot-1.4.0.RELEASE.jar!/:1.4.0.RELEASE]
    at org.springframework.boot.BeanDefinitionLoader.load(BeanDefinitionLoader.java:207) ~[spring-boot-1.4.0.RELEASE.jar!/:1.4.0.RELEASE]
    at org.springframework.boot.BeanDefinitionLoader.load(BeanDefinitionLoader.java:144) ~[spring-boot-1.4.0.RELEASE.jar!/:1.4.0.RELEASE]
    at org.springframework.boot.BeanDefinitionLoader.load(BeanDefinitionLoader.java:127) ~[spring-boot-1.4.0.RELEASE.jar!/:1.4.0.RELEASE]
    at org.springframework.boot.SpringApplication.load(SpringApplication.java:701) ~[spring-boot-1.4.0.RELEASE.jar!/:1.4.0.RELEASE]
    at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:364) ~[spring-boot-1.4.0.RELEASE.jar!/:1.4.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:311) ~[spring-boot-1.4.0.RELEASE.jar!/:1.4.0.RELEASE]
    at hello.Application.main(Application.java:8) [classes!/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60]
    at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60]
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [gs-integration-0.1.0.jar:na]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [gs-integration-0.1.0.jar:na]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [gs-integration-0.1.0.jar:na]
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:58) [gs-integration-0.1.0.jar:na]
Caused by: org.xml.sax.SAXParseException: cvc-complex-type.3.2.2: Attribute 'mode' is not allowed to appear in element 'file:outbound-channel-adapter'.
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.error(ErrorHandlerWrapper.java:134) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:437) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:325) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator$XSIErrorReporter.reportError(XMLSchemaValidator.java:458) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.reportSchemaError(XMLSchemaValidator.java:3237) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.processAttributes(XMLSchemaValidator.java:2714) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.handleStartElement(XMLSchemaValidator.java:2056) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.emptyElement(XMLSchemaValidator.java:766) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:356) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2786) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243) ~[na:1.8.0_60]
    at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:348) ~[na:1.8.0_60]
    at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:429) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:391) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE]
    ... 18 common frames omitted

Update Spring Boot to the latest version

Update the guide to use the most recent Spring Boot version.

Files that require changes are:

initial/build.gradle
initial/pom.xml
complete/build.gradle
complete/pom.xml

wrong dependency

if using : compile("org.springframework.integration:spring-integration-twitter:3.0.0.RELEASE")
then we have error :
java.lang.ClassNotFoundException: org.springframework.integration.MessagingException

fixed by upgrading the version to this :
compile("org.springframework.integration:spring-integration-twitter:4.0.2.RELEASE")

Guide asks to create wrong directory

On the main page(https://spring.io/guides/gs/integration/), the guide asks users to create the standard src/main/java/hello bunch of directories, but the application actually has the directory structure src/main/java/blog/. This isn't my first guide, so not a problem for me. But it could be a problem for someone not familiar with things.

Quote:

Create the directory structure

In a project directory of your choosing, create the following subdirectory structure; for example, with mkdir -p src/main/java/hello on *nix systems:

└── src
    └── main
        └── java
            └── hello

System.in does not pause

I found when following this guide that System.in.read() in Application.java did not pause as expected when using gradlew bootRun. (It did work with the standalone jar.) Apparently, this is due to a default override in Gradle.

https://discuss.gradle.org/t/why-doesnt-system-in-read-block-when-im-using-gradle/3308
http://stackoverflow.com/questions/13172137/console-application-with-java-and-gradle

The workaround is to add the following to the build script.

bootRun {
  standardInput = System.in
}

Little issues like this are a source of friction when getting started, so it would be good to either update the build script or add a footnote about it.

Test

@Autowired
private SourcePollingChannelAdapter newsAdapter;
idea prompt error:
Could not autowire. No beans of 'SourcePollingChannelAdapter' type found.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.