streamsets/ansible-datacollector-sample-playbook

Sample Ansible playbook for use with StreamSets Data Collector

streamsets.datacollector

StreamSets Data Collector - An open source data collector.

Requirements

The role currently expects that you already have a JRE installed on the target system. Oracle Java 8 is recommended. This example playbook includes usage of a third-party role for installing Oracle Java 8.

Role Variables

Defaults are provided for the most commonly changed parameters in defaults/main.yml. For a full list of available variables please review the templates directly.

Dependencies

None.

Example Playbook

This playbook has a datacollectors.yml which has an example of how to install StreamSets Data Collector using the official streamsets.datacollector role, optionally use the sdc_config module to re-configure an existing instance, and then deploy and start a pipeline.

Using the streamsets.datacollector role for installation already includes configuration using templates and variables, so using the sdc_config module is not necessary and included for illustration purposes only. The sdc_config module is more useful when using other installation or deployment methods such as RPM packaging or Docker containers to customize configuration.

Basic usage only requires specifying the role. To run multiple instances on a single machine you can specify custom values for sdc_instance and http_port

- hosts: datacollectors
  roles:
     - { role: streamsets.datacollector, sdc_instance: 'datacollector_1', http_port: 18630 }
     - { role: streamsets.datacollector, sdc_instance: 'datacollector_2', http_port: 18640 }

To enable JMX with no SSL or authentication configure your vars or host_vars file with:

jmx_enable: true
jmx_port: <port number>
jmx_authenticate: false
jmx_ssl: false

Enabling JMX with authentication also requires specifying a username and password. For example:

jmx_authenticate: true
jmx_monitor_user: streamsetsMonitor
jmx_monitor_password: mysecretpassword

An entry in ${SDC_CONF}/sdc-security.policy will automatically be added to allow access to all MBeans.

Enabling SSL will use the $SDC_CONF/keystore.jks by default. To override this setting set the following parameters:

jmx_keystore: </absolute/path/to/keystore>
jmx_keystore_password: <password or $(cat /path/to/password_file.txt)>

License

Apache License, Version 2.0

Author Information

• Adam Kunicki | mailing list | Twitter

ansible-datacollector-sample-playbook's Issues

Posting data to SDC HTTP server url from javascript throwsCORS issue

Hi, I am using Streamset pipeline for streaming data from browsers. So for that, I created a pipeline with HTTP server origin to post data from browsers Javascript and tried to write in that URL using REST client, and it writes successfully. Response Headers are already set in the SDC.properties file.

http.access.control.allow.origin=*
http.access.control.allow.headers=origin, content-type, accept, authorization, x-requested-by, x-ss-user-auth-token, x-ss-rest-call
http.access.control.allow.methods=GET, POST, PUT, DELETE, OPTIONS, HEAD

But when I try to write some data from JavaScript using XMLHTTPRequest, it throws error for pre-flight request. Below is the JavaScript code:

  var http = new XMLHttpRequest();
  var value = '{ "prop1": "value 2", "prop2": "value 2" }';
  var url ='http://13.68.93.97:8100/'
  var async = true
 if ('withCredentials' in http) {
	http.open('OPTIONS', url, async);
} else if (typeof XDomainRequest != "undefined") {
	http = new XDomainRequest(); //for IE
	http.open('OPTIONS', url);
} else {
	http.open('OPTIONS', url, async);
}
    http.open('POST', url, true);
    http.setRequestHeader('Content-Type', 'application/json; charset=utf-8');
    http.setRequestHeader('X-SDC-APPLICATION-ID', 'testApp1');
    // Below headers are set in sdc.properties, so doesn't make any difference in javascript
http.setRequestHeader("Access-Control-Allow-Origin", "*");
http.setRequestHeader('Access-Control-Allow-Methods', "POST, OPTIONS, GET");
http.setRequestHeader('Access-Control-Allow-Credentials', "true");
http.setRequestHeader('Access-Control-Expose-Headers', "X-Region");
http.setRequestHeader("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept,Accept-language");
http.withCredentials = true;
http.onreadystatechange = function () {
    if (http.readyState == 4 && http.status == 200) {
        console.log(http.responseText);
    }
}
http.send(value); 

Error thrown from the above code execution :

XMLHttpRequest cannot load http://13.68.93.97:8100/. Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost' is therefore not allowed access.

Any help will be really helpful. Thanks..!!

If wrong port is specified, sdc_pipeline steps always returns OK

We're running SS on 18630, but the default example play used port 18640. When you use the sdc_pipeline module in this scenario, the steps always return OK.

Once I corrected this, the play ran as expected.

Not a major issue, but just wanted to cite it so a) it can be fixed and b) others don't spend cycles trying to figure out what's up :)

When you run the equivalent CLI command, it returns "Already connected" instead of an error stating that it can't connect on that port at all.

[vagrant@ss-receiver-01 ~]$ /opt/streamsets-datacollector/bin/streamsets cli --auth-type form --url http://somehost.example.com:18640 -u admin -p admin store import --name logstash-receiver --file /tmp/logstash-receiver.json
Already connected

When run with the correct port, it spits out the JSON that the module is expecting to parse:

[vagrant@ss-receiver-01 ~]$ /opt/streamsets-datacollector/bin/streamsets cli --auth-type form --url http:/someserver.example.com:18630 -u admin -p admin store import --name logstash-receiver --file /tmp/logstash-receiver.json
{
  "RemoteException" : {
    "message" : "com.streamsets.datacollector.store.PipelineStoreException: CONTAINER_0201 - Pipeline 'logstash-receiver' already exists",
    "errorCode" : "CONTAINER_0201",
    "localizedMessage" : "CONTAINER_0201 - Pipeline 'logstash-receiver' already exists",
    "exception" : "PipelineStoreException",
    "javaClassName" : "com.streamsets.datacollector.store.PipelineStoreException",
    "stackTrace" : "com.streamsets.datacollector.store.PipelineStoreException: CONTAINER_0201 - Pipeline 'logstash-receiver' already exists\n\tat com.streamsets.datacollector.store.impl.FilePipelineStoreTask.create(FilePipelineStoreTask.java:161)\n\tat com.streamsets.datacollector.store.impl.CachePipelineStoreTask.create(CachePipelineStoreTask.java:95)\n\tat com.streamsets.datacollector.restapi.PipelineStoreResource.createPipeline(PipelineStoreResource.java:185)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)\n\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:152)\n\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)\n\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)\n\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)\n\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)\n\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)\n\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)\n\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)\n\tat org.glassfish.jersey.internal.Errors.process(Errors.java:315)\n\tat org.glassfish.jersey.internal.Errors.process(Errors.java:297)\n\tat org.glassfish.jersey.internal.Errors.process(Errors.java:267)\n\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)\n\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)\n\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)\n\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)\n\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)\n\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)\n\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)\n\tat com.streamsets.datacollector.http.LocaleDetectorFilter.doFilter(LocaleDetectorFilter.java:43)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)\n\tat com.streamsets.pipeline.http.MDCFilter.doFilter(MDCFilter.java:51)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)\n\tat org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)\n\tat org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:542)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:332)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:499)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)\n\tat org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)\n\tat java.lang.Thread.run(Thread.java:745)\n"
  }
}