Giter Site home page Giter Site logo

quarks-edge / quarks Goto Github PK

View Code? Open in Web Editor NEW
122.0 122.0 42.0 20.91 MB

Quarks is an open source programming model and runtime for edge devices that enables you to analyze data and events at the device.

Home Page: http://quarks-edge.github.io/

License: Apache License 2.0

Java 79.89% HTML 13.48% JavaScript 5.87% CSS 0.29% Shell 0.43% Batchfile 0.05%

quarks's People

Contributors

ddebrunner avatar dlaboss avatar home4slc avatar kathyssaunders avatar mikespicer avatar susancline avatar vdogaru avatar vlasov01 avatar wmarshall484 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

quarks's Issues

initial fork/clone experience w/eclipse: top level artifacts, .gitignore

I forked quarks and used Eclipse/git to create my workspace and download a clone of my fork and import all of the projects. The top level repo artifacts like build.xml, etc don't show up in the project explorer.
Following an "ant all" there are now many untracked files reported because "/target" isn't in the .gitignore. Ditto for /test/svt/lib.

fix eclipse "output dir" config to be independent of ant build

There are annoying interactions between the ant build tooling and eclipse (building/cleaning) resulting from an overlap in some locations of their javac output directories. Going to change things so they're independent of each other.

Leaving the ant stuff as is: class files under "classes" directories - one or more per Java-Project.

Going to change the eclipse build configs to start using the default eclipse project convention for the output dir: /bin.

detecting zero tuples flowing through a stream for the purpose of tagging

I'm writing a sample where I know a stream (Stream) has no tuples, however, it's a rigged example, and I'm trying to demonstrate that there is a way to test for no tuples flowing.

If no tuples are flowing, then I want to tag it in a different way then if tuples are flowing.

I talked with @vdogaru and we got it to the point where if I put a counter on the stream that I suspect has no tuples and then after submitting the topology go through all the counters and look at their values I can see which oplets (counters) have no tuples.

But I can't figure out a way programmatically to test if a stream has no tuples, and if that is the case tag it. Anyone have any thoughts on this?

If not, I will log it as an enhancement. Thanks.

Add Range analytic utility?

A common sensor analytic is to filter for values being within a range. It's common for range values to be initialized from static properties files and/or dynamically set from external sources.

The Guava Range class offers a very convenient way for dealing with ranges. The Guava jar is significant in size. There are ways to mitigate that using ProGuard though as noted at that link it's not clear that approach would be appropriate for quarks/library use.

The SensorAnalytics sample app contains a simple Range implementation of a portion of the Guava Range API. Should that be incorporated into quarks.analytics.sensors to "make the simple simple"?

Enable Travis CI

I've added a .travis.yml to try and see if Travis CI can be used.

Had to switch to the trusty beta as the default setup used Ant 1.8, while Quarks uses some 1.9 features.

Still a work in progress.

Add jQuery and jQuery-ui javascript libraries to the console?

I'd like to start adding a few widgets to the console (dialogs, etc) and also to come up with a consistent look and feel for the console.
If we add jQuery and jQuery-ui js libraries, I believe it would be a good base to do so.
Both libraries have either the MIT license or an MIT license with a few added clauses:

JQuery: https://github.com/jquery/jquery/blob/master/LICENSE.txt
JqueryUI: http://contribute.jquery.org/CLA/

Any thoughts on either the choice of libraries, or the licenses?

Discussion Question: Why use Quarks

Recently one of my colleagues attended Spark Summit East and spoke with people about Quarks. One of the questions he was asked was "Why wouldn't I just write Quarks myself?" His answer was:
You always could, but Quarks removes the developer from having to write a lot of boiler-place code for things like:
- Keeping track of the last ten minutes of data, or last N items on a data stream.
- Connecting to Kafka, MQTT, JDBC, files, and websockets.
I think that's a good answer, but wanted to ask if others have more input on this. I can add it to our FAQs, but thought it was worth having a larger discussion on this topic.

Add join support

Quarks windowing still requires the following support:

// Join a TStream against a window with a certain key function
TStream join(Function<T,K> keyer, TWindow<U,K> window, BiFunction<T, List, J> joiner);

// Join the last tuple of this stream against the last tuple of another stream with partitioning
TStream joinLast(Function<? super T, ? extends K> keyer, TStream lastStream, Function<? super U, extends K> lastStreamKeyer, BiFunction<T, U, J> joiner);

// Create a new window with the same configuration as another window
TWindow<T,Object> window(TWindow configWindow);

Copyright statement in Java samples

Java samples have a header, which states the following:

/*
# Licensed Materials - Property of IBM
# Copyright IBM Corp. 2015, 2016 
*/

I think it doesn't match the project license.

Aliases for quarks added control service MBeans?

With JobMXBean it's fairly obvious that its alias for control should be the job name ( #60 ). This would allow a remote entity to start and stop the job by its name.

With an oplet specific control interface like PeriodicMXBean it's not so obvious.

PeriodicMXBean is automatically created for a periodic source, so if that periodic source represents a sensor reading then a remote system might want to change the polling frequency. E.g. when everything is normal a one minute poll is sufficient, but during anomalies it reads the sensor every 100ms by changing the polling rate. So how to get the alias for the control object so that it can be addressed from a remote system.

With a polling like

topology.poll(sensorFunction, 1, MINUTES).tag("engineTemp")

It's tempting to say the alias would be engineTemp, but what happens if multiple tags are set, or a default tag by Quarks ( #41)?

Jetty use with sample app DevelopmentSample?

After dusting off literally decades old java experience I was able to gat a clean build of all the quarks projects on (OSX 10.11.3) and the Hello world application runs fine in Eclipse Mars 4.5.1. However when I run the DevelopmentSample application I get the error below before the series of '.' start printing in the console. Looking at the code from that particular sample app it's not obvious to what the necessary jetty dependencies are. Should I have copied something from the quarks distribution to jetty webapps?


Feb 17, 2016 7:04:02 PM org.eclipse.jetty.util.log.Log initialized
INFO: Logging initialized @253ms
Feb 17, 2016 7:04:02 PM org.eclipse.jetty.server.Server doStart
INFO: jetty-9.3.6.v20151106
Feb 17, 2016 7:04:02 PM org.eclipse.jetty.server.handler.ContextHandler doStart
INFO: Started o.e.j.s.ServletContextHandler@34ce8af7{/jobs,null,AVAILABLE}
Feb 17, 2016 7:04:02 PM org.eclipse.jetty.server.handler.ContextHandler doStart
INFO: Started o.e.j.s.ServletContextHandler@b684286{/metrics,null,AVAILABLE}
Feb 17, 2016 7:04:02 PM org.eclipse.jetty.webapp.WebInfConfiguration getCanonicalNameForWebAppTmpDir
WARNING: Can't generate resourceBase as part of webapp tmp dir name: java.lang.IllegalStateException: No resourceBase or war set for context
Feb 17, 2016 7:04:02 PM org.eclipse.jetty.webapp.WebAppContext doStart
WARNING: Failed startup of context o.e.j.w.WebAppContext@880ec60{/console,null,null}{}
java.lang.IllegalStateException: No resourceBase or war set for context
at org.eclipse.jetty.webapp.WebInfConfiguration.unpack(WebInfConfiguration.java:406)
at org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:72)
at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:480)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:516)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:405)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at quarks.console.server.HttpServer.startServer(HttpServer.java:90)
at quarks.providers.development.DevelopmentProvider.(DevelopmentProvider.java:65)
at quarks.samples.topology.DevelopmentSample.main(DevelopmentSample.java:19)

Feb 17, 2016 7:04:02 PM org.eclipse.jetty.server.AbstractConnector doStart
INFO: Started ServerConnector@643b1d11{HTTP/1.1,[http/1.1]}{0.0.0.0:51600}
Feb 17, 2016 7:04:02 PM org.eclipse.jetty.server.Server doStart
INFO: Started @491ms
http://localhost:51600/console
.................................

Sensors analytic module

I'm proposing a new Quarks module under analytics/sensors

This would contain analytics that are aimed at sensor readings.

My initial contribution would be a deadband filter, other contributions would be welcome.

Here's the Javadoc comment.

    /**
     * Deadband filter with maximum suppression time.
     * 
     * A filter that suppresses (discards) any tuples that are in the deadband, uninteresting to downstream consumers.
     * An exception exists such that when input tuple arrives it is forwarded downstream
     * regardless of its value if it has been more than {@code maximumSuppression} seconds since
     * the last tuple was passed through.
     * <P>
     * A tuple {@code t} is passed through the deadband filter if:
     * <UL>
     * <LI>
     * {@code deadband.test(value.apply(t)) == false} // outside of the deadband
     * </LI>
     * <LI>
     * or it has been more than {@code maximumSuppression} seconds (in unit {@code unit}) 
     * </LI>
     * 
     * @param <T>
     * 
     * @param stream Stream containing readings.
     * @param value Function to obtain the value passed to the deadband function.
     * @param deadband Function that defines the deadband.
     * @param maximumSuppression Maximum amount of time to suppress values that are in the deadband.
     * @param unit Unit for {@code maximumSuppression}.
     * @return Filtered stream.
     */

A simple example use.

        TStream<Double> temps = ..;

       // Discard temperature readings that are in the range 10.0 to 30.0  but send
       // a reading if it's been more than a minute since the last one.
        temps = deadband(temps, identity(),
                v-> v >= 10.0 && v <= 30.0, 1, MINUTES);

Control Service

The vision for control service is to allow Quarks and Quarks applications to be controlled remotely from centralized systems, such as start/stop applications and change the polling rate for a periodic source.
In addition applications can register arbitrary control beans that can be registered with the control service.

This issue would fully flesh out the scheme by adding a remote implementation implemented against the generic IotDevice, where device commands result in calls to the control mbeans.

DevelopmentProviderTest - console.war not found & can't load StaticLoggerBinder

java.lang.RuntimeException: java.lang.Exception: console.war not found. Run 'ant' from the top level quarks directory, or 'ant' from 'console/servlets' to create console.war under the webapps directory. at quarks.test.providers.dev.DevelopmentTestSetup.createTopologyProvider(DevelopmentTestSetup.java:20) at quarks.test.providers.dev.DevelopmentTestSetup.createTopologyProvider(DevelopmentTestSetup.java:13) at quarks.test.topology.TopologyAbstractTest.setup(TopologyAbstractTest.java:34) Caused by: java.lang.Exception: console.war not found. Run 'ant' from the top level quarks directory, or 'ant' from 'console/servlets' to create console.war under the webapps directory. at quarks.console.server.HttpServer.getInstance(HttpServer.java:70) at quarks.providers.development.DevelopmentProvider.<init>(DevelopmentProvider.java:63) at quarks.test.providers.dev.DevelopmentTestSetup.createTopologyProvider(DevelopmentTestSetup.java:17) [TEST-quarks.test.providers.dev.DevelopmentProviderTest.xml.txt](https://github.com/quarks-edge/quarks/files/148646/TEST-quarks.test.providers.dev.DevelopmentProviderTest.xml.txt)

Cannot rebase upstream/master

I'm trying to create a new branch so I can work on an issue this morning, however I can not do a rebase upstream/master.

Here is what I have done so far:

git checkout master <-- works
git fetch upstream <-- works
git rebase upstream/master <-- does not work:

Susans-MacBook-Pro-247:quarks susancline$ git rebase upstream/master
First, rewinding head to replay your work on top of it...
error: The following untracked working tree files would be overwritten by checkout:
samples/connectors/src/main/java/quarks/samples/connectors/iotf/IotfSensors.java
Please move or remove them before you can switch branches.
Aborting
could not detach HEAD

Does anyone know how to fix this so I can proceed?

Susans-MacBook-Pro-247:quarks susancline$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

Heads-up - Code upload to ASF early next week (2016/03/07).

FYI

Now that the CCLA has be received by the ASF the code can be uploaded to the ASF repo.

I'm planning to do this early next week (March 7th or 8th).

There may still be a short period while this Quarks github repo is the one to use for active development, until we understand the process of making changes using git at the ASF (and mirroring of the repo to github). Any changes made in that period here after the initial code drop to the ASF would be migrated to the ASF repo.

Code being at the ASF is one of the requirements to fulfill before the project can use the Apache name.

http://mail-archives.apache.org/mod_mbox/incubator-quarks-dev/201603.mbox/%3CCAFFD%2B2zNBLF7fvz4bnasMM_0d4PvMot%3DySVx-MpL940LCDYjMA%40mail.gmail.com%3E

MqttOpenTest.testConnectRetrySub AssertionFailedError

The test output file is from Windows.
TEST-quarks.test.connectors.mqtt.MqttOpenTest.xml.txt

Here's an excerpt:

junit.framework.AssertionFailedError: 11bd1462_bb66_44ff_a1c1_4045e01339ff_testConnectRetrySub_sub contents:[]
at quarks.test.topology.TopologyAbstractTest.completeAndValidate(TopologyAbstractTest.java:85)
at quarks.test.connectors.common.ConnectorTestBase.completeAndValidate(ConnectorTestBase.java:90)
at quarks.test.connectors.common.ConnectorTestBase.completeAndValidate(ConnectorTestBase.java:79)
at quarks.test.connectors.mqtt.MqttStreamsTestManual.testConnectRetrySub(MqttStreamsTestManual.java:916)

Python api request

This a fantastic new tool for embedding in edge devices. Many analytical tools these days support access from different languages. While Java was an obvious and logical first step I would very much like to see support added for python as well.

Add a WebSocket connector

Features:

  • Web Socket Client
  • Web Socket Server
  • support binary and string payloads

pull #32 contains the proposed API

Java 8 Support - ME Or SE

Would like to try Quarks on the IOT oriented ARM Cortex M3 / M4. Will Quarks run on Java ME? Or even SE embedded?

Add batch support to the Quarks windowing API

The API changes are the following additions to TWindow<T,K>:

"
TStream batch(BiFunction<List,K,B> batcher)
TStream batch(BiFunction<List,K,B> batcher, long time, TimeUnit unit)

Process a window in batches. batcher is called when a partition in this window is complete.

Once a partition in this window is complete batcher is called passing the current contents of the partition. When batcher returns, the partition contents are discarded and the partition starts to fill again, the window partition tumbles.

This means the window partition is processed in non-overlapping batches, a tuple is only processed once by batcher.

For a count based window of size N a partition becomes complete when it contains N tuples.

For example with last(10) batcher will be called after the partition has received ten tuples passing the first ten tuples (1-10), then after a further ten tuples have been received batcher is called passing tuples 11-20, etc. Thus at the time the partition is visible, when batcher is called, the partition contains the last ten tuples on the stream for its key, which is its definition.

For time based windows a partition of S seconds becomes complete (approximately) every S seconds starting when it receives its first tuple on the stream. If no tuples arrived during the time the partition was waiting to become complete then batcher is called passing an empty list of tuples.

For example last(5, SECONDS) at the time the partition is visible, when batcher is called, the partition contains the last fives seconds of tuples on the stream for its key, which is its definition."

Feedback for /

At the top under "Quarks Documentation" it says:

Documentation for Quarks Project

Which is not very helpful, seems like a brief overview of what Quarks is would be useful here.

MqttOpenTest java.lang.RuntimeException: java.util.concurrent.TimeoutException

I am running the tests and they seemed to be progressing well but then I got the error below on MqttOpenTest testConnectRetrySub

I am guessing this is user error of some sort, but wanted to see if anyone knows the solution off hand.

I am running on Windows 7

java.lang.RuntimeException: java.util.concurrent.TimeoutException

java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrrent.TimeoutException
at java.util.concurrent.FutureTask.report(FutureTask.java:123)
at java.util.concurrent.FutureTask.get(FutureTask.java:207)
at quarks.runtime.etiao.TrackingScheduledExecutor$TrackedFuture.gget(TrackingScheduledExecutor.java:238)
at quarks.topology.spi.tester.AbstractTester.complete(AbstractTester.java:32)
at quarks.test.topology.TopologyAbstractTest.complete(TopologyAbstractTest.java:59)
at quarks.test.topology.TopologyAbstractTest.completeAndValidate(TopologyAbstractTest.java:83)
at quarks.test.connectors.common.ConnectorTestBase.completeAndValidate(ConnectorTestBase.java:90)

at quarks.test.connectors.common.ConnectorTestBase.ccompleteAndValidate(ConnectorTestBase.java:79)
at quarks.test.connectors.mqtt.MqttStreamsTestManual.testCConnectRetrySub(MqttStreamsTestManual.java:916)
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException
at quarks.runtime.etiao.Exeecutable.invokeAction(Executable.java:202)
at quarks.runtime.etiaao.Executable.start(Executable.java:154)
at quarks.runtime.etiao.EtiaoJob.stateChange(EtiaoJob.java:70)
at quarks.providers.direct.DirectTopology.execute(DirectTopology.java:73)
at quarks.providers.direct.DirectTTopology$1.call(DirectTopology.java:61)
at quarks.providers.direct.DirectTopology$1.call(DirectTopology.java:57)
at jjava.util.concurrent.FutureTask.run(FutureTask.java:267)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:1180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at quarks.runtime.etiao.TrackingScheduledExecutor$TrackedFuture.run(TrackingScheduledExecutorr.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143)
at java.util.concurrent.ThreadPooolExecutor$Worker.run(ThreadPoolExecutor.java:618)
at quarks.runtime.etiao.ThreadFactoryTracker$2.run(ThreadFactoryTracker.java:70)
at java.lang.Thread.run(Thread.java:785)
Caused by: java.util.concurrent.TimeoutException

timebatchWindowTest failing

I just pulled from master and now seeing the test fail on a clean clone.

junit.framework.AssertionFailedError
at quarks.test.window.WindowTest.timeBatchWindowTest(WindowTest.java:371)

Create a window from another streams's window

Allows creation of a stream window with the same configuration as an existing window.

// Create a new window with the same configuration as another window
TWindow<T,Object> window(TWindow<?,?> configWindow);

Tag naming convention

  • Each tag has a unique name, which is a simple dotted name, like "com.myapp.MyFilter".
    This allows developers to encode context info directly into a tag's name.
  • Quarks does not check a tag name for compliance to this notation rule.
    Developers are encouraged to follow this convention usage in order to avoid
    name clashes with tags generated by Quarks or other Quarks applications.
  • Tags starting with 'quarks' are reserved for use by Quarks.
    A set of helper methods are used for easily creating tag names:

Is there a better place to add these to, instead of default Tagable methods?

    /**
     * Concatenates a class name and additional names to form a string where 
     * elements are separated by dots.
     * <p>
     * Empty or null names are ignored.
     */
    default String name(Class<?> clazz, String... names) {
        ...
    }

    /**
     * Concatenates the specified firstName and additional names to form a 
     * string where elements are separated by dots.
     * <p>
     * Empty or null strings are ignored.
     */
    default String name(String firstName, String... names) {
       ... 
    }

Adding tagging at all topology levels

Currently, TStreams can be tagged when a topology is declared. Tags are inserted in the JSON graph representation, so the console can display tags associated with the graph elements, show streams by tag name, etc.

Application developers should have the ability to add tags at all topology levels. This requires a Tagable interface that may be implemented by children of TopologyElement:

public interface Tagable {
    /**
     * Adds the specified tags. Adding the same tag multiple times will not change 
     * the result beyond the initial application.
     * 
     * @param values  Tag values.
     */
    void tag(String... values);

    /**
     * Returns the set of tags associated with this instance.
     * 
     * @return set of tag values.
     */
    Set<String> getTags();
}

DevelopmentProvider with flag to suppress metrics

Proposition:
As I'm writing a sample console application I'd like to have control over having a Metrics counter for the entire topology. As far as I can tell, if I want to get the other additions in DevelopmentProvider over DirectProvider, like the JMXControlService, I need to use DevelopmentProvider.

However, at times I don't want counter metrics, but instead would like to add these to individual streams.

I propose that we add a flag or provide an additional constructor to turn off metrics in the submit() method.

Update Kafka connector to 0.9 client jars and features

  • pickup necessary kafka-0.9 jars, update connector if/as needed, run tests
  • add support to use SSL support added in 0.9 [spin off into separate issue?]
  • use new client API added in 0.9 [spin off into separate issue?]

MqttStreamsTestManual.testConnectFail fails with java.util.concurrent.TimeoutException

OS: Rhel7
Mosquitto 1.4.2

Start mosquitto:
From /homes/hny7/maywone/MQTT/org.eclipse.mosquitto-1.4.2
-bash-4.2$ installed/usr/local/sbin/mosquitto -v -c ~/MQTT/org.eclipse.mosquitto-1.4.2/mosquitto.conf โ€“v

Run test:
ant -Dtest.base.pattern='**/MqttStreamsTestManual.java' test

Test run 1:
unittests_run1.zip
Excerpt from /homes/hny7/maywone/git/quarks/connectors/mqtt/unittests/

java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException at java.util.concurrent.FutureTask.report(FutureTask.java:133) at java.util.concurrent.FutureTask.get(FutureTask.java:217) at quarks.runtime.etiao.TrackingScheduledExecutor$TrackedFuture.get(TrackingScheduledExecutor.java:238) at quarks.topology.spi.tester.AbstractTester.complete(AbstractTester.java:32) at quarks.test.topology.TopologyAbstractTest.complete(TopologyAbstractTest.java:59) at quarks.test.topology.TopologyAbstractTest.completeAndValidate(TopologyAbstractTest.java:83) at quarks.test.connectors.common.ConnectorTestBase.completeAndValidate(ConnectorTestBase.java:90) at quarks.test.connectors.common.ConnectorTestBase.completeAndValidate(ConnectorTestBase.java:79) at quarks.test.connectors.mqtt.MqttStreamsTestManual.testConnectFail(MqttStreamsTestManual.java:490) Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException at quarks.runtime.etiao.Executable.invokeAction(Executable.java:202) at quarks.runtime.etiao.Executable.start(Executable.java:154) at quarks.runtime.etiao.EtiaoJob.stateChange(EtiaoJob.java:70) at quarks.providers.direct.DirectTopology.execute(DirectTopology.java:73) at quarks.providers.direct.DirectTopology$1.call(DirectTopology.java:61) at quarks.providers.direct.DirectTopology$1.call(DirectTopology.java:57) at java.util.concurrent.FutureTask.run(FutureTask.java:277) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:191) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at quarks.runtime.etiao.TrackingScheduledExecutor$TrackedFuture.run(TrackingScheduledExecutor.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at quarks.runtime.etiao.ThreadFactoryTracker$2.run(ThreadFactoryTracker.java:70) at java.lang.Thread.run(Thread.java:785) Caused by: java.util.concurrent.TimeoutException #

Test run 2:
Increase the timeout from 10s to 100s
In quarks.runtime.etiao.Executable.invokeAction, edit line 198, changing 10 to 100.
Future completed = completer.poll(10, TimeUnit.SECONDS); //original
Future completed = completer.poll(100, TimeUnit.SECONDS); //new

unittests_run2_Using100s.zip

The result is slightly different:

java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask.get(FutureTask.java:216)
at quarks.runtime.etiao.TrackingScheduledExecutor$TrackedFuture.get(TrackingScheduledExecutor.java:238)
at quarks.topology.spi.tester.AbstractTester.complete(AbstractTester.java:32)
at quarks.test.topology.TopologyAbstractTest.complete(TopologyAbstractTest.java:59)
at quarks.test.topology.TopologyAbstractTest.completeAndValidate(TopologyAbstractTest.java:83)
at quarks.test.connectors.common.ConnectorTestBase.completeAndValidate(ConnectorTestBase.java:90)
at quarks.test.connectors.common.ConnectorTestBase.completeAndValidate(ConnectorTestBase.java:79)
at quarks.test.connectors.mqtt.MqttStreamsTestManual.testConnectFail(MqttStreamsTestManual.java:490)

Note: I see similar results using the Really Small Message Broker.

Link to current committers does not work

When browsing through the web site: http://quarks-edge.github.io/

If I go to this page:
http://quarks-edge.github.io/quarks.documentation/docs/quarks/committers/
it works, but if I go to the 'Current committers' page it does not:

  1. https://github.com/orgs/quarks-edge/teams/quarkscommitters
  2. On this page:
    http://quarks-edge.github.io/quarks.documentation/docs/quarks/quarks-getting-started/
    The section called 'Adding the console to your application' has formatting issues. It is difficult to tell the lines of code from the comments section.

If I find other broken links or areas with formatting that needs improvement, I'll add them to this issue. Thanks!

Add pub-sub model between jobs

Allow a running job to publish tuples that can be subscribed to by other running jobs.

Method in TStream:

<U super/extends T> TSink<T> publish(String topic, Class<U> streamType)

Method in Topology

TStream subscribe(String topic, Class streamType)`

This would be tied with a PublishSubscribe service that can be added to a provider.

Initial implementation would allow pub-sub within provider.

Default system tags

Quarks should add default tags to topology elements. The console should use known Quarks tags to show useful information.

TBD

  • What sort of information would default tags convey? The graph JSON representation already provides type info via the className attribute...
  • e.g. would tags attached to a Filter propagate to the downstream reachable TStreams?

Expose more tag information in the console

While writing a sample application for the console I realized we need more tag support exposed in the console. Today I noticed 3 issues opened also relating to tagging: #39 , #40 , #41 .

Here are some thoughts about how the console could be improved wrt tagging:

  1. Topology graph representation: Make the color of the oplets light grey when 'View by' is set to 'Stream tags'. Also, change the legend circles that show the tag color to squares to more closely mimic the stream dimensions - not to be confused with the circles that represent the oplets.
  2. Allow the user to select one, some or all tags to view in the topology graph at one time
  3. Add tag information to all popups:
    The 'View all oplet properties'
    The hover over the oplet
    The hover over the stream

Additional thoughts folks have??

Allow batch() option to not process empty windows?

The current batch implementation will process empty windows when the window is the last N seconds.

I wonder if this is the desired or expected behaviour. I see batching as breaking a stream into batches of tuples, thus processing an empty one doesn't seem to make much sense.

Batch could have an option to allow processing of empty windows, defaulting to false, or just not support it.

Throughts?

iotf connector and javax.servlet

@ddebrunner what's the scoop with the presence of iotf/ext/lib/javax.servlet_3.0.0.jar?
Is that cruft from a dependency of an earlier version of com.ibm.iotf.client ? It's not listed as a dependency on https://github.com/ibm-messaging/iot-java.
Similarly, iotf/ext/lib/com.ibm.json4j is present and it's not listed as a dependency (gson is).
Both jars are referenced in connectors/.classpath. Can the be removed?

The reason why I ask is that I'm having an issue in Eclipse related to developing the WebSocket client connector. As part of the tests I need a WebSocket server and I'm using jetty componentry to create it. The problem is that connectors is a single project with a single .classpath file. My jetty WebSocket server needs its servlet-api-3.1.jar. However the presence of javax.servlet_3.0.0.jar in the .classpath causes problems running my server from Eclipse. This problem doesn't happen from ant as each connector has its own build.xml.

SingleMetricAbstractOplet always creates a metric name

This code (line 42) creates a metric name even when metrics are not enabled, and thus the name is never used. Basically a minor footprint overhead that is not needed.

        this.metricName = context.uniquify(shortMetricName);
        MetricRegistry registry = context.getService(MetricRegistry.class);
        if (registry != null) {
            registry.register(getMetricName(), getMetric());
        }

MqttStreams/MqttDevice Connectors: add SSL support

  • update MqttConfig for key/trust store info
  • add support to MqttConnector
  • add test cases
  • expose support from samples/MqttClient and MqttSample&mqtt.properties
  • expose support from samples/apps applicationTemplate.properties SensorAnalytics sample

runiotfSensors.sh script - documentation on required parameter and IoTf setup

-bash-4.2$ ./runiotfsensors.sh
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 0
at quarks.samples.connectors.iotf.IoTFSensors.main(IoTFSensors.java:39)

I'm looking for information (ex. README) on how to run runiotfSensor.sh.

Excerpt from runiotfSensor.sh:
.# Runs the Sample MQTT Publisher or Subscriber

.# ./runmqttsample.sh pub
.# ./runmqttsample.sh sub
export CLASSPATH=${quarks}/samples/lib/quarks.samples.connectors.jar
java quarks.samples.connectors.iotf.IoTFSensors $1

Where to find information on how to specify '$1'? Also the comments on MQTT probably need updating.

Per IoTFSensors.java requires a deviceCfg file.
It looks like I need to set up IoTF โ€“ is the standard setup sufficient to run this script?

Add a generic serial port connector api

Some devices access information through a serial port, add a generic connector interface that represents a serial port, to allow applications/libraries be written against the interface, rather than a specific implementation.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.