lsds/seep

Cluster computing using Stateful Dataflow Graphs

seep's Issues

Force operatorsId > 0, so that we can propagate erros when opId = -1

Single scale out mode per query

The scale out mechanism is chosen from the checkpoint mechanism. This means that only one can be used at a given time for a given query, which forces all the stateful operators of the query to use the same checkpointing mechanism.

Set system ready for single node deployment

Initial work is done, check and validate.

Build Failure, missing dependency soot

Hello, SEEP Team.

I tried to build you project. I am running into the problem, that the following dependency was not available via the default public Maven repositories. Is there a repository declaration missing, or is it necessary to manually install the dependency into the local repository?

    <dependency>
        <groupId>soot</groupId>
        <artifactId>soot-framework</artifactId>
        <version>2.5.0</version>
    </dependency>

Thank you very much.

Tuple implementation falls short with the new generalised dataflow model.

Transform into maven project

Include Kryo and Metrics as external dependencies. So far it's been good enough with Ant, but now I will include some java collections external implementation, so it makes sense to migrate the project to Maven.

Not possible to checkpoint non-partitionable state with the large-memory check-pointing primitive.

Improve performance of log handling

Reconfigure barrier when more than one InputDataIngestionMode per operator.

In particular, the problem is how to understand how many of the total number of upstream operators have to synchronise in the barrier. (These are those that have scaled out from the same original operator).

redundant .setting and target folders

On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
    modified:   seep-java2sdg/.settings/org.eclipse.jdt.core.prefs
    modified:   seep-system/.settings/org.eclipse.jdt.core.prefs

Untracked files:
    seep-java2sdg/.settings/org.eclipse.core.resources.prefs
    seep-java2sdg/.settings/org.eclipse.m2e.core.prefs
    seep-streamsql/.settings/
    seep-streamsql/target/
    seep-system/.settings/org.eclipse.core.resources.prefs
    seep-system/.settings/org.eclipse.m2e.core.prefs
    seep-system/examples/stateless-simple-query/dist/
    seep-system/libs/

These could be deleted or git ignored.

Sort runtimeengine package

There are different layers in this package. It has to be consistent with the rest of the codebase.

One InputDataIngestionMode per upstream logic operator, even if all use same mode.

Implications on hop-by-hop semantics.
Correct but not sure impact on performance, in particular when one of the upstream have scaled out and N threads compete for a queue, while only 1 (potentially) competes for its unique queue.

Reuse channels created on state backup when streaming state from disk to memory

Remove concept of local and remote operators

Any optimisation regarding which "logical" operators should be placed in the same node is a high-level optimisation. Therefore, when deploying operators in the cluster, each "deployable" operator may implement the body of N "logical" operators, but it is always only one operator being deployed. This reduces complexity at deployment time.

New interfaces for state. Versionable and Streamable need to be specified when creating "large state"

Name all threads in the system

DataConsumer is not ready to accept data from different bridges when these have different InputDataIngestionMode

Star topology implementation for FT

With this done, it is necessary to figure out how to include output buffers in the backup.

Provide feedback when loading query

For example, if there is a failure due to lack of sources or sinks, indicate this. Or if there is a declared stateful operator without state. This would already increase the usability of the current interface

No multicore support when there are more than 1 inputDataIngestionModes in a given operator

It's highly improbable that you want to leverage multicore when you are synchronising in a barrier with a dynamic number of threads, and still has another number N of threads to support more ingestion modes.
However, it would be nice to at least provide these support for modes that are one-at-a-time for example.

Unable to run the stateless-simple-query example in local mode.

I put both seep-system-0.0.1-SNAPSHOT.jar and stateless-simple-query.jar in the dist folder and run:

java -jar seep-system-0.0.1-SNAPSHOT.jar Master stateless-simple-query.jar Base

It gives me the following errors:

java.lang.ClassNotFoundException: Base
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at uk.ac.imperial.lsds.seep.infrastructure.master.MasterController.executeComposeFromQuery(MasterController.java:153)
    at uk.ac.imperial.lsds.seep.Main.executeMaster(Main.java:69)
    at uk.ac.imperial.lsds.seep.Main.main(Main.java:44)

I know it can not locate the Base class. Could you please tell me where is the problem? Thanks.

Deployment ordering

Make both workers and master to poll until system is ready to deploy. This is in contrast to enforce an ordering when deployint the system. (master -> workers)

Include property to indicate "local" deployment

This will avoid an annoying but not harmful exception thrown by backupHandler when deploying in local mode.

Kryo buffer underflow

Right before and after receiving state chunks in all the receivers of the star topology. It does not break the thread or the program, but they are always thrown.

Pluggable serialisation mechanism

Provide a convenient interface for serialisation/deserialisation that accepts different mechanisms.

Best way of starting the system (with artificial sources)

Provide better defaults for box-n-arrow interface

GC error when buffering tuples after manually scaling out

When manually scaling out, some upstream operators are not registered down. Operators cannot forward ACKs upstream, and thus upstream output buffers fill.

send to streamId fails with only one operator

It may not make sense to use it if there is only one stream. However, a meaningful message should warn the user that the call will fall back to the default send in such case.