ftsrg/ire

This repository is only kept for historical reasons, the projects was merged to the ingraph repository

ire's Issues

Synchronize the names w/ the ones used in ingraph

• Trimmer -> ProjectionNode
• Checker -> SelectionNode
• AlphaNode -> UnaryNode
• BetaNode -> BinaryNode

Also:

• HashAntijoiner -> AntiJoinNode
• HashJoiner -> JoinNode

Use Vectors instead of Maps

We should get rid of maps as they 1) consume a lot of memory 2) provide much worse performance than vectors. For now, we should restrict the Rete network to fixed with tuples, with vertex/edge ids and simple properties (used for selection). This will work fine for queries that return the IDs:

MATCH (n)
RETURN ID(n)

However, it will not not suffice even for simple queries, e.g.

MATCH (n)
RETURN n

This query returns all the properties of the n variable.

To get the properties of n, we will have to join n to it's properties in the production node. For this, we have to implement a node similar to the properties() method in Neo4j. E.g. the get-properties nullary operator should return 2-tuples of <ID, properties>, e.g. <1, {name: "John", age: 34}>, etc.

Remove remoting components

ire should focus on query processing, and not remote messaging - we should remove the akka-remote components and the related source code files.

Include the source code in JARs

• http://stackoverflow.com/questions/11474729/how-to-build-sources-jar-with-gradle
• http://fw-geekycoder.blogspot.ca/2012/12/how-to-include-source-files-into-jar-in.html

Use shorthand for generating test tuples

Refactor this method an use for all tests (where applicable): https://github.com/FTSRG/ire/blob/5c574a1894f9e8f95bac4b9ec358a37f3504fb42/src/test/scala/HashAntijoinerTest.scala#L15

Incremental evaluation of standard Cypher constructions

(Related ticket: ftsrg/ingraph#35)

Use this issue to keep track of supported Cypher constructs.

Clauses

• Query constructs
  • MATCH
    • Transitive closure #12
  • RETURN (Production node)
  • UNWIND
  • OPTIONAL MATCH (left outer join)
  • WITH this is translated to joins in the query plan
  • UNION #13
• Data manipulation
  • CREATE
  • MERGE
  • SET
  • DELETE
  • DETACH DELETE
  • REMOVE
• Procedure calls
  • CALL ... YIELD (as this call may run an arbitrary Java code, there is no way we can handle these incrementally)

Sub-clauses

• Conditions
  • WHERE ftsrg/ingraph#43
• Ordering
  • ORDER BY #3
  • SKIP #4
  • LIMIT #4
• Data manipulation
  • ON CREATE
  • ON MATCH

Operators

• DISTINCT #14
• String operations
  • STARTS WITH
  • CONTAINS
  • ENDS WITH
• Arithmetic comparison ftsrg/ingraph#43
  • =
  • <>
  • <
  • >
  • <=
  • >=
• Arithmetic operators
  • +
  • -
  • /
  • *
• Logical operators https://github.com/FTSRG/ingraph/tree/master/queries/ldbc-snb
  • NOT
  • AND
  • OR
  • XOR
• NULLs
  • IS NULL
  • IS NOT NULL
• Accessors
  • . (property access) not required as the query plan will be translated to tuples
  • [] (subscript)

Expressions

• List comprehensions
• Pattern comprehensions
• Parameters with new syntax ($)
• Literals they are just constants
• Aggregation functions #2
  • avg()
  • count()
  • max()
  • min()
  • sum()
  • collect()

Functions

• Statistical functions
  • stdDev()
  • stdDevP()
• Predicate functions
  • exists()
• Scalar functions
  • coalesce()
  • endNode()
  • head()
  • length()
  • last()
  • properties() #19
  • size()
  • startNode()
  • type() #19
  • toFloat()
• List functions
  • relationships()
  • tail()
  • keys() #19
  • labels() #19
  • nodes()
  • range()
• Numeric functions
  • abs()
  • ceil()
  • floor()
  • rand()
  • round()
  • sign()
• Logarithmic functions
  • e()
  • exp()
  • log()
  • log10()
  • sqrt()
• Trigonometric functions
  • acos()
  • asin()
  • atan()
  • atan2()
  • cos()
  • cot()
  • degrees()
  • pi()
  • radians()
  • sin()
  • tan()
• String functions
  • left()
  • right()
  • trim()
  • lTrim()
  • rTrim()
  • replace()
  • reverse()
  • split()
  • substring()
  • toString()
  • toLower() (lower())
  • toUpper() (upper())
• Statistical functions
  • percentileCont()
  • percentileDisc()
• Not included in Cypher:
  • toBoolean()
  • toInteger() (called toInt())

Types

• primitives
• list
• map
• node
• relationship
• path

Implement UNWIND operator

Skeleton: https://github.com/FTSRG/ire/blob/master/src/main/scala/hu/bme/mit/ire/nodes/unary/UnwindNode.scala

Implement left outer join node

Add operator for transitive closure

R+(src, trg)

Implement basic monitoring features

E.g. track the number of tuples in the network.

Nodes should switch between bag/multiset/list semantics

Nodes have quite a large variation in their inputs and outputs:

Add operators for aggregation

Aggregation

Aggregation functions including count(), sum(), avg(), max(), min(), collect() and others. Includes DISTINCT.

https://neo4j.com/docs/developer-manual/current/cypher/#query-aggregation

• count()
• sum()
• avg()
• max()
• min()
• collect(): https://github.com/FTSRG/ingraph/blob/master/queries/fraud-detection/financial-risk.cyp#L10

SKIP/LIMIT operator

Implement a SKIP...LIMIT operator, which works on ordered expressions: https://neo4j.com/docs/developer-manual/current/cypher/#_skip_and_limit

Make Transaction AutoClosable

I think it would be a good idea to make transactions Autoclosable, similarly to Neo4j's Transaction.

Add operator for union

The union operator is a simple binary node, which produces – you guessed it – the union of its input relations.

Cleanup unused code

• RDFReader and RdfReaderTest classes.
• container_performance
• DistributedQueries, RemoteOnlyQuerie, Slave

Add operator for duplicate elimination

Related issue: #5

Selection node expectedTerminatorCount

Why do we override the value of expectedTerminatorCount to 2.

If we keep it with a value of 2, the following test case becomes: nondeterministic

  "Unary nodes" must {
    "propagate terminator messages" in {

Implement ORDER BY operator

ORDER BY

A sub-clause following RETURN or WITH, specifying that the output should be sorted in particular way.

https://neo4j.com/docs/developer-manual/current/cypher/#query-order

The user of the library should specify a list of variable-direction pairs (e.g. person-ASC, city-DESC) for the ordering.

Support metaqueries

We should definitely support labels() and type(). I am not sure about properties() and keys().