opencypher/morpheus

Morpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.

The Cypher Property Graph Query Language

This repository holds the specification of Cypher, a declarative property graph query language. Its purpose is to be central to the process of evolving the specification and standardisation of Cypher as a graph query language.

• Open Cypher Improvement Requests (CIRs)

• Open Cypher Improvement Proposals (CIPs)

Overview of the process

Changes to openCypher are made through consensus in the openCypher Implementers Group (oCIG). The process for proposing changes, voting on proposals and measuring consensus is described in this set of slides.

Refer to the Cypher Improvement Process document for more details on CIPs, CIRs, their structure and lifecycle.

The structure of this repository

• Cypher Improvement Proposals (CIP), /cip

  • Contains a list of accepted CIP documents.

• Cypher grammar, /grammar

  • Contains the Cypher grammar specification, in XML source format.

  • A more readily consumable form of the grammar is generated as output from the build and can be found here:

    • Railroad diagrams

    • EBNF

    • ANTLR4 Grammar

• Cypher Technology Compatibility Kit (TCK), /tck

  • Contains a set of Cucumber features that define Cypher behaviour, and documentation on how to use it.

• openCypher developer tools, /tools

  • Contains code that tests the integrity of the repository, generates release artifacts, and aids implementers of openCypher.

Building

This repository uses a Maven build and supports cross building for Scala 2.12 and Scala 2.13:

• For Scala 2.12, use mvn -U clean install -P scala-212

• For Scala 2.13 use mvn -U clean install -P scala-213

Contact us

There are several ways to get in touch with the openCypher project and its participants:

• Are you interested in implementing openCypher for your platform, but you have general questions and want to reach out to other community members with similar interests? Post to our Google Groups mailing list: https://groups.google.com/forum/#!forum/opencypher

• For specific feature requests or bug reports, please open an issue on this repository.

• Do you have a particular contribution in mind, and concrete ideas on how to implement them? Open a pull request.

Copyright

© Copyright 2015-2017 Neo Technology, Inc.

Feedback

Any feedback you provide to Neo Technology, Inc. through this repository shall be deemed to be non-confidential. You grant Neo Technology, Inc. a perpetual, irrevocable, worldwide, royalty-free license to use, reproduce, modify, publicly perform, publicly display and distribute such feedback on an unrestricted basis.

License

The openCypher project is licensed under the Apache license 2.0.

morpheus's Issues

Parameterise IRGraph with expression type

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/logical/LogicalPlanner.scala#L243

Fix type inference "Don't know how to type MapExpression"

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/test/scala/org/opencypher/caps/impl/spark/cypher/FunctionsAcceptanceTest.scala#L186

Rewrite implicit inner projection into variable during IR construction or logical planning

https://github.com/opencypher/cypher-for-apache-spark/blob/52f228eb4ee53f0fcdab5eabce075de00e903d0e/spark-cypher/src/main/scala/org/opencypher/spark/impl/table/SparkTable.scala#L151

Extract node label info from predicates

The Neo4j frontend rewrites node label info to where predicates. We need CTNode to have the correct label information as this is used by SparkCypherGraph.nodes(var, nodeType) to load only the necessary subset of nodes (scans).

Query builder tests should also make assertions about graphs

// TODO: Properly assert on graphs, also below

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/test/scala/org/opencypher/caps/ir/impl/CypherQueryBuilderTest.scala#L48

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/test/scala/org/opencypher/caps/ir/impl/CypherQueryBuilderTest.scala#L84

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/test/scala/org/opencypher/caps/ir/impl/CypherQueryBuilderTest.scala#L198

Optimize var expand when cardinality is 0

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/spark/physical/PhysicalResultProducer.scala#L558

Support unlabeled nodes in GDL

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/test/scala/org/opencypher/caps/impl/spark/cypher/FunctionsAcceptanceTest.scala#L98

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/test/scala/org/opencypher/caps/impl/spark/cypher/FunctionsAcceptanceTest.scala#L149

Find a good place for Schema -> RecordHeader conversion

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/api/record/RecordHeader.scala#L172

Normalize ScanGraphs

Remove redundant columns for implied Schema information.

Should we drop aliases of incoming CAPSRecords?

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/api/spark/CAPSScanGraph.scala#L31

Refactor test helper class

// TODO: Pull out/move over to GraphMatching

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/test/scala/org/opencypher/caps/impl/spark/CAPSRecordsAcceptanceTest.scala#L141

Cartesian product should deal with RHS graphs

Assert that they are the same?

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/spark/physical/PhysicalResultProducer.scala#L295

Support AnyVal pattern

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/ir/api/pattern/Elements.scala#L18

Define graph schema with case classes that implement traits

Spark can extract Dataset schemas from case classes. If we additionally expect these classes to extend a Node/Relationship interface, then we might use this as a convenient way to define a graph schema. These case class definitions could also be used to define the graph schema for an existing Dataset/DataFrame (provided that it has the necessary ID columns).

This is what the readme example might look like:

case class Person(id: Long, name: String) extends Node
case class Friendship(id: Long, source: Long, target: Long) extends Relationship

val persons = Seq(Person(0, "Alice"), Person(1, "Bob")) 
val friendships = Seq(Friendship(0, 0, 1), Friendship(1, 1, 0))
val graph = CAPSGraph.create(NodeScan(persons), RelationshipScan(friendships))

This is what the conversion might look like for object instances. Turning an existing Dataset + case class combination into a node/relationship scan would probably not be much different.

NodeScan/RelationshipScan.apply[T <: PropertyGraph.Node : TypeTag](
    nodes: Seq[T]
)(implicit caps: CAPSSession): NodeScan = {
  /**
    * Turn the nodes into a Dataset (also offer an API for `DataSet` + case class type that extends node/rel).
    * The Node/Relationship traits identify and guarantee the existence of ID columns that encode the graph structure.
    * Get the relationship type and primary node label from the node/relationship class names
    * (e.g. relationshipTag.getSimpleName.toUpperCase).
    * Turn `extraLabels` (if they exist) into additional node labels. All other fields are turned into properties with the field name as the property key.
    */
}

If a case class implements Node, then a sequence of it or a matching Dataset can be turned into a NodeScan. Same for Relationship/RelationshipScan.

sealed trait Entity extends Product {
  def id: Long
}

trait Node extends Entity {
  def extraLabels: Seq[String] = Seq.empty
}

trait Relationship extends Entity {
  def source: Long

  def target: Long
}

What do you think?

Add support for DISTINCT

Reduce table width when passing through filters

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/flat/FlatOperatorProducer.scala#L71

Could be related to #121

Matching for unknown label does not work

Given the following test

  test("Match on unknown label") {
    // Given
    val given = TestGraph(
      """
        |(:Person {firstName: "Alice", lastName: "Foo"})
      """.stripMargin)

    // When
    val result = given.cypher(
      """
        |MATCH (a:Animal)
        |RETURN a.name
      """.stripMargin)

    // Then
    result.records.toMaps should equal(Bag())

    result.graph shouldMatch given.graph
  }

I get the following error:

Did not find slot for a:Animal :: BOOLEAN
org.opencypher.spark.api.exception.SparkCypherException: Did not find slot for a:Animal :: BOOLEAN

Add unit tests

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/convert/toJavaType.scala#L22

Avoid duplicating data in result producer `project`

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/spark/physical/PhysicalResultProducer.scala#L136

Fix type to value for relationships

//TODO: This supports var-expand where we only track rel ids, but it's not right

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/spark/RowUtils.scala#L53

Respect return value of RecordHeader construction

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/api/record/RecordHeader.scala#L201

Result producer should propagate header in node scan

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/spark/physical/PhysicalResultProducer.scala#L57

Only distinct on id column

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/api/spark/CAPSUnionGraph.scala#L53

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/api/spark/CAPSUnionGraph.scala#L64

Support union types

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/test/scala/org/opencypher/caps/impl/spark/CAPSRecordsAcceptanceTest.scala#L61

Add unit tests

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/api/util/parsePathOrURI.scala#L20

Return Option from type conversion

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/typer/fromFrontendType.scala#L21

Push error handling to API consumers in RecordHeader

https://github.com/opencypher/cypher-for-apache-spark/blob/d95e6000439def6ae44b5932ab7f5b5f8e7e66fe/src/main/scala/org/opencypher/caps/api/record/RecordHeader.scala#L74

Compute column name without using var rel

PhysicalResultProducer.iterate: // TODO: Try and get rid of the Var rel here

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/spark/physical/PhysicalResultProducer.scala#L614

Id generation

Currently, we accept ids of external graphs and hope they never collide with one another. Additionally, the graph projection capabilities need to generate fresh ids for new entities, and they must also not collide with any other ids that may have been imported.

The only way we know of to get guarantees from Spark when it comes to id generation is the function monotonically_increasing_id(), which divides the id space per partition to generate unique numbers for each partition, given some assumption on the size of each partition and the number of partitions.

That function could only be reliably used if we also map all incoming graph entity ids to values generated from it -- which is costly. There seems to be no great way of solving this.

Come up with a caching strategy for eagerly initialised graphs

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/ir/impl/IRBuilderContext.scala#L73

Test `ExprOps.computeDependencies`

Also consider consolidation into better hierarchy

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/syntax/ExprSyntax.scala#L34

Support construction of lists and maps

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/spark/convert/rowToCypherMap.scala#L44

Result producer should not discard prev records in scans

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/spark/physical/PhysicalResultProducer.scala#L64

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/spark/physical/PhysicalResultProducer.scala#L74

Use count queries when loading from Neo4j

// TODO: use count queries

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/spark/io/neo4j/EncryptedNeo4j.scala#L52

Reuse filter code in UDF

Try to share code with cypherFilter()

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/spark/Udfs.scala#L31

Support for Python API

Python API would open this library to a significant section of developers who are not in JAVA/SCALA camp. WIth many of us doing data science in PYTHON, having only scala support seems to be a limiting factor. Is there a chance that PYTHON will be supported as well?

Fix test

// TODO: Figure out what invariant this was meant to measure

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/test/scala/org/opencypher/caps/impl/spark/CAPSRecordsAcceptanceTest.scala#L127

Plan RelationshipScan in Expand operators

Currently, relationships are loaded from the input graph. This requires code duplication in the Expand operators. Instead, we should leverage the existing RelationshipScan instead.

Verify during import that imported IDs do no collide with generated Ids

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/spark/physical/PhysicalResultProducer.scala#L201

Fuse monads in IR building

IRBuilder is monadic and calls into the SchemaTyper monad -- these calls could be fused.

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/ir/impl/IRBuilderContext.scala#L48

Consider best semantics for `GraphSource.schema` vs. `GraphSource.graph.schema`

Consider caching a computed schema, but maybe defer to implementer?

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/spark/io/file/FileCsvGraphSource.scala#L39

Clean up BlockRegistry

Make it inherit from Register or some other nicer interface

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/ir/impl/BlockRegistry.scala#L24

Schema redesign

Consider duplication vs. implied labels

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/test/scala/org/opencypher/caps/test/fixture/OpenCypherDataFixture.scala#L115

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/test/scala/org/opencypher/caps/test/fixture/OpenCypherDataFixture.scala#L110

Change `skip` to data frame based implementation

https://github.com/opencypher/cypher-for-apache-spark/blob/52f228eb4ee53f0fcdab5eabce075de00e903d0e/spark-cypher/src/main/scala/org/opencypher/spark/impl/table/SparkTable.scala#L105-L114

Consider splitting up planning of MATCH patterns

Currently it plans the whole pattern, scans and filters, as one operation. We could split this up to become more modular, similar to how projections are planned.

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/logical/LogicalPlanner.scala#L93

Verify that planned predicates have all dependencies met

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/logical/LogicalPlanner.scala#L167

Make converter produce `Option`

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/typer/fromFrontendType.scala#L21

Clean up IRBuilderContext

Needs some refactoring after multiple-graph changes

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/ir/impl/IRBuilderContext.scala#L41

Read relationship type from variable

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/flat/FlatOperatorProducer.scala#L127

CypherGraph should handle AllGiven and AnyGiven

Currently, CypherGraph.nodes and CypherGraph.relationships expect CTNode and CTRelationship arguments which are handles as AND (for ndoes) and OR (for rels) internally. To be more generic and also support the OR case for nodes (and the AND case for rels), the methods should expect Elements[Label] and Elements[RelType] and match internally.

Update frontend dependency to one where `SemanticError: Variable .* already declared` is fixed

// TODO: Fix by updating frontend dependency
// Related to using bound variables in GRAPH OF

https://github.com/opencypher/cypher-for-apache-spark/blob/3afbaabb55d6ec808e7b181f80911e332fef8aa1/src/main/scala/org/opencypher/caps/impl/parse/CypherParser.scala#L32