dgraph-io/benchmarks

Run benchmarks with RDF data

benchmarks's Issues

Remove all adult content from benchmark data

In order to avoid having adult movie titles appearing during demos using https://play.dgraph.io we should remove all the related data from the benchmark dataset.

Currently, there are 6516 adult movies on the dataset, as shown in the query below:

{
  q(func:eq(name@., "Pornographic film")) {
    name@.
    count(~genre)
  }
}

{
  "extensions": {...},
  "data": {
    "q": [
      {
        "name@.": "Pornographic film",
        "count(~genre)": 6516,
        "uid": "0x62b974"
      }
    ]
  }
}

Neo4j MERGE queries inefficient

benchmarks/data/neo4j/main.go

Lines 171 and 176:

"MERGE (n { xid: {xid1} }) MERGE (n2 { xid: {xid2} }) MERGE (n) - [r:`%s`] - (n2)"
...
"MERGE (n { xid: {xid} }) ON CREATE SET n.`%s` = {val} ON MATCH SET n.`%s` = {val}"

These merges are being performed without any supplied label on n or n2, and an index should be present for property xid on whatever label is chosen for these nodes.

As it is now, neither of these merges uses the index, and in each case this requires all nodes in the db to be scanned looking for matches on the xid property of all nodes with each merge. Label-less and index-less merging or matching kills performance and is highly discouraged.

Please fix.

Is there an error in release.schema?

data/release/release.schema file.

type Actor {
    name: string
    actor.film: [Film]
    actor.dubbing_performances: [Film]
}

Is it correct to change to actor.film: [Performance]？

Error default value 'full', should be 'all'

benchmarks/cachebench/ristretto/bench.go

Line 48 in 8ed23af

"full",

	flagSuite = flag.String(
		"suite",
		"all",
		`You can chose from the following options:
		"all"   - hit ratio and speed performance
		"hits"  - hit ratio
		"speed" - throughput
		`,
	)

Address Neo4j misconfiguration and inefficiency issues

From Michael Hunger's response to your benchmarking article:

Just really quickly. Unfortunately, your benchmark has a number of issues, that invalidates all its Neo4j measurements.

We recommend users in general to ignore vendor benchmarks and test with their own hardware, data, use-cases for relevant and reliable results.

Here is a quick list from just skimming over, I didn't measure or test anything so don't assume it is all correct / working:

General:

• Neo4j is no RDF database, so RDF data model makes no sense
• using a community provided Go Driver for which performance has not been validated, not an official driver like JS, Java, .Net
• incorrect information in feature table
• memory usage can be configured

Writes:

• merge query without labels (:Node) each statement does 2 full all-node-scans
• no constraint for :Node(xid)
• no timing published for csv import (11s on my mac 1.1M triples)
• doesn't use transactions of eg. 50k or 100k updates per request
• which can be best achieved with a single query per tx and UNWIND of a payload of a array of structs

Reads:

• no use of parameters in reads
• disabled query plan cache (which was incorrectly understood as query result cache, which doesn't exist)
• no constraint for :Film(filmId)

Good luck with your development of dgraph, it looks like a good technology for RDF use-cases.

Cheers, Michael@neo4j

Is Readme incomplete for data?

I am referring to readme here

https://github.com/dgraph-io/benchmarks/blob/master/data/README.md

It refers to different files but following files are not checked in

names.gz
rdf-films.gz

names.gz is surely not there. But rdf-films.gz is there but renamed somehow?

Can someone please clear this?

Compare with Cayley

https://github.com/google/cayley

Add test code to Arangos big comparison

ArangoDB added an imho interesting comparison to its homepage and the comment section is full with requests for dgraph:

https://www.arangodb.com/2018/02/nosql-performance-benchmark-2018-mongodb-postgresql-orientdb-neo4j-arangodb/

Based on the results with neo4j on your own blog and this results is it obvious that dgraph might probably provide super results.

This a chance to make dgraph even more popular :D

Test Code is outdated

Within benchmarks/data/neo4j/query_test.go,

import "github.com/dgraph-io/dgraph/query/graph"

is obsolete.

And maybe the code & schema need to be checked as well.

Update grafana dashboard

As of DGRAPH v1.1 the default grafana dashboard is missing a lot. Prometheus data seems to have changed since...