liberation-data/drivine

Best and fastest graph database client (Neo4j, AgensGraph, Apache AGE, FalkorDB) for Node.js & TypeScript.

drivine

⚠️ Drivine website has moved: here. Please report any broken links. Even better, a PR is very welcome! 🙏

Drivine is a graph database client for Node.js and TypeScript. It was created with the following design goals:

• Support multiple graph databases (simultaneously, if you wish). Currently AgensGraph and Neo4j (or other BOLT compatible graph DBs).
• Scale to hundreds and thousands of transactions per second, without compromising architectural integrity.

---

With regards to the second point on scaleability, let's break that down into component facets.

Features

• Facilitates the use of well understood object-oriented and functional programming patterns.
• Supports implementation of code that adheres to a single responsibility principle (SRP). NestJS will be optional, but is recommended.
• Takes care of infrastructure concerns, so that you can focus on making the most of your data.
• Removes boiler plate code, especially the tedious and error-prone kind.
• Supports streaming, without back-pressure. Large amounts of data can be managed in a timely and memory efficient manner.
• Light-weight use-case specific object graph mapping (OGM). Drivine is NOT an OGM in the traditinoal sense. The results are then mapped to/from a use-case specific model object.

---

Quick Start

Follow the Quick Start section in our User Guide or clone the sample app and use it as a starter template.

Start creating repositories like the one below.

@Injectable()
export class RouteRepository {
    constructor(
        @InjectPersistenceManager() readonly persistenceManager: PersistenceManager,
        @InjectCypher('@/traffic/routesBetween') readonly routesBetween: CypherStatement) {
    }

    @Transactional() // Has default Propagation.REQUIRED - partipicate in a current txn, or start one.
    public async findFastestBetween(start: string, destination: string): Promise<Route> {
        return this.persistenceManager.getOne(
            new QuerySpecification<Route>()
                .withStatement(this.routesBetween)
                .bind([start, destination])
                .limit(1)
                .transform(Route)
        );
    }
}

---

Example Applications

If you use Drivine and your code is public, feel free to make PR and add yourself to the list.

Quick Start Template

Clone the official sample app and start hacking.

Repository: github.com/liberation-data/drivine-inspiration

This sample is a basic starter template, with some tutorials. It has some endpoints for traffic routing, movies/films and other typical use-cases. Rather than start from scratch, choose the one that closely matches yours, and modify.

SlackMap

New version of slackmap.com is full rewrite with technology update. The OrientDB was replaced with Neo4j and we choose Drivine as the way to work with the database.

Check out how Drivine supports Neo4j in Full Stack JavaScript App using Angular + Nest + Nx Workspace Monorepo.

Repository: github.com/SlackMap/slackmap

• master branch - not released yet
• develop branch - has all the current work

---

Documentation

Best way to learn Drivine is with our User Guide.

---

Tutorials

New to graph databases? Read some tutorials.

• Rock & Roll Traffic Routing, with Neo4j
• Rock & Roll Traffic Routing, with Neo4j, Chapter Two

Have a tutorial you'd like to share? Get in touch with me.

---

Feedback

I'm not sure how to do [xyz]

If you can't find what you need in the Quick Start or User Guides, please post a question on StackOverflow, using the Drivine tag.

Interested in contributing?

Great! A contribution guide, along with detailed documentation will be published in the coming days.

I've found a bug, or have a feature request

Please raise a GitHub issue.

---

Have you seen the light?

Drivine is a non-profit, community driven project. We only ask that if you've found it useful to star us on Github or send a tweet mentioning us (@doctor_cerulean). If you've written a Drivine related blog or tutorial, or published a new Drivine-powered app, we'd certainly be happy to hear about that too.

Drivine is sponsored and led by Jasper Blues with contributions from around the world.

---

License

Copyright (c) 2022 Jasper Blues

Drivine is free software: you can redistribute it and/or modify it under the terms of the APACHE LICENSE, VERSION 2.0 as published by the Apache Software Foundation, either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the Apache Software License, Version 2.0 for more details. You should have received a copy of the Apache Software License along with this program. If not, see http://www.apache.org/licenses/.

drivine's Issues

Transaction decorator should support optional named connection parameter

PersistenceManager:default can't be resolved

PersistenceManager:default can't be resolved if you don't use @InjectPersistenceManager() before DrivineModule.withOptions()

It was working be course we were importing Repositories with @InjectPersistenceManager() decorator, but if you run DrivineModule.withOptions() before importing the Repository, the default provider will be not defined

@jasperblues I'm working on the PR with e2e tests to illustrate the issue, if you can check it out, for you it should be quicker to debug

[DUPLICATE HAS-WORKAROUND] .cypher files missing after transpilation

After the build step, there are no .cypher files in the dist directory. That is because of tsc behaviour to only include ts/js files. How did you manage to get it to work?

Cannot find module '/****/api/dist/user/repository/userById.cypher'
Require stack:

/****/api/node_modules/@liberation-data/drivine/DrivineInjectionDecorators.js

Support streaming writes, with and without periodic commit

This will allow, for example to pipe data from a postgres source into a Neo4j sink.

postgresRepo.listCompanies().asStream({transform: (it) => toNeo4j(it)).pipe(neo4jWriter); 

Use index.ts to export public API

It's common practice to use one index.ts file to re-export the public API of the libraries, so we can hide inner structure of the files from the outside world.

so instead of

import { Transactional } from '@liberation-data/drivine/transaction/Transactional';
import { PersistenceManager } from '@liberation-data/drivine/manager/PersistenceManager';

we will have this

import { Transactional, PersistenceManager   } from '@liberation-data/drivine';

Is there any reason Drivine is not doing this ?

Implement Drivine logger.

Pre-requisiste to split NestJS into add-on for #32

TransactionContextHolder should set a flag to enable @Transactional() decorators.

This means that unless transaction context is enabled transaction decorator will do nothing - prevent impact on unit tests, etc.

Add bookmarks support for Neo4j

RunWithDrivine config params should be flattened.

QuerySpecification post processors

Currently:

QuerySpecification supports map and transform and only in that order.

After:

new QuerySpecification(statement).transform(type).map(fn).reduce(fn)

• Post processors run in the order they're added
• Add reduce, filter, etc from Array.

example of using without NestJS

Provide sample app (AWS lambda?) for using without NestJS.

Working with relations.

👋 Hi there. I've been playing with Drivine as a candidate for integrating a NestJS app with Neo4j. The design strikes me as interesting, but I'm having trouble working out how relations work. I've used the Java OGM through Scala in the past, and this example strikes me as very intuitive. But I'm not sure how it would map to this architecture.

I fear I'm just missing something obvious. If not, though, I'm curious to know what you'd recommend.

Support Apache Gremlin with CYPHER

[AgensGraph] Can't bind array of strings to query parameter

This will work:

const createMetros = `UNWIND ['NYC, 'Cavite Island'] AS metro MERGE (m:Metro {name: metro})`
await this.persistenceManager.execute(new QuerySpecification<void>(createMetros));

This will not:

const createMetros = `UNWIND $metros AS metro MERGE (m:Metro {name: metro})`
await this.persistenceManager.execute(
    new QuerySpecification<void>(createMetros).bind({metros: ['NYC', 'Cavite Island'));

Graceful shutdown & connections

I found that we need to close all db connections manually sometimes. Great example is unit tests:

1. In scenario, when we setting up a postgres/agensgraph database before running tests, and then stopping it once all tests are finished, we need to close database connection before stopping the database. Otherwise an error arise: Connection terminated unexpectedly

2. even if we are not setting a database for tests, jest will warn Jest did not exit one second after the test run has completed.

In order to close connection, we need to do:
In AgensGraphConnectionProvider.ts
we need to run await this.pool.end()

I think we might add a method to ConnectionProvider interface to end all connections immediately and a helper static method to ConnectionProviderBuilder to close all connections in registry.

In terms of NestJS, I think should also hook on onApplicationShutdown and close all connections there.

Use `nests-style-filenames.ts` rather than `CamelCaseNames.ts`

Use nests-style-filenames.ts rather than CamelCaseNames.ts

Different batch size for cursors produces different results

Hello

We like the clean principles by which this library is being developed and have been using it with great joy so far.
I came across some strange behavior when using the cursor implementation though.
According to the sample:

async myRepositoryFunction(params: any): Promise<Cursor<any>> {
   ...
   const result = await this.persistenceManager.openCursor(
         new CursorSpecification<any>().withStatement(myQuery).bind(params).batchSize(X)
   );
   return result;
}

Consuming the result like this

for (await const t of timeline) {
      const [myEntities,...] = result;
}

I've started to experiment with different batch sizes X. In my setup I get back 544 result records, no matter what value I choose for the batch size. What is interesting however is that certain elements are missing in the result set, depending on what batch size I choose. I've only checked 3 elements definitely missing in the result set so far, but there are probably more. The fact that the size is always the same of course indicates that there can be duplicates.

To compare the varying results of different batch sizes, I take the non-cursor implementation (using QuerySpecification and .query) as my reference of the correct result.
Choosing a cursor batch size of 1 leads to the same (correct result), 100 does not, 500 does and 544 does as well (reminder that 544 is the total number of records). Not specifying the batch size will default to a value of 100 which as stated skips some values in the result set.

I don't expect this behavior to be wanted from Drivine's side. As far as I've checked using cursors adds a LIMIT X to the cypher query, which shouldn't change the logic of the query. So I'm a bit lost here.

I can try to come up with a repo to reproduce that issue if needed.

Cheers!

Dejan

Add sample of mapping graph structure using pattern comprehension.

Fully-fledged implementation of `Cursor<T>` for Neo4j.

Requires Neo4j support at driver level: neo4j/neo4j-javascript-driver#542

Otherwise current implementation is the best compromise.

Links to API docs on website are broken

Support named parameters.

Comes out-of-box on Neo4j. On Postgres/Agens these need to be faked - transform to ordered params.

Requires AgensGraph driver to be installed

Requires AgensGraph driver to be installed, even if only using Neo4j. And vice versa.

Use with Neo4j version <= 4.0.0

Hi. I'm trying to use drivine with neo4j 3.8.X and when I'm trying to access the database, I get the next error:

[ WARN ] [2020-10-29T09:27:07.094Z] [Neo4jConnection] Closing session with error: Error: Driver is connected to the database that does not support multiple databases. Please upgrade to neo4j 4.0.0 or later in order to use this functionality

Do you know if there is an option in the Drivine Options to disallow this functionality?

Thanks

Bind parameter on QuerySpecification should accept varargs as well as array

Drivine @Transactional() decorator breaks @UsePipes(new ValidationPipe)

Drivine @transactional() decorator breaks Nest ValidationPipe if used with @UsePies()

import { Controller, Post, Body, ValidationPipe, UsePipes } from '@nestjs/common';
import { Transactional } from '@liberation-data/drivine';
import { SpotSaveRequestDto, SpotSaveDto } from '@slackmap/api/spot/dto';
import { SpotRepository } from '@slackmap/api/spot/data';

@Controller()
export class SpotSaveController {

  constructor(private spotRepository: SpotRepository) {}

  @Post('spot/save')
  @Transactional() // <<< if you comment this out, validation will work
  @UsePipes(new ValidationPipe())
  async process(@Body() data: SpotSaveRequestDto): Promise<SpotSaveDto> {

    const spot = await this.spotRepository.create(data.spot)

    return {spot}
  }
}

It will work if you use it on the method param like:

  async process(@Body(new ValidationPipe()) data: SpotSaveRequestDto): Promise<SpotSaveDto> {

    const spot = await this.spotRepository.create(data.spot)

    return {spot}
  }

Allow standard SQL queries on AgensGraph

This can be used for example to pipe data from Postgres to graph db. Very simple to implement - just modify the method that maps parameters for a given platform.

[How to] : Use this with aws lambda

I am pretty new to graph db and node ... so might be silly question.
However would be great how I can use this with aws lambdas.
Do I need nestjs to work with drivine ?
Is it possible to publish a sample aws lambda with drivine ?

neo4j-driver required even for postgres/agensgraph users

I'm running app using Postgres database, and once dependencies moved to peer dependencies, my app won't start until I install neo4j-driver manually to my app.

Is it possible to import neo4j related modules, only if neo4j-driver installed?

Error: Cannot find module 'neo4j-driver' 
Require stack: 
- /node_modules/@liberation-data/drivine/mapper/Neo4jResultMapper.js 
- /node_modules/@liberation-data/drivine/connection/Neo4jConnectionProvider.js 
- /node_modules/@liberation-data/drivine/connection/ConnectionProviderBuilder.js 
- /node_modules/@liberation-data/drivine/connection/DatabaseRegistry.js

freshen dependencies

Some dependencies are stale and giving warnings.

Turn off logging while unit testing

Hi.
I'm running unit tests, I was unable to turn off logging.

Here is similar issue: nestjs/nest#3246

I found that for regular run we can do the following:

  const app = await NestFactory.create(AppModule, {
    logger: ['error', 'warn'],
  });

but for unit tests, we cannot do so.

Maybe it's worth adding a logger to option to DrivineModuleOptions ?

RedisGraph support

Add Caching Example

Why is it important to add @liberation-data/agensgraph is I am using neo4j

Remove hard import dependency from class-transformer

Currently, drivine lists class-transformer as peerDependencies. However, https://github.com/liberation-data/drivine/blob/master/src/query/QuerySpecification.ts#L1 uses a hard-import from class-transformer making class-transformer a required dependency.

Not working with Neo4j 4

https://twitter.com/marioammonteiro/status/1238248639962775555?s=20

today I have a spare time to try to figure out what is that error `Client network socket disconnected before secure TLS connection was established`
it happens when using neo4j 4.0.0 Ent, I try today 3.5.15 Ent and it works has expected
strange I think I do that```

Likely will be solved by updating driver. 

It should be possible to use map and transform together. Map goes first.

Relates to #41

NestJS GraphQL like for discovering Cypher files

https://docs.nestjs.com/graphql/quick-start#schema-first

Currently, new NestJS apps that are created by the CLI uses Webpack under the hood to build and run the application even in dev mode (off of dist folder). Hence, the current approach of using directory path for @InjectCypher() doesn't work unless users copy the .cypher files manually (or alter NestCLI webpack config).

Utilizing an approach like typePaths from nestjs/graphql might be a better solution for .cypher

Cypher and SQL injections load platform specific version, if exists.

Remove path alias style imports

I personally like these, however folks have been puzzled by how they work - can be a distraction for first time users.

Async Hooks leak memory when used with Jest

Not a problem in production, since we have one namespace for the lifecycle of the app, but excessive memory consumed for integration tests:

From @alexgarbarev:

My investigations regarding memory leak. (this is all with --runInBand --logHeapUsage)

1. All test suits (files) runs on same process (same PID), that means that real HEAP is shared across all test files.
   That means, if we leak something in test 1, it will remain there.

2. Though, we have same PID and hence HEAP, each test suite (file) ran in isolation (jest feature). That means if you store variable to global in file 1, you won't be able to access it in file 2.
   Practical consequence: we cannot store something to global or static variable in file 1, and then reuse it in file 2. I guess it requires modules independently.

3. By localizing problem, commeting bare-minimal code line by line, I found that DatabaseRegistry and TransactionContextHolder might leak, as they store shared instance. > > That instance might leak after end of test file if not tear it down. I have added simple tear down method, but it didn't help. See next.

4. Although we can tear down all our static variables at the end of test suite, cls-hooked is leaking. Once you create cls.createNamespace(namespaceName) it allocates a lot of space (like 10mb) and it persists forever. There is no API clean that. I have checked the implementation here: https://github.com/Jeff-Lewis/cls-hooked/blob/master/context.js and it looks like destory method doesn't do anything with created async_hook. It just zeroing reference to own object. I've seen in Chrome memory profiler that AsyncHook objects are created again and again without releasing. So we cannot fix that, and we cannot reuse same async hooks, as jest isolate each test suite.

So far, I found the only solution is to build own implementation of async hook. There is brand new API, called AsyncLocalStorage, which ships with node since v13.10.0: https://nodejs.org/api/async_hooks.html#async_hooks_class_asynclocalstorage
It looks pretty clear, but bare minimal. Requires some code in order to store more than "pseudo thread id"

Notes not created when testing with transaction rollback: true

If you try to create node and run your test with automatic rollback, the nodes are not created and there is no errors

RunWithDrivine({
    transaction:{rollback: true}
})

Just created PR with tests in HealthRepository.spec.ts to illustrate the problem

    it('should create new user', async () => {
        const data = {id: 2, name: 'Jasper'};
        // create user
        const user = await repo.create(data);
        expect(user).toMatchObject(data) // <--- node created OK
        // find created user
        const user2 = await repo.findById(data.id); // <--- but here the node is 404 Not Found
        expect(user2).toMatchObject(data)
  
      });

This was working with [email protected] commit 06cf4d5 Update Neo4j driver to support v4

and the bug was introduced with commit Use neo4j connection scheme, rather than bolt. or more likely with Use reactive neo4j api.

@jasperblues can you help with the sollution ? Just wasted 2 days to find out where was the problem and don't have energy to look deep down into it

Transform should accept an array for multi-column results.

Multi database support

Support distributed transactions across multiple registered databases.

Allow databases to be registered on bootstrap or dynamically.

Add hljs to the code samples on website

AgensGraphResultMapper is missing mapper to extract edge (relationship) properties

Test query: MATCH (people:Person)-[relatedTo]-(:Movie {title: "Cloud Atlas"}) RETURN people.name, Type(relatedTo), relatedTo

dgraph support

Cannot find module '@liberation-data/agensgraph/lib'

lets make @liberation-data/agensgraph optional if you use neo4j, now i have error Cannot find module '@liberation-data/agensgraph/lib' from '../../../../node_modules/@liberation-data/drivine/connection/AgensGraphConnectionProvider.js' and I'm forced to install it

Errors thrown from assert() are not readable in tests

Errors from assert() function are not helpful in stack trace when testing with jest

    async getOne<T>(spec: QuerySpecification<T>): Promise<T> {
        const results = await this.persistenceManager.query(spec);
        assert(results.length === 1, `Expected exactly one result`);
        return results[0];
    }

In tests doesn't say what happened

FAIL  libs/api/db/src/lib/user/user.repository.spec.ts
  ● UserRepository › .update() › should update user

    assert(received)

    Expected value to be equal to:
      true
    Received:
      false

    Message:
      Expected exactly one result

      71 |       const data = await userFixture.createFakeUser();
      72 |       const name = 'new name'
    > 73 |       const updated = await userRepository.update(data.rid, {name});
         |                       ^
      74 | 
      75 |       const res: UserEntity = {
      76 |         rid: expect.any(String),

      at FinderOperations.getOne (../../../node_modules/@liberation-data/src/manager/FinderOperations.ts:11:9)
      at TransactionalPersistenceManager.getOne (../../../node_modules/@liberation-data/src/manager/TransactionalPersistenceManager.ts:23:16)        
      at runInTransaction (../../../node_modules/@liberation-data/src/transaction/Transactional.ts:39:24)
      at src/lib/user/user.repository.spec.ts:73:23

We should always use custom errors like we do here:

    async maybeGetOne<T>(spec: QuerySpecification<T>): Promise<T | undefined> {
        const results = await this.persistenceManager.query(spec);
        if (results.length === 0) {
            return undefined;
        } else if (results.length === 1) {
            return results[0];
        } else {
            throw new DrivineError(`Expected one result, received ${results.length}.`);
        }
    }

CI Builds

How to return IDs and Labels?

Consider the following query:

const movies = await this.persistenceManager.query<Movie>(
    new QuerySpecification<Movie>(`
        MATCH (m :Movie)
        RETURN m
    `)
);

There seems to be no way to get the ID or list of labels for each of the Movie nodes that the query will return. Instead, only the properties on the node can be retrieved. The following snippet of code is the relevant part of drivine that does this mapping:

export class Neo4jResultMapper extends ResultMapper {

    ...

    toNative(val: any): any {
        if (val == undefined) {
            return val;
        }
        if (val instanceof neo4j.types.Node) {
            return this.toNative(val.properties);
        }
        if (val instanceof neo4j.types.Relationship) {
            return this.toNative(val.properties);
        }
        if (val instanceof neo4j.types.Point) {
            return val;
        }
        if (neo4j.isInt(val)) {
            return this.toNumberOrThrow(<Integer>val);
        }
        if (Array.isArray(val)) {
            return val.map((a) => this.toNative(a));
        }
        if (this.isRecord(val)) {
            return this.toNative(this.recordToNative(val));
        }
        if (typeof val === 'object') {
            return this.mapObj(this.toNative.bind(this), val);
        }
        return val;
    }

    ...

}

Perhaps it would be better to return the whole neo4j.types.Node instead of just the properties? If a user just wanted the properties without the Id and Labels, they could always just change the query to

const movies = await this.persistenceManager.query<Movie>(
    new QuerySpecification<Movie>(`
        MATCH (m :Movie)
        RETURN PROPERTIES(m)
    `)
);