liveforeverx/dlex

Dlex is a gRPC based client for the Dgraph database.

Dlex

Dlex is a gRPC based client for the Dgraph database in Elixir. It uses the DBConnection behaviour to support transactions and connection pooling.

Small, efficient codebase. Aims for a full Dgraph support. Supports transactions (starting from Dgraph version: 1.0.9), delete mutations and low-level parameterized queries. DSL is planned.

Now supports the new dgraph 1.1.x Type System.

Installation

If available in Hex, the package can be installed by adding dlex to your list of dependencies in mix.exs:

Preferred and more performant option is to use grpc:

def deps do
  [
    {:jason, "~> 1.0"},
    {:dlex, "~> 0.5.0"}
  ]
end

http transport:

def deps do
  [
    {:jason, "~> 1.0"},
    {:castore, "~> 0.1.0", optional: true},
    {:mint, github: "ericmj/mint", branch: "master"},
    {:dlex, "~> 0.5.0"}
  ]
end

Usage examples

# try to connect to `localhost:9080` by default
{:ok, conn} = Dlex.start_link(name: :example)

# clear any data in the graph
Dlex.alter!(conn, %{drop_all: true})

# add a term index on then `name` predicate
{:ok, _} = Dlex.alter(conn, "name: string @index(term) .")

# add nodes, returning the uids in the response
mut = %{
  "name" => "Alice",
  "friends" => [%{"name" => "Betty"}, %{"name" => "Mark"}]
}
{:ok, %{json: %{"uid" => uid}}} = Dlex.mutate(conn, mut, return_json: true)

# use the nquad format for mutations instead if preferred
Dlex.mutate(conn, ~s|_:foo <name> "Bar" .|)

# basic query that shows Betty
by_name = "query by_name($name: string) {by_name(func: eq(name, $name)) {uid expand(_all_)}}"
Dlex.query(conn, by_name, %{"$name" => "Betty"})

# delete the Alice node
Dlex.delete(conn, %{"uid" => uid})

Alter schema

Modification of schema supported with string and map form (which is returned by query_schema):

Dlex.alter(conn, "name: string @index(term, fulltext, trigram) @lang .")

# equivalent map form
Dlex.alter(conn, [
  %{
    "predicate" => "name",
    "type" => "string",
    "index" => true,
    "lang" => true,
    "tokenizer" => ["term", "fulltext", "trigram"]
  }
])

Developers guide

Running tests

1. Install dependencies mix deps.get
2. Start the local dgraph server (requires Docker) ./start-server.sh This starts a local server bound to ports 9090 (GRPC) and 8090 (HTTP)
3. Run mix test

NOTE: You may stop the server using ./stop-server.sh

Updating GRPC stubs based on api.proto

Install development dependencies

1. Install protoc(cpp) here or brew install protobuf on MacOS.
2. Install protoc plugin protoc-gen-elixir for Elixir . NOTE: You have to make sure protoc-gen-elixir(this name is important) is in your PATH.

mix escript.install hex protobuf

Generate Elixir code based on api.proto

3. Generate Elixir code using protoc

protoc --elixir_out=plugins=grpc:. lib/api.proto

4. Files lib/api.pb.ex will be generated

5. Rename lib/api.pb.ex to lib/dlex/api.ex and add alias Dlex.Api to be compliant with Elixir naming

Credits

Inspired by exdgraph, but as I saw too many parts for changes or parts, which I would like to have completely different, so that it was easier to start from scratch with these goals: small codebase, small natural abstraction, efficient, less opinionated, less dependencies.

So you can choose freely which pool implementation to use (poolboy or db_connection intern pool implementation) or which JSON adapter to use. Fewer dependencies.

It seems for me more natural to have API names more or less matching actual query names.

For example Dlex.mutate() instead of ExDgraph.set_map for JSON-based mutations. Actually, Dlex.mutate infers the type (JSON or nquads) from data passed to a function.

License

Copyright 2018 Dmitry Russ

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

dlex's Issues

Usage across app

Thanks for releasing this library!

Just looking for a little clarification, and forgive me because I've been using Elixir for ~6 months.

Is there a way to Dlex.start_link in one location and then get the pid anywhere it's needed through the app? Or is the recommended approach to use Dlex.start_link in each function where queries/mutation are needed?

For example, I tried doing this in my application.ex:

def start(_type, _args) do
    children = [
      # ...
      {Dlex, [name: :dlex]},
    ]

And then doing
{:ok, result} = Dlex.query(:dlex, query) in other modules, but that produces a bunch of Run time errors.

I did see that the tests do something to this effect by using setup and passing in the pid/(conn) to each test, but I'm wanting know if this can be done for the whole application across multiple contexts.

Thanks again!

Compilation error in file lib/dlex/api.ex

I'm trying to integrate dlex to a new fresh Phoenix project.

These are my deps:
defp deps do [ {:phoenix, "~> 1.5.4"}, {:phoenix_live_view, "~> 0.13.0"}, {:floki, ">= 0.0.0", only: :test}, {:phoenix_html, "~> 2.11"}, {:phoenix_live_reload, "~> 1.2", only: :dev}, {:phoenix_live_dashboard, "~> 0.2"}, {:telemetry_metrics, "~> 0.4"}, {:telemetry_poller, "~> 0.4"}, {:gettext, "~> 0.11"}, {:jason, "~> 1.0"}, {:plug_cowboy, "~> 2.0"}, {:cowlib, "~> 2.9.1", override: true}, {:dlex, "~> 0.1.0"}, ] end

I can do mix deps.get without any problem but when I try to compile the project it shows this error:

== Compilation error in file lib/dlex/api.ex == ** (UndefinedFunctionError) function Dlex.Api.LinRead.Sequencing.key/1 is undefined (module Dlex.Api.LinRead.Sequencing is not available) Dlex.Api.LinRead.Sequencing.key(0) lib/protobuf/dsl.ex:33: anonymous fn/2 in Protobuf.DSL."MACRO-__before_compile__"/2 (elixir 1.10.1) lib/enum.ex:2111: Enum."-reduce/3-lists^foldl/2-0-"/3 expanding macro: Protobuf.DSL.__before_compile__/1 lib/dlex/api.ex:192: Dlex.Api.LinRead (module) could not compile dependency :dlex, "mix compile" failed. You can recompile this dependency with "mix deps.compile dlex", update it with "mix deps.update dlex" or clean it with "mix deps.clean dlex"

I have to say that without {:dlex, "~> 0.1.0"} declared on my mix.ex file everyhting works fine. The problem is when I add dlex to my deps

Node field type [uid] doesn't work through Repo

Maybe I'm not understanding how this is supposed to work, but this doesn't work

test "setting uid fields" do
  user = %User{name: "George", age: 52}
  assert {:ok, %User{uid: uid}} = TestRepo.set(user)

  friend = %User{name: "David", age: 48, friends: [uid]}
  assert {:ok, %User{uid: new_uid}} = TestRepo.set(friend)
end

test "setting uid fields in changeset" do
  changeset = Ecto.Changeset.cast(%User{}, %{name: "George", age: 52}, [:name, :age])
  assert {:ok, %User{uid: uid}} = TestRepo.set(changeset)


  friendset = Ecto.Changeset.cast(%User{}, %{name: "David", age: 48, friends: [uid]}, [:name, :age, :friends])
  assert {:ok, %User{uid: new_uid}} = TestRepo.set(friendset)
end

Errors:

1. test schema operations setting uid fields (Dlex.RepoTest)
   test/dlex/repo_test.exs:39
   match (=) failed
   code: assert {:ok, %User{uid: new_uid}} = TestRepo.set(friend)
   left: {:ok, %Dlex.User{uid: new_uid}}
   right: {:error, %Dlex.Error{exception: true, action: :execute, reason: %GRPC.RPCError{message: "Input for predicate "user.friends" of type uid is scalar. Edge: entity:11063 attr:"user.friends" value:"0x2b36" value_type:STRING ", status: 2}}}
   stacktrace:
   test/dlex/repo_test.exs:44: (test)

2. test schema operations setting uid fields (Dlex.RepoTest)
   test/dlex/repo_test.exs:39
   match (=) failed
   code: assert {:ok, %User{uid: new_uid}} = TestRepo.set(friend)
   left: {:ok, %Dlex.User{uid: new_uid}}
   right: {:error, %Dlex.Error{exception: true, action: :execute, reason: %GRPC.RPCError{message: "Input for predicate "user.friends" of type uid is scalar. Edge: entity:11092 attr:"user.friends" value:"0x2b53" value_type:STRING ", status: 2}}}
   stacktrace:
   test/dlex/repo_test.exs:44: (test)

Question: Database sandbox for tests

One problem I've had while working with Dgraph, is that Dgraph does not have a test sandbox like Ecto has for Postgres. Since Dgraph does not offer multiple databases in a single instance, I'm wondering how other developers are solving this problem.

To illustrate, when running tests: The test may create new data and fetch old data:

1. First run: Create node with external_id: "1", returns uid: 0x1, external_id: "1"
2. First run: Fetch node with external_id: "1", returns uid: 0x1, external_id: "1"
3. Second run: Create node with external_id: "1", returns uid: 0x2, external_id: "1"
4. Second run: Fetch node with external_id: "1", returns nodes with uid: 0x1, external_id: "1" and uid: 0x2, external_id: "1"

A workaround I that use, is to add first: 1 and sometimes orderasc: created_on to queries. This works, and makes tests pass using the new data; but adding code simply to make tests work is obviously bad practice.

How are others solving this?

Conditional upsert support on mutation?

Hi!

How can I use the conditional upsert functionality, similar to this https://github.com/dgraph-io/pydgraph#running-a-conditional-upsert

I can contribute if you give me some guidelines.

Request for supporting associations and resolvers

Hi,

This is a request for supporting associations and resolvers. Something like following:

defmodule Person do
  use Dlex.Node
  @derive {Phoenix.Param, key: :uid}

  import Ecto.Changeset

  schema "persons" do
    field :name,      :string
    field :age,       :integer
    field :works_at,  :uid, model: Company
    field :company_count, :integer, virtual: true, resolve: "count(works_at)"
  end

  ...

Thanks,
Gorav

Unknown method Mutate for api.Dgraph

Hi!

I'm trying to use this library on my local DGraph install, but I'm getting some errors. I replicated the TestRepo structure you have in the tests, and I also tried using the methods in the readme (without any repo) and I get the same error with both methods when trying to add data to my DB:

{:ok, pid} = MyApp.TestRepo.start_link
user = %Myapp.User{name: "Alice", age: 25}
MyApp.TestRepo.set(user)
{:error,
 %Dlex.Error{
   action: :execute,
   reason: %GRPC.RPCError{
     message: "unknown method Mutate for service api.Dgraph",
     status: 12
   }
 }
}

Any idea of what might be happening here?

Thanks!

Edit: I'm using version 0.2.1 of this library.

Repo.set creates new node rather than updating existing node

Expected behaviour: Existing node should be updated

Actual behaviour: New node is created every time

version: 0.5.1

Same implementation was working fine with earlier version.

Implementation:

file lib/lms/courses/course.ex contains:

defmodule Lms.Courses.Course do
  use Dlex.Node
  @derive {Phoenix.Param, key: :uid}

  import Ecto.Changeset

  schema "courses" do
    field(:description, :string)
    field(:duration_allowed, :integer)
    field(:duration_units, :integer, default: 3)
    field(:position, :integer)
    field(:status, :integer)
    field(:title, :string, index: ["term"])
    field(:guidelines, :string)
    field(:guidelines_title, :string)
    field(:has_learning_path, :uid)

    field(:created_at, :datetime)
    field(:updated_at, :datetime)
  end

  @doc false
  def changeset(course, attrs) do
    course
    |> cast(attrs, [
      :title,
      :description,
      :duration_allowed,
      :duration_units,
      :status,
      :position,
      :guidelines,
      :guidelines_title
    ])
  end
end

and lib/lms/courses.ex contains:

def update_course(%Course{} = course, attrs) do
    course
    |> Course.changeset(attrs)
    |> Repo.set()
  end

Following creates a new node rather than updating the existing node:

id = "0x7533" # its a valid id and returns Course struct
course = Repo.get!(id)
course_params = [
  attrs: %{
    "description" => "",
    "duration_allowed" => "",
    "guidelines" => "",
    "guidelines_title" => "",
    "position" => "",
    "status" => "1",
    "title" => "updated title"
  }
]
Lms.Courses.update_course(course, course_params)

Course.changeset(attrs) produces:

changeset: #Ecto.Changeset<
  action: nil,
  changes: %{status: 1, title: "updated title"},
  errors: [],
  data: #Lms.Courses.Course<>,
  valid?: true
>

and data key contains:

%Lms.Courses.Course{
  created_at: nil,
  description: nil,
  duration_allowed: nil,
  duration_units: 3,
  guidelines: nil,
  guidelines_title: nil,
  has_learning_path: nil,
  position: nil,
  status: 1,
  title: "now we got it",
  uid: "0x7533",
  updated_at: nil
}

Tried replacing Repo.set with Repo.mutate but same issue persists.

Documentation and help with i18n

Hi @liveforeverx,

Thank you for the fantastic library and amazing implementation of Dlex.Node. I am working on developing a LMS prototype - my first attempt at using a Graph database. I had to dig into Dlex code to understand the implementation. I intend to add documentation and may be a sample repo for anybody else wanting to use the library. I have done the following:

• Implemented a Grepo module similar to Repo.ex for making calls to Dgraph. I may be using postgres in the project as well and hence, retained Repo for the purpose.
• added Grepo to supervision tree.
• implemented a the model (Course.ex) using Dlex.Node and the context (Courses.ex).
• got CRUD action to work with a html view.
• i18n feature in model - this is where I need help at the moment.
• define associations in the models - needs update in Dlex for supporting [:uid] added in Dgraph v 1.1
• define unique constraints on fields.

defmodule Lms.Courses.Course do
  use Dlex.Node
  @derive {Phoenix.Param, key: :uid}

  import Ecto.Changeset

  schema "courses" do
    field(:title, :string, index: ["term"])
    field(:description, :string)
    field(:duration_allowed, :integer)
   ...

how do I use title@en or other locales in this context?

any pointers to the pending items would be great help.

Best,
Gorav

Create schema via GraphQL

I can see how to create schema via alter using N-Quad notation, but not how to do it w/ GraphQL notation as outlined here:

https://dgraph.io/docs/graphql/admin/

Happy to help upgrade the docs here if needed!

set() and mutate() with Ecto Changesets results in error

I get the error no function clause matching in MyApp.Post.__schema__/2 when using Repo.set() with an Ecto Changeset.

Steps to reproduce:

1. Create changeset: changeset = Post.create_changeset(Post{}, params).
2. Insert changeset: Repo.set(changeset).
3. Error no function clause matching in MyApp.Post.__schema__/2 is received.
   Dlex supports inserting changesets via
   def mutate(conn, %{__struct__: Ecto.Changeset, changes: changes, data: %type{uid: uid}}, opts) do data = struct(type, Map.put(changes, :uid, uid)) mutate(conn, data, opts) end

I made a pull request in #70 that solves this issue.

Repo get/set doesn't handle :datetime fields using DateTime structs

See my Pull Request to see a test that demonstrates the problem.

I modified the schema definition thus

defmodule Dlex.User do
  use Dlex.Node

  schema "user" do
    field :name, :string, index: ["term"]
    field :age, :integer
    field :friends, :uid
    field :location, Dlex.Geo
    field :modified, :datetime
    field :cache, :any, virtual: true
  end
end

now = DateTime.utc_now()
changeset = Ecto.Changeset.cast(%User{}, %{name: "TimeTraveler", age: 20, modified: now},[:name, :age, :modified])
assert {:ok, %{uid: uid}} = TestRepo.set(changeset)

assert {:ok, %User{name: "TimeTraveler", age: 20, modified: now}} = TestRepo.get(uid)

No longer maintained? Happy to jump in 👋

Hi @liveforeverx! I've been jumping in with Dlex, and noticed you're not as active anymore.

If you could give me a quick overview of how everything works (Confused about Adapters, Protocols, etc), I'm happy to take over. Lmk 😃

.get! throws error with master branch

Repo.get!("0x7") <- this uid exists in db, connection info is passed through repo.ex

throws following error:
** (ArgumentError) raise/1 and reraise/2 expect a module name, string or exception as the first argument, got: {:untyped, %{"courses.description" => "testing the course", "courses.duration_units" => 3, "courses.title" => "Refresher Course", "dgraph.type" => ["type.courses"], "uid" => "0x7"}}

It works fine with this commit

Upsert returning uid

I'm trying to achieve an upsert, also modifying the node, and always returning the uid. The upsert criteria is just an eq lookup on a field, no logic involved. What would be the best way to achive this?

The dgraph docs mention two ways to do upserts:

1. Using a transaction - this would naturally cause another request round trip but that's ok
2. Using an upsert block - this doesn't return the uid if the node already existed, at least in Ratel, but I could query for the uid right after the upsert

I tried both approaches with dlex, but feel that the documentation plus looking at the code doesn't give me enough knowledge to achieve this.

Here's what I tried:

id = "user1"
statement =
  ~s"""
  upsert {
    query {
      v as var(func: eq(id, "#{id}"))
    }

    mutation {
      set {
        uid(v) <id> "#{id}" .
      }
    }
  }
  """
Dlex.mutate!(conn, statement)

grpc gave me a parsing error:

while lexing upsert { at line 1 column 0: Unexpected char 'p' when parsing uid keyword

Then, I tried it with HTTP, which gave me a different parsing error:

at line 1 column 7: Invalid character '{' inside mutation text

I also tried using a transaction, but got:

StartTs cannot be zero while committing a transaction

I'm happy to help with an PR to extend the documentation once this is solved :)

Thanks for your great work! Cheers

Ping failure

I get an annoying ** (CaseClauseError) no case clause matching: {:error, {:stream_error, :internal_error, :"Stream reset by server."}} periodically when I'm on shell. Is Dgraph rejecting pings for some reason?

Question about "return_json"

Hi! I'm seeing that this correctly inserts nodes into the db:

Dlex.mutate!(conn, %{set: mutation})

while this returns the correct json, but doesn't insert the nodes:

Dlex.mutate!(conn, %{set: mutation}, return_json: true)

any reason why this might be?