dmgolembiowski/edgemorph

EdgeDB manipulator of relational, polymorphic hierarchies

⚠️ WIP ⚠️

Notice: This project is under development and is not appropriate for production projects.

Continue Writing EdgeQL as Normal

abstract type Named {
    required property name -> str;
}

abstract type HasAddress {
    property address -> str;
}

type User extending Named, HasAddress {
    # define some user-specific properties and a link
    multi link friends -> User;

    # define an index for User based on name
    index on (__subject__.name);
}

but now — your projects can leverage edm to manage EdgeDB module schemas captured in a edgemorph.toml, where database modules are configured at the project-level.

[edgemorph]
project_root    = "edgedb_app"
mod_directories = ["/edb_modules"]

[edgemorph.codegen]
schema_name = "Edgemorph"

[edgemorph.codegen.rust]
enabled = "true"

[edgemorph.codegen.rust.modules]
    [edgemorph.codegen.rust.modules.edgedb_app]
    source = "/edb_modules/edgedb_app.esdl"
    output = "/src/lib/edm_edgedb_app.rs"

[edgemorph.codegen.python]
enabled = "true"

[edgemorph.codegen.python.modules]
    [edgemorph.codegen.python.modules.edgedb_app]
    source = "/edb_modules/edgedb_app.esdl"
    output = "/edgedb_app/edm_edgedb_app.py"

[edgedb]
[edgedb.databases]
[edgedb.databases.primary]
name = ""
dsn = ""

[edgedb.databases.primary.modules]
edgedb_app = "edb_modules/edgedb_app.esdl"

Unlike traditional object-relational mappers, Edgemorph requires users to write database-level code. Using EdgeDB's rich query language, Edgemorph combines the strictly typed qualities of EdgeDB with a library-factory written in Rust. This unconventional strategy allows users to compile entirely custom bytecode libraries on a per-project basis, but continue to program in the stylings of a typical ORM.

For instance, if we have an EdgeDB module within the user.esdl file, then executing edm compile -f user.esdl would compile the user module to return both a dynamic library, edm_user.so, as well as a native code file in Rust or Python for reaching it. Currently, I am only planning to support Rust and Python API outputs, however I would like JavaScript, either via wasm32 architecture or pure JS, to join the adventure as well.

Here are examples of what the compiler generates in Rust and Python given the EdgeQL above.

Rust API Output

// user.rs

use ::edm_user::{NamedType, UserType, HasAddressType};
use edgemorph::*;

#[derive(NamedType)]
#[type(abstract="true")]
struct Named {
    #[property("str", required="true")]
    name: Property<String>
}

#[derive(HasAddressType)]
#[type(abstract="true")]
struct HasAddress {
    #[property("str")]
    address: Property<String>
}

#[derive(UserType)]
#[type(extending=("Named", "HasAddress")]
#[index("name")]
struct User {
    #[link("User"), multi="true")]
    friends: Link<User>
}

Python API Output

# user.py

from edgemorph import ( edgetype, property, link, multi )
from .edm_user import ( NamedType, HasAddressType, UserType )

@edgetype(abstract=True, edb=NamedType)
class Named:
    name: property[str]

@edgetype(abstract=True, edb=HasAddressType)
class HasAddress:
    address: property[str]

@edgetype(extending=(Named, HasAddress), edb=UserType)
class User:
    friends: multi[ link[__qualname__] ]
    index:   {
        "name": lambda title : "User name index"
    }

Development / Build Instructions for edm

Note: Pre-requisite dependencies include having Python 3.8+, Poetry, Git, and Rust nightly installed.

# Have Python 3.8+, Poetry, and Rust nightly installed
git clone https://github.com/dmgolembiowski/edgemorph.git
cd edgemorph
git submodule init
git submodule update --recursive
cd edm
poetry shell
poetry install
cd bootstrap/edgedb-python && python -m pip install -e .
cd ../../bootstrap/edgedb && python -m pip install -v -e .

# And viola! You can now run `edm` based commands.
edm init app

Roadmap (incomplete)

---

EDM Features

• edm init
• edm make
• simple offline SDL syntax checker
• hidden AST dump to EDB modules file
• "makeability" protected by project-level configuration file edgemorph.toml
• single and (concurrent) multi-file lex checking
• edm add
• edm make install
• edm test

In the future, I would like to see edm support multi-language target compilation so that changes to the native programming language code can result be retrofitted onto the original shema with either DDL modifications or 1-to-1 SDL modifications.

edgemorph's Issues

`edm make <something valid>` raises critical warning

Replication Steps:

1. edm init abc
2. cd abc/edb_modules
3. Add these contents to abc.esdl

module abc {
    abstract type Named {
        required property name -> str;
    }

    abstract type HasAddress {
        property address -> str;
    }

    type User extending Named, HasAddress {
        # define some user-specific properties and a link
        multi link friends -> User;

        # define an index for User based on name
        index on (__subject__.name);
    }
}

3. edm make abc.esdl

Output:

CRITICAL WARNING: One or more EdgeDB module files are missing from the local filesystem!
These include:
 + abc/edb_modules/abc.esdl
If this does not seem correct, please double check `edgemorph.toml` and remove any unused entries before running this command again.

This is strange since running edm make | edm make * (which requires a great deal more of coding and careful checking should be more error prone) actually gets it right. For example:

Additional Information:

1. edm init abc
2. cd abc

3. (Modify abc.esdl like before)

4. edm make
   returns the expected output:

Compiling your EdgeDB modules....
Compiling /home/david/code/edgemorph/edm/abc/edb_modules/abc.esdl....
Schema <0x7f6ca22ba6a0> (
    declarations = [
        ModuleDeclaration <0x7f6ca22ba310> (
            name         = ObjectRef<0x7f6ca22ba4f0>(name='abc'),
            declarations = [
                CreateObjectType <0x7f6ca216ab80> (
                    is_abstract = True,
                    name        = ObjectRef<0x7f6ca216a640>(name='Named'),
                    commands    = [
                        CreateConcreteProperty <0x7f6ca216ad60> (
                            is_required = True,
                            target      = TypeName <0x7f6ca2176280> (
                                maintype = ObjectRef<0x7f6ca216a9a0>(name='str')
                            ),
                            name        = ObjectRef<0x7f6ca216ae20>(name='name')
                        )
                    ]
                ),
                CreateObjectType <0x7f6ca216ad30> (
                    is_abstract = True,
                    name        = ObjectRef<0x7f6ca216a820>(name='HasAddress'),
                    commands    = [
                        CreateConcreteProperty <0x7f6ca2176580> (
                            target = TypeName <0x7f6ca2176a30> (
                                maintype = ObjectRef<0x7f6ca21760a0>(name='str')
                            ),
                            name   = ObjectRef<0x7f6ca2176610>(name='address')
                        )
                    ]
                ),
                CreateObjectType <0x7f6ca2316730> (
                    name     = ObjectRef<0x7f6ca221bc40>(name='User'),
                    commands = [
                        CreateConcreteLink <0x7f6ca2063250> (
                            target      = TypeName <0x7f6ca2063700> (
                                maintype = ObjectRef<0x7f6ca216a670>(name='User')
                            ),
                            cardinality = <SchemaCardinality.Many: 'Many'>,
                            name        = ObjectRef<0x7f6ca20632e0>(name='friends')
                        ),
                        CreateIndex <0x7f6ca2063190> (
                            expr = Path <0x7f6ca2063ac0> (
                                steps = [
                                    Subject<0x7f6ca2063a60>(),
                                    Ptr <0x7f6ca2063880> (
                                        ptr       = ObjectRef<0x7f6ca2063d60>(name='name'),
                                        direction = <PointerDirection.Outbound: '>'>
                                    )
                                ]
                            ),
                            name = ObjectRef<0x7f6ca2093190>(name='idx')
                        )
                    ],
                    bases    = [
                        TypeName <0x7f6ca21767f0> (
                            maintype = ObjectRef<0x7f6ca216aca0>(name='Named')
                        ),
                        TypeName <0x7f6c9fad2f70> (
                            maintype = ObjectRef<0x7f6ca2176c70>(name='HasAddress')
                        )
                    ]
                )
            ]
        )
    ]
)

The error likes occurs at or around edm/edm line 190.

ModuleNotFoundError: No module named 'edb._edgeql_rust'

Summary:

Currently the poetry build fails since rusty imports are not built by default. I need to fix this.

Traceback:

Traceback (most recent call last):
  File "/home/edgemorph/.cache/pypoetry/virtualenvs/edm-N0-moBkx-py3.8/bin/edm
", line 2, in <module>
    from edm.__main__ import cli_main
  File "/home/edgemorph/edgemorph/edm/edm/__main__.py", line 22, in <module>
    from edb.edgeql import parser as qlparser
  File "/home/edgemorph/edgemorph/edm/bootstrap/edgedb/edb/edgeql/__init__.py"
, line 22, in <module>
    from . import ast  # NOQA
  File "/home/edgemorph/edgemorph/edm/bootstrap/edgedb/edb/edgeql/ast.py", lin
e 27, in <module>
    from edb.common import ast, parsing
  File "/home/edgemorph/edgemorph/edm/bootstrap/edgedb/edb/common/parsing.py",
 line 37, in <module>
    from edb._edgeql_rust import TokenizerError
ModuleNotFoundError: No module named 'edb._edgeql_rust'

Python ORM/FRM Design Standards

@nsidnev, I want to entrust the Python user experience to you, and relinquish all decision-making on the Python API to you. I'm too distracted focusing on the Rust and Python API's, and it's becoming too much for me to think about on my own.

If you are feeling up to it, creating a Python-equivalent to the wiki's Rust document with your own ideas and opinions would be greatly appreciated.

P.S. The codeblocks script is an easy way to generate code in an HTML table, but not required by any means.

Reconsider Python API (post PEP-614)

With the stable release of Python 3.9, it's helpful to discuss supplemental improvements to the edgemorph (framework? library?) Python API. Because 3.9 allows more permissive decorator grammar with PEP 614, I'm excited about ways in which SDL functions can be expressed on modules.

Truly, this blows the gates wide open.
With this enhancement, it is possible to use decorators (both for the type implementation classes and the SDL functions) in an entirely new, more expressive way.

buttons = [QPushButton(f'Button {i}') for i in range(10)]

# Do stuff with the list of buttons...

@buttons[0].clicked.connect
def spam():
    ...

@buttons[1].clicked.connect
def eggs():
    ...

# Do stuff with the list of buttons...

From the excerpt, we can infer that it's possible to call __getitem__-derived methods -- leveraging new patterns for @index_on.<some field>, @constraint.(abs=True), etc.

CPython implementation details are available at: python/cpython#18570

Setup CI + codestyle tools

While working on #6, I ran into several issues and to solve them I want to suggest the following:

1. Set up GitHub Actions for all parts of this project (Rust framework, Python framework, EDM CLI) to check if all PRs and branches are in the correct state.
2. Add autoformatters (now) + linters (after a while) that will handle common errors (for example, extra spaces after lines).

Here are the checklists:

Python:

• Setup black for the Python code style.
• Setup isort for correct imports order.
• Setup autoflake to remove all unused variables and imports.
• Setup flake8 to handle common errors. Maybe extend it with wemake-python-styleguide (optional).
• Setup GitHub Actions for tests and style checking.

Rust:

• Setup Clippy to handle common errors.
• Setup GitHub Actions for tests and style checking (at least using rustfmt).

This isn't a complete or required list, but these are the tools I've been using for a long time, and they help me maintain a good codebase in my personal and work projects.

@dmgolembiowski. What do you think about this?

Compile-time & Runtime dynamic query generation

Similar to the problem case at tql — implement a strategy for handling intermediate mutations on EdgeDB.

The key here is that we either need to hack a Postgres trigger, or we need to invent an algorithm with source analysis.

Deserialization grammar misses `tree_node_child`

TL;DR

edm/src/common/deserialize_ast.pest isn't capturing tree_node_child, and it needs to.

Overview

The simplest of AST deserialization examples yields a surprisingly large node tree. I've observed that EdgeDB's SDL parser will generate a schema in case one is not supplied (a good thing), and this makes it tougher to catch a simple inductive hypothesis. I spent the better part of three hours on prettier.io and pest.rs to find out where the problem lies, but still no luck yet. Hopefully tonight I'll work out where the grammatical flaw lies.

SDL module (input)

module this_module_name {
    type User {
        required property name -> str;
    }
}

The true SDL abstract syntax tree (output) is too unwieldy to put on the issue, so I prepared a link to the Gist.
The full version is also available at prettier.io/playground. Here's a snippet of the cleaner output.

<TreeNode 
	id=140142061683664, name=Schema, children=[
      <TreeNodeChild 
        id=None, 
        label='declarations', 
        node=<List id=140141820332544, 
               items=CheckedList[Markup]([
               		<TreeNode 
                             id=140142061684912, 
                             name='ModuleDeclaration', 
                              children=CheckedList[TreeNodeChild]([
                                       <TreeNodeChild 
                                               id=None, ...

and the complexity only increases from there.

1) schema or root -> ...

schema = ${ SOI ~
    "<" ~ __ ~ "TreeNode" ~ __
    ~ "id" ~ "=" ~ id ~ __
    ~ "name" ~ "=" ~ __ ~ "Schema" ~ __
    ~ "children" ~ __ ~ "=" ~ __ ~"[" ~ __ ~ module ~ __ ~ "]" ~ __ ~ ">" ~ EOI
}

* side note*:

where:

• SOI and EOI mean start of input and end of input, respectively;
• X ~ Y means X followed by Y;
• __ is a blanket catch-all for undesired tokens in the compound atomic groups
  • (things like thing_x = ${ thing_y ~ __ ~ thing_z }) where __ is junk in between. It includes:

__ignore = _{
      " " 
   | "," 
   | NEWLINE
   | "trimmed=False"
   | ("at 0x" ~ ASCII_ALPHANUMERIC{12})
   | ("edb." ~ ( ASCII_ALPHA_LOWER* ~ "." )+ )
}

__ = _{ __ignore* } // The potential kinds of erroneous syntax is
                 // not feasible to recreate with the EdgeDB QL Parser

this ^ is useful for making the deserializer more fault-tolerant to accidental modifications —
ensuring that the following:
children=edb.common.checked.CheckedList[edb.common.markup.elements.lang.TreeNodeChild], or

children = edb.common.checked.CheckedList[ 
   edb.common.markup.elements.lang.TreeNodeChild
]

still gets reduced to children=CheckedList[TreeNodeChild].

2) ... -> module -> module_node -> ...

// `module` is a special case of the the `tree_node_child`
module = { 
     "[" ~ "<" 
     ~ "TreeNodeChild"
     ~ "id" ~ "=" ~ id
     ~ "label" ~ "=" ~ "'declarations'" 
     ~ "node" ~ "=" ~ module_node 
     ~ ">" ~ "]"
}

// `module_node` is a special case of `checked_list`
module_node = {
    "<" ~ "List" ~ " " 
    ~ "id" ~ "=" ~ id ~ __
    ~ "items" ~ "=" ~ __ ~ "CheckedList"
    ~ "[" ~ __ ~ "Markup" ~ __ ~ "]"
    ~ "(" ~ __ ~ "[" ~ __ ~ module_declaration ~ __ ~ "]" ~ __ ~ ")" ~ __
}

3) ... -> module_declaration

module_declaration = {
    "<" ~ __ ~  "TreeNode" ~ __
    ~ "id" ~ __ ~ "=" ~ id ~ __
    ~ "name" ~ __ ~ "=" ~ __ ~ "'ModuleDeclaration'" ~ __ // Beware the internal single-quotes ( ' ... ' )
    ~ "children" ~ __ ~ "=" ~ __ ~ "CheckedList" ~ __   
    ~ "[" ~ __ ~ "TreeNodeChild" ~ __ ~ "]" ~ __       
    ~ "<" ~ __ ~ "label" ~ __ ~ "=" ~ "'name'" ~ __   
    ~ "node" ~ __ ~ "=" ~ __ ~ tree_node ~ __
}

4) ... -> tree_node -> ...

tree_node = ${
    "<" ~ __ ~ "TreeNode" ~ __
    ~ "id" ~ __ ~ "=" ~ __ ~ id ~ __
    ~ "name" ~ __ ~ "=" ~ __ ~ name ~ __
    ~ "children" ~ __ ~ "=" ~ __ ~ checked_list ~ __
    ~ ( "brackets" ~ "=" ~ brackets ~ __ )? ~ __
    ~ ">"
}

where: id = { "None" | ('0'..'9'){15} }, str_value = { !(__forbidden_char)+ } and name = { str_value }.

5) ... -> checked_list -> ...

checked_list = ${
    __ ~ "CheckedList" ~ "[" ~ ( __ ~ "TreeNodeChild" | __ ~ "Markup" ) ~ __ ~ "]"
    ~ "(" ~ __ ~ "[" ~ ( __ ~ tree_node_child)+ ~ "]" ~ __ ~ ")"
}

6) ... -> tree_node_child -> ...

tree_node_child = ${
    "<" ~ "TreeNodeChild" ~ __
    ~ "id" ~ "=" ~ id ~ __
    ~ "label" ~ "=" ~ label ~ __
    ~ "node"  ~ "=" ~ rhs ~ ">"
}

where rhs is given by:

rhs = ${
      list_kind 
    | string_kind 
    | atomic_node
    | true_constant_type_kind
}

I'm thinking the error probably lies in either label, rhs, or both. If any clever people notice something, please share it on this thread. 🙂

Asyncio-PyO3 Build CI Migration: Maturin + Poetry CI + edgedb-setup

Hello 2021 World

Last year, Edgemorph encountered a few roadblocks that stifled its progress. Thankfully, a number of those barriers have been removed for me by the work of others because the Open-Source community is great.

Here's the trinity of reasons why Edgemorph's development went cold for several months:

---

Blocker 1.) edm: Importing bootstrapped submodule

It requires a lot of setup just to get edm working, and available, on the system's PATH. @nsidnev was clever enough to introduce me to Poetry, which leverages pyproject.toml to create reproducible builds easily. For the most part, his solution gets the job done.

@nsidnev saving the day here:

Check #8 please. I think I wrote a suitable solution that will handle this case (also added > a separate CI for the Python EDM version and fixed some linter issues).

Originally posted by @nsidnev in #7 (comment)

Unfortunately, it gets trickier when an automated build process needs to also include the steps from edgedb.com under the "Internals" tab. By itself, pyproject.toml just can't handle the added work of compiling two local Python packages without some other integrations.

I was stuck and could not figure out what to do.

Update 1:

Since edm (edgemorph development manager) bootstraps EdgeDB's SDL parser from the raw source under a git submodule, the poetry provided an excellent solution for non-hacky importing. Likewise, edgedb-setup (ee1a54b) makes it possible to automate integration tests under github actions. The testing harness can now be reworked for both unit and integration tests for compilation steps that need to perform introspection queries.

---

Blocker 2.) Rust & Python's asyncio event loop compatibility

EdgeDB's biggest commercial selling point is its low-latency set operations. And since many of its users come from Python, asyncio compatibility with Edgemorph would be necessary. But getting really low-level asyncio bindings is virtually undocumented on the internet. Nevertheless, a recent development makes this possible for me.

Namely, the wizards at pyo3-asyncio are developing async-std and tokio bindings to asyncio for Rust. Just two days ago, they released this.

Update 2:

Post Release

* [x]  Verify docs.rs links to `pyo3-asyncio` in guide
* [x]  v0.13.0 tag
* [x]  Verify crates.io and docs.rs links in README

Released and tagged 0.13.1 to fix docs.rs features

Originally posted by @awestlake87 in awestlake87/pyo3-asyncio#8 (comment)

This was an essential but missing piece from my radar.

---

Blocker 3.) Maturin, edgedb-setup, and PyOxidizer

Getting a working edm command-line executable is not easy or reliable for newcomers. This likely remains a major pain-point to prospective contributors and users, alike. It's even more work when your compiler framework needs an automated way to conduct integration tests. Thankfully, Elvis P. from EdgeDB published a Github-Action that makes the latter simple as .. well... typing letters. As for the former:

Update 3:

Upcoming work will incorporate Maturin's build system -- like the demonstration by rust-numpy at commit ee1a54b -- into Edgemorph so that two distinct build processes can occur.

• edm: To preserve the executable's ability to pull in edgedb and edb Python modules, while also being able to use maturin's build-backend for setup.py support (for edgedb's and edgedb-python's build steps), maturin was essential. In addition, PyOxidizer will be used for creating standalone binaries of edm! This means that my dream will come true: Edgemorph will be available to users who do not want to install Python! That's right, pre-packaged edm binaries can be served without any need to install Python! Instead, Python will only be needed by contributors as a dev dependency!
• edgemorph-rs and edgemorph-py: The low-level Python APIs to be used by edm have been challenging to incorporate into an incremental build process. Using maturin and poetry together should make it simpler to build EdgeDB's edb package (see the "Building Locally" steps at edgedb.com for reference) alongside Edgemorph's own libraries.

---

Closing Remarks

I'm glad to be relieved of these bothersome roadblocks, and to have some light on the other side of the dark tunnel. I've cooked up a number of ideas that I'm pumped to bring into 2021, and hopefully by this time next year we'll be able to celebrate a language-agnostic alpha-release of Edgemorph on all operating systems.

---

Interesting Mentions

There are a couple community projects which deserve your love and attention:

• Pumpkin DB:
  • https://github.com/PumpkinDB/PumpkinDB
• Strawberry GraphQL:
  • Website: https://strawberry.rocks
  • Repository: https://github.com/strawberry-graphql/strawberry.rocks
• JINA:
  • Website: https://docs.jina.ai/
• Piccolo:
  • Repository: https://github.com/piccolo-orm
• Lofi (Low-Overhead Fibers) [Rust crate]
  • Repository: https://gitlab.com/niksaak/lofi

A few questions about future

Hi! Cool project. I did something similar but only for python a few weeks ago (but haven't uploaded it yet and not sure if I will now), but my goal was to provide the correct type hints for conveniently using the return values from the driver with IDEs.

I have a few questions about what and how this project will do. Most of my questions are about Python (since this is my primary language).

NOTE: I'm not a native speaker, so if my questions sound rude - sorry, they shouldn't be, I'm just really interested in the answers.

Consider the following Python snippet from the README:

from edgemorph import ( edgetype, property, link, multi )
from .edm_user import ( NamedType, HasAddressType, UserType )

@edgetype(abstract=True, edb=NamedType)
class Named:
    name: property[str]

@edgetype(abstract=True, edb=HasAddressType)
class HasAddress:
    address: property[str]

@edgetype(extending=(Named, HasAddress), edb=UserType)
class User:
    friends: multi[ link[__qualname__] ]
    index:   {
        "name": lambda title : "User name index"
    }

Questions:

1. Why do we need separate types such as NamedType, HasAddressType or UserType (and BTW .edm_user module)?

2. Why use decorators instead of extending classes like in other ORMs? This will add autocomplete to IDEs for the parent fields and it looks more pythonic(IMO). I understand that in Rust, you can simply expand all necessary fields through a macro, but for Python this method seems to me not suitable. Something like that:

class Named(edgemorph.Type):
    __abstract__ = True  # or something like "class Meta" from Django 
    ...

...

class User(Named, HasAddress):
    ...

3. Why not move all the metadata such as index and annotation definitions into something like the Meta class, like Django or Tortoise-ORM does. Or use descriptors for some field definitions (as done in Pydantic and SQLAlchemy) and do something like this:

@edgetype(extending=(Named, HasAddress), edb=UserType)
class User:
    ...

    class Meta:
        indieces = ("name",)

or

@edgetype(extending=(Named, HasAddress), edb=UserType)
class User:
    ...
    name: Property[str] = Property(index=True)

4. What about link properties definition? I had a couple of problems with them in my internal tool.

5. Will this project also be a query builder for EdgeDB? It would be great if so, but then there are also a couple more questions about types definition and general usage.

6. If this project is going to be a query builder, then it is not obvious how to use types as type hints for the values ​​returned by the driver (in fact, it is not obvious to me even now). For example:

def select_user_by_id(conn, *, id) -> User:
    ...

def my_func(conn: edgedb.BlockingIOConnection) -> None:
    user = select_user_by_id(conn, id=uuid.uuid4())

    # not sure if linters and IDEs will understand that the .name property is str
    print("username starts with 'test: {}`.format(user.name.startswith("test")))  

6. Do you need any help with it? I'll be glad to help :)

Credentials entry is not compatible with the official shell script installation

Need to adapt the credentials configuration to reflect parameters set up by the official script

TypeError: compile() takes 1 positional argument but 2 were given

Summary

Running edm make | edm make * fails when multiple module files are reachable within edgemorph.toml.

Recreation Steps

1. edm init etest
2. Update edgemorph.toml manually:

[edgemorph]
project_root    = "etest"
mod_directories = ["/edb_modules"]

[edgemorph.codegen]
schema_name = "Test"

[edgemorph.codegen.rust]
enabled = "true"

[edgemorph.codegen.rust.modules]
    [edgemorph.codegen.rust.modules.etest]
    source = "/edb_modules/etest.esdl"
    output = "/src/lib/edm_etest.rs"
    [edgemorph.codegen.rust.modules.cinema]
    source = "/edb_modules/cinema.esdl"
    output = "/src/lib/edm_cinema.rs"

[edgemorph.codegen.python]
enabled = "true"

[edgemorph.codegen.python.modules]
    [edgemorph.codegen.python.modules.etest]
    source = "/edb_modules/etest.esdl"
    output = "/etest/edm_etest.py"
    [edgemorph.codegen.python.modules.cinema]
    source = "/edb_modules/cinema.esdl"
    output = "/etest/edm_cinema.py"

[edgedb]
[edgedb.databases]
[edgedb.databases.primary]
name = ""
dsn = ""

[edgedb.databases.primary.modules]
etest = "/edb_modules/etest.esdl"
cinema = "/edb_modules/cinema.esdl"

2. cd etest/edb_modules
3. Edit etest.esdl:

module etest {
    type User {
        property email -> str;
        property password -> str;
        property name -> str;
    };
    type Session {
        required single link user -> User;
        required single property allottedDuration -> std::duration {
            default := (WITH
                MODULE api
            SELECT
                <duration>'24 hours'
            );
        };
        required single property createdAt -> std::datetime {
            default := (WITH
                MODULE api
            SELECT
                datetime_current()
            );
        };
        required single property sessionID -> std::str;
        single property token -> std::str;
    };
    function create_new_session(EMAIL: std::str, PASS: std::str) -> SET OF Session using (
        FOR USER IN {validate_credentials(EMAIL, PASS)}
            UNION (
                INSERT Session {
                    token := <str>(SELECT random_big_str_id()),
                    sessionID := <str>(SELECT random_big_str_id()),
                    user := USER
                }   
            )
         );

    function validate_credentials(EMAIL: std::str, PASS: std::str) -> SET OF User using (
            SELECT
                User
            FILTER
                .email = EMAIL
                AND
                .password = PASS
            LIMIT 1
        );
}

then write and quit.
4. In the same directory, open and write cinema.esdl :

module cinema {
    type Movie {
        required property title -> str;
        # the year of release
        property year -> int64;
        required link director -> Person;
        required multi link actors -> Person;
    }
    type Person {
        required property first_name -> str;
        required property last_name -> str;
    }
}

Output:

(bootstrap) 1018 $ edm make
../edgemorph.toml
Compiling your EdgeDB modules....
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/david/.pyenv/versions/3.8-dev/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
TypeError: compile() takes 1 positional argument but 2 were given
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/david/code/edgemorph/edm/edm", line 715, in <module>
    main(args)
  File "/home/david/code/edgemorph/edm/edm", line 599, in main
    eval(f"{func_name}(arg)")
  File "<string>", line 1, in <module>
  File "/home/david/code/edgemorph/edm/edm", line 293, in make
    batch_compilation(retrieved)
  File "/home/david/code/edgemorph/edm/edm", line 309, in batch_compilation
    async_res  = [
  File "/home/david/code/edgemorph/edm/edm", line 310, in <listcomp>
    job.get(timeout=12)
  File "/home/david/.pyenv/versions/3.8-dev/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
TypeError: compile() takes 1 positional argument but 2 were given

Replace edgemorph.toml with edgemorph.ron

I align with the reasons at https://github.com/ron-rs/ron, so edgemorph will be ditching the TOML format before the alpha release. See RON grammar at https://github.com/ron-rs/ron/blob/HEAD/docs/grammar.md

Leverage Type Introspection (via edgedb-rust) with Deserialized ESDL AST (via Python EDB)

Abstract

We propose to use ESDL abstract syntax trees as the basis for code generation in the Edgemorph framework.

---

Motivation

To Edgemorph, EdgeDB's compiler is the root of all magic. By digesting an SDL module into AST tokens, we share a common tongue between any supported programming language. To this end, it is reasonable to use a strongly-typed programming language like Rust or TypeScript to build boilerplate code structures from a user's schema definitions.

Warning: Do not stand in front of the EdgeDB bullet train.

Since EdgeDB's capabilities continue to rapidly evolve, and since its SDL language continues to mature in ways that enrich each user's experience, it becomes imperative for Edgemorph to jump out of the way and trek behind — following the smokestacks. We do this by capturing abstract syntax structures during the edm make process, and disk-cache them for interpretation during edm make install.

We believe this approach offers the greatest amount of backward and forward compatibility between successive EdgeDB version releases — because each tag (i.e. alpha-3, alpha-4, ... ) will correspond to its own variant of AST deserialization requirements. Moreover, whenever EdgeDB announces a new release, e.g. version alpha-N, AST changes resulting from alpha-N's release will only need to be developed on a fork of the latest Edgemorph edition (the one corresponding to EdgeDB version alpha-N - 1 ).

To be clear, this is a high-effort, high-maintenance approach but the tradeoff is guaranteed backwards compatibility with EdgeDB.

---

Type Specifications

The purpose of this RFC's Abstract Specifications section is to identify generic templates that will be coded in Rust. For example, the serialized abstract syntax below must have each of its fields meaningfully converted into a Rust type at compile time. (Note: The following list of types is not complete, but it does cover the most common AST token kinds.)

Example of a serialized module's abstract syntax tree

<TreeNode id=139713944389664, name='ModuleDeclaration', children=edb.common.checked.CheckedList[edb.common.markup.elements.lang.TreeNodeChild]([<TreeNodeChild id=None, label='name', node=<TreeNode id=139713904379744, name='ObjectRef', children=edb.common.checked.CheckedList[edb.common.markup.elements.lang.TreeNodeChild]([<TreeNodeChild id=None, label='name', node=<String str='etest' at 0x7f119fe79880> at 0x7f119ed86580>]) at 0x7f119ed86100> at 0x7f119f0cb400>, <TreeNodeChild id=None, label='declarations', node=<List id=139713904382336, items=edb.common.checked.CheckedList[edb.common.markup.elements.base.Markup]([<TreeNode id=139713904380176, name='CreateObjectType', children=edb.common.checked.CheckedList[edb.common.markup.elements.lang.TreeNodeChild]([<TreeNodeChild id=None, label='name', node=<TreeNode id=139713904380224, name='ObjectRef', children= ...

---

TreeNode

• id: <i32>
• name: N_T such that N_T ∈ T ´ and T ´ satisfies the size requirements for each of the following identifiers. :

{ 'BinOp',
  'CreateAlias', 
  'CreateConcreteLink', 
  'CreateConcreteProperty',
  'CreateFunction',
  'CreateIndex',
  'CreateLink', 
  'CreateObjectType', 
  'CreateScalarType',
  'ForQuery',
  'FuncParam',
  'FunctionCall', 
  'FunctionCode',
  'InsertQuery', 
  'IntegerConstant',
  'ModuleAliasDecl', 
  'ModuleDeclaration',
  'ObjectRef', 
  'Path', 
  'Ptr',
  'Schema', 
  'SelectQuery', 
  'Set',
  'SetAnnotation',
  'SetField', 
  'ShapeElement', 
  'ShapeOperation',
  'StringConstant', 
  'TypeCast', 
  'TypeName' }  

• children: CheckedList<TreeNodeChild, Markup>

---

TreeNodeChild

• id: Optional<i32>
• label: String s, such that s ∈ L = { "name", "target", "maintype" }
• node: enum <String ; TreeNode; List >, with the following corollaries:
  • node::String → <String str = '%s'>;
  • node::TreeNode → &'a Sized<RefCell<Weak<TreeNode<'a>>>>. 'a is the lifetime specifier for the TreeNode it ellides. Sized<T> is a type with known size. RefCell<T> is a mutable memory location with dynamically checked borrow rules¹. Weak<T> is a pointer that holds a non-owning reference to the managed allocation². TreeNode is an EdgeDB language markup base object subtype.

---

Schema

• declarations: Vec<ModuleDeclaration>

---

ModuleDeclaration

• name: ObjectRef<&str>
• declarations: Vec<Declaration>

---

Declaration

• CreateAlias:

• CreateObjectType:

  name: String,
  commands: Vec<CreateConcreteProperty> | Vec<CreateConcreteLink>

• CreateFunction:

  name: String
  params: Vec<FuncParam>
  returning: Optional<TreeNodeChild>
  returning_typemod: Optional<TreeNodeChild>

---

BinOp

• left: Expr
• op: String
• right: String

---

Implementation Considerations

It can be difficult to write a parser for any complex serialized AST where all types must have known sizes at compile time. While a "TT (token tree) Muncher" seems like a viable option, the practicality of a TT muncher at this scale is brutal. The sheer volume of terms and tokens to match (or discard) makes this difficult to maintain. A more suitable approach would be to write the deserializer with some formal grammar modularity. My preference leans toward PEG and Pest.

Regardless of the approach taken for the intermediate deserialization step, Edgemorph will either need to create the innermost leaf nodes and run to_owned() once inside their owner's ::new(...) method, or Edgemorph could adapt the builder methods in datastructures.rs to stitch distinct nodes together within their owners. Lastly, Edgemorph could operate upon the AST and match against a giant enum-like structure to allocate each of the codegen Rust types.

References

1. std::cell::RefCell
2. std::rc::Weak