berlincode/ndb-orm

Python-NDB-ORM: ndb orm for use with google-cloud-datastore (python3 compatible)

ndb-orm's Issues

async variants?

Hi @berlincode ,

We are switching from GAE standard to GAE flexible. I'm looking for a good NDB replacement and we need performance.
For the C10K problem Python3 has asyncio that provides high performance by event loop and asynchronous I/O.

Why doesn't your ndb-orm have _async variants?

How do we make ndb-orm to support fully async operations?
I found this one. Is it possible to combine ndb-orm with it?

Usage of type 'unicode' in Python 3.x is not defined

I'm adding a KeyProperty to a model object and I'm getting an error about unicode type being not defined.

File "/usr/local/lib/python3.6/site-packages/ndb_orm/model.py", line 1961, in __init__ if isinstance(kind, unicode): NameError: name 'unicode' is not defined

`query.py` file missing

The ndb-orm/query.py file is missing. Running any query with Model.query() results in error:

File "/home/.../venv/lib/python3.6/site-packages/ndb_orm/model.py", line 3420, in _query
    from .query import Query  # Import late to avoid circular imports.
ModuleNotFoundError: No module named 'ndb_orm.query'

README.md says:

This is not a drop-in replacement for the whole ndb package (no ndb.context, ndb.tasklets, ndb.query or ndb.Key), but allows you to use ndb.model classes.

So the file might be intentionally omitted. This code throws an exception when properly used though.

How to reproduce:

• Create an entity Person
• Call Person.query()

__get_arg logic behaviour from Model.__init__

I think I've spotted an issue and I'm not sure if this is the intended behaviour, line ~2935 in model.py:

@classmethod
  def __get_arg(cls, kwds, kwd):
    """Internal helper method to parse keywords that may be property names."""
    alt_kwd = '_' + kwd
    if alt_kwd in kwds:
      return kwds.pop(alt_kwd)
    if kwd in kwds:
      obj = getattr(cls, kwd, None)
      if not isinstance(obj, Property) or isinstance(obj, ModelKey):
        return kwds.pop(kwd)
    return None

more exactly: if not isinstance(obj, Property) or isinstance(obj, ModelKey):

Shouldn't this actually be: if not (isinstance(obj, Property) or isinstance(obj, ModelKey)): in order to pop out the potential keyword argument from the kwds dict? I think the logic of this is actually the following:

If the requested keyword argument name isn't by any chance an already existing property of the model (as Property or the special one ModelKey), only then retrieve & return the value (and pop the key out of kwds). Otherwise, we need to keep it in, because the value belongs to the property and it's not a parameter of the model initializer.

Or, for example in the key param case, I may be totally wrong, because ModelKey is subclass of Property and requesting it from the kwds should actually return it, but will fail in the first check, that's why there's the right side of the or which checks for this special property in order to return it (and pop it out from the properties), so the key can be post-processed and set inside the __init__ method, rather than being passed to the properties setting method furthermore.

[Enhancement] Useful BaseModel class (not an issue)

I think most "legacy" GAE users will find this useful:

"""Base models and utilities used by the derived ones."""


import datetime
import logging

import ndb_orm as ndb
from google.cloud import datastore

from <project> import app, settings


# Datastore default client settings.
PROJECT = settings.PROJECT_ID
NAMESPACE = app.config["DATASTORE_NAMESPACE"]
NDB_KWARGS = {"project": PROJECT, "namespace": NAMESPACE}

# Module level singleton client used in all DB interactions. This is lazy inited when
# is used only, so we don't have any issues with Datastore agnostic tests/debugging,
# because creating a client will require valid credentials.
client = None


class BaseModel(ndb.Model):

    """Common model properties and functionality."""

    # String used for properties with no available data.
    NOT_SET = "N/A"

    created_at = ndb.DateTimeProperty(auto_now_add=True)

    def __init__(self, *args, **kwargs):
        self._get_client()  # Activates all NDB ORM required features.
        kwargs.update(NDB_KWARGS)
        super().__init__(*args, **kwargs)

    @classmethod
    def model_name(cls):
        return cls.__name__.replace("Model", "")

    @classmethod
    def normalize(cls, value):
        if value is None:
            return cls.NOT_SET
        return value

    @staticmethod
    def _get_client():
        global client
        if not client:
            client = datastore.Client(**NDB_KWARGS)
            ndb.enable_use_with_gcd(client=client, **NDB_KWARGS)
        return client

    @classmethod
    def query(cls, **kwargs):
        query = cls._get_client().query(kind=cls._get_kind(), **kwargs)
        return query

    @classmethod
    def all(cls, query=None, keys_only=False, **kwargs):
        query = query or cls.query()
        query.order = ["-created_at"]
        if keys_only:
            query.keys_only()
        return list(query.fetch(**kwargs))

    @property
    def myself(self):
        """Return the current DB version of the same object."""
        return self.key.get()

    @property
    def exists(self):
        """Checks if the entity is saved into the Datastore."""
        try:
            return bool(self.myself) if self.key and self.key.id else False
        except Exception:
            return False

    def put(self):
        """Saving the entity into the Datastore."""
        self._get_client().put(self)
        return self.key

    @classmethod
    def put_multi(cls, entities):
        """Multiple save in the DB without interfering with `cls.put` function."""
        cls._get_client().put_multi(entities)
        return [entity.key for entity in entities]

    def remove(self):
        """Removes current entity and its dependencies (if any)."""
        self.key.delete()

    @classmethod
    def remove_multi(cls, keys):
        cls._get_client().delete_multi(keys)

    @property
    def urlsafe(self):
        return self.key.to_legacy_urlsafe().decode(settings.ENCODING)

    @classmethod
    def get(cls, urlsafe_or_key):
        if isinstance(urlsafe_or_key, (str, bytes)):
            key = ndb.Key(cls, **NDB_KWARGS)
            complete_key = key.from_legacy_urlsafe(urlsafe_or_key)
        else:
            complete_key = urlsafe_or_key

        item = complete_key.get()
        if not item:
            raise Exception("item doesn't exist")
        return item

Don't be sceptical, this is tested by these:

• conftest.py
• test_base.py

Just replace <project> with your package name (under a Flask app for example).

Key class conflicts

Looks like there is a conflict between the ndb_orm key module and the associated google.datastore.cloud key module.

Any timeline into integrating the ndb.Key functionality ?

"KeyBase is not implemented yet" in version 0.9.0

I'm trying to directly instantiate a ndb.Key() to use with my Entity object.

However looking at the source code, this instantiates a KeyBase class which is not yet implemented.
I'm not sure if you have a different version of this class.

raise NotImplementedError('KeyBase class is not set up (yet)') NotImplementedError: KeyBase class is not set up (yet)
Instantiated from KeyClass line 49.

Using namespace with ndb-orm

I need to setup multi tenancy using the namespace concept of the Cloud DataStore.
Inspecting the code, I'm not sure this is fully implemented within the current version of the library.

Can you help me understand what is missing to support that ? Thanks