onyb/reobject

Python without ifs and buts - an ORM layer for Python objects, inspired by Django

reobject

reobject is an ORM layer for your objects. It allows you to track and query objects at runtime using a familiar query langauge inspired by Django ORM.

Note: reobject is NOT a database ORM. It keeps track of regular objects in the memory.

This is highly experimental code, and not safe for production.

Installation

reobject supports Python 3 only.

pip install reobject

Example usage

from reobject.models import Model, Field

class Book(Model):
    title = Field()
    authors = Field()
    price = Field()

>>> # Create a bunch of objects
>>> Book(title='The C Programming Language', authors=['Kernighan', 'Ritchie'], price=52)
>>> Book(title='The Go Programming Language', authors=['Donovan', 'Kernighan'], price=30)

>>> Book.objects.all()  # All books
[Book(title='The C Programming Language', authors=['Kernighan', 'Ritchie'], price=52),
 Book(title='The Go Programming Language', authors=['Donovan', 'Kernighan'], price=30)]

>>> Book.objects.filter(price__lt=50).values('title')  # Titles of books priced under $50
[{'title': 'The Go Programming Language'}, {'title': 'The C Programming Language'}]

>>> # Titles of books co-authored by Brian Kernighan
>>> Book.objects.filter(authors__contains='Kernighan').values_list('title', flat=True)
['The Go Programming Language', 'The C Programming Language']

Features

• Elegant data-model syntax inspired by Django ORM.
• Class-level model fields, out of the box object protocols, pretty reprs; powered by attrs.
• Advanced query language and chainable querysets. Read the QuerySet API docs.
• Transactions. See example.
• Many-to-one model relationships. See example
• [TBA] Attribute indexes for fast lookups.

Crunching Design Patterns

Pattern	Description	Pure Python	reobject
Flyweight	Reuse existing instances of objects with identical state	Link	Link
Memento	Transactional rollback of an object to a previous state in case of an exception	Link	Link
Prototype	Create clones of a prototype without instantiation	Link	Link
Singleton	Restrict a class to provide only a single instance	Link	Link
Facade	Encapsulate a complex subsystem within a single interface object	Link	Link
Flux	Event-driven state management inspired by Facebook Flux	Link	Link

_{Note: Some of the examples above may be inaccurate. The idea is to demonstrate what reobject is capable of. Pull requests are most welcome.}

Contributing

Want to help? You can contribute to the project by:

• Using reobject in your projects, finding bugs, and proposing new features.
• Sending pull requests with recipes built using reobject.
• Trying your hand at some good first bugs.
• Improving test coverage, and writing documentation.

reobject's Issues

Support complex queries in filters

This involves a major refactoring in the way we process queries, notably something along the lines of Django's Q objects.

Links:

• AST implementation: https://ruslanspivak.com/lsbasi-part7
• Django Q objects: https://docs.djangoproject.com/en/1.10/topics/db/queries/#complex-lookups-with-q-objects
• Things which might be interesting to have in Q objects: https://github.com/jamescooke/blog/blob/master/content/1603-comparing-django-q-objects.rst

Q objects should be smart enough to understand queries like:

• <field>__isnull=<bool>
• <field>__contains=<item>
• <field>__in=<list>
  etc.

Finally all kwargs in a filter() should be translated to Q objects composed together with AND operator.

Packaging

Bring the pip install reobject goodness.

Doesn't support with random objects, `order_by('?')`

In [11]: from practical_foobar_egg_labs.courses.models.course import CourseModel

In [12]: from practical_foobar_egg_labs.courses.utils.storage import CourseData

In [13]: courses = CourseData().get_courses()
INFO 2021-05-26 00:00:40,271 storage 1 139861811062592 [storage.py:87] [queryset_courses] Retrieved courses from cache.

In [14]: courses
Out[14]: 
[{'title': 'Foobar Foundation Course',
  'description': 'The Foobar Foundation course is our beginner course which prepares you for the professional, expert and architect courses',
  'image': 'cdp2.png',
  'keywords': ['SAST', 'DAST', 'IaC', 'SCA'],
  'flow': ['linux-basics',
   'docker',
   'continuous-integration-and-continous-delivery'],
  'details': {'steps': [{'title': 'Step 1 - Download the code from the repo',
     'text': 'step1.md'},
    {'title': 'Step 2 - Install the TruffleHog', 'text': 'step2.md'},
    {'title': 'Step 3 - Run the SAST Scan', 'text': 'step3.md'}],
   'intro': {'text': 'intro.md'},
   'finish': {'text': 'finish.md'}},
  'course_slug': 'foobar-foundation'},
 {'title': 'Python Foundation Course',
  'description': 'The Python Foundation course is our beginner course which prepares you for the professional, expert and architect courses',
  'image': 'cdp2.png',
  'keywords': ['Python', 'Programming'],
  'flow': ['python-basics', 'python-intermediate', 'python-professional'],
  'details': {'steps': [{'title': 'Step 1 - Download the code from the repo',
     'text': 'step1.md'},
    {'title': 'Step 2 - Install the TruffleHog', 'text': 'step2.md'},
    {'title': 'Step 3 - Run the SAST Scan', 'text': 'step3.md'}],
   'intro': {'text': 'intro.md'},
   'finish': {'text': 'finish.md'}},
  'course_slug': 'python-foundation'}]

In [15]: 

In [15]: for course_data in courses:
    ...:     course_slug = course_data.pop('course_slug', None)
    ...:     CourseModel.objects.get_or_create(
    ...:         course_slug=course_slug,
    ...:         defaults=course_data
    ...:     )
    ...: 

In [16]: 
In [17]: CourseModel.objects.all()
Out[17]: 
[CourseModel(course_slug='foobar-foundation', title='Foobar Foundation Course', description='The Foobar Foundation course is our beginner course which prepares you for the professional, expert and architect courses', image='cdp2.png', keywords=['SAST', 'DAST', 'IaC', 'SCA'], flow=['linux-basics', 'docker', 'continuous-integration-and-continous-delivery'], details={'steps': [{'title': 'Step 1 - Download the code from the repo', 'text': 'step1.md'}, {'title': 'Step 2 - Install the TruffleHog', 'text': 'step2.md'}, {'title': 'Step 3 - Run the SAST Scan', 'text': 'step3.md'}], 'intro': {'text': 'intro.md'}, 'finish': {'text': 'finish.md'}}),
 CourseModel(course_slug='python-foundation', title='Python Foundation Course', description='The Python Foundation course is our beginner course which prepares you for the professional, expert and architect courses', image='cdp2.png', keywords=['Python', 'Programming'], flow=['python-basics', 'python-intermediate', 'python-professional'], details={'steps': [{'title': 'Step 1 - Download the code from the repo', 'text': 'step1.md'}, {'title': 'Step 2 - Install the TruffleHog', 'text': 'step2.md'}, {'title': 'Step 3 - Run the SAST Scan', 'text': 'step3.md'}], 'intro': {'text': 'intro.md'}, 'finish': {'text': 'finish.md'}})]

In [18]: CourseModel.objects.all().order_by('?')
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-18-71eea73a8983> in <module>
----> 1 CourseModel.objects.all().order_by('?')

/usr/local/lib/python3.8/site-packages/reobject/query/queryset.py in order_by(self, *fields)
    161 
    162         return type(self)(
--> 163             sorted(self, key=cmp(*fields)),
    164             model=self.model
    165         )

/usr/local/lib/python3.8/site-packages/reobject/utils.py in g(obj)
     11 
     12     def g(obj):
---> 13         return tuple(
     14             resolve_attr(obj, attr) for attr in items
     15         )

/usr/local/lib/python3.8/site-packages/reobject/utils.py in <genexpr>(.0)
     12     def g(obj):
     13         return tuple(
---> 14             resolve_attr(obj, attr) for attr in items
     15         )
     16 

/usr/local/lib/python3.8/site-packages/reobject/utils.py in resolve_attr(obj, attr)
     26             obj = obj.get(name)
     27         else:
---> 28             obj = getattr(obj, name)
     29 
     30     return obj if not reverse else -obj

AttributeError: 'CourseModel' object has no attribute '?'

In [19]: 

How to custom objects Manager?

from reobject.models.manager import Manager



class ReobjectManager(Manager):

    def get_or_create(self, defaults=None, **kwargs):
        self.model.objects.filter(**kwargs).delete()
        return super().get_or_create(defaults, **kwargs)



class CourseModel(Model):
    course_slug = Field()
    title = Field()
    description = Field()

    objects = ReobjectManager()

But I'm seeing an error:

TypeError: __init__() missing 1 required positional argument: 'model'

when I fill it with objects = ReobjectManager(CourseModel) then it CourseModel not defined.

Implement first, last, and random

To implement (ref: https://docs.djangoproject.com/en/1.10/ref/models/querysets/#earliest)

• earliest
• latest
• first
• last
• random

Cannot set `id` as field

I found an error when my data have an id or pk field. But, when I removed it id, it just working fine.

from reobject.models import Model, Field

data = [
  {'id': 1, 'name': 'brad', 'color': 'red', 'tags': [], 'author': {'name': 'admin'}},
  {'id': 2, 'name': 'sylvia', 'color': 'blue', 'tags': [], 'author': {'name': 'user'}},
  {'id': 3, 'name': 'sylwia', 'color': 'green', 'tags': [], 'author': {'name': 'admin'}},
  {'id': 4, 'name': 'shane', 'color': 'red', 'tags': [], 'author': {'name': 'admin'}},
  {'id': 5, 'name': 'shane', 'color': 'red', 'tags': ['python', 'django'], 'author': {'name': 'user'}}
]

class Book(Model):
    id = Field()
    name = Field()
    color = Field()
    tags = Field()
    author = Field()


for item in data:
    Book(**item)


Book.objects.all()
Book.objects.filter(author__name='admin')

---

Also it doesn't work for:

Book.objects.filter(tags__in=['python'])

oh it working with:

Book.objects.filter(tags__contains='python')

Implement get_or_create

See Django reference here: https://docs.djangoproject.com/en/1.10/ref/models/querysets/#get-or-create

A possible use case of get_or_create would be implementation Singleton pattern. The method should work on both QuerySet as well as Manager.

Support reverse ordering in values and values_list

This is supported:
Foo.objects.filter().order_by('-foo')

whereas, this is not:
Foo.objects.filter().values('email', 'foo').order_by('-foo')

Implement model relationships

Django-style many-to-many relationships is a highly desirable feature. There might be some refactoring involved in reobject.manager.Manager.

class Publication(models.Model):
    def __init__(self, title):
        self.title = title

class Article(Model):
    def __init__(self, headline):
        self.publications = models.ManyToManyField(Publication)
        self.headline = headline

ReObject should allow the following usages:

• a.publications.all()
• p.article_set.all()
• a.publications.create(..)
• a.publications.add(p)
• Article.objects.filter(publications__title__startswith="Science")
• Publication.objects.filter(article__headline__startswith="NASA")

Allow functional constructs to be chained with the QuerySet

The idea is to do something like:

SomeClass.objects.filter(val__lte=<foo>).value_list('val', flat=True).map(callable)

This needs to be spec'd properly, but some inspiration can be taken from:

• http://jrsinclair.com/articles/2017/javascript-without-loops/
• http://jrsinclair.com/articles/2017/indentation-is-the-enemy-less-complex-javascript/

How to bypass unknown keys?

Hello @onyb, can you please help me how to bypass unkonwn keys?

For example, we only accepted 3 fields:

class Book(Model):
    title = Field()
    authors = Field()
    price = Field()

But in another case, the payloads become dynamic, where allow_trial is not defined in the field yet.
It can be 1-or more undefined fields will be come. and not using Field(default=xxx) as approach.

payloads = {
    "title": "Hello world",
    "authors": [],
    "price": 21.5,
    "allow_trial": True,  # here is
}

book = Book(**payloads)

So, based on above, it will raise an error;

TypeError: __init__() got an unexpected keyword argument 'allow_trial'

Meanwhile, in my case, I just need to bypass the allow_trial field, and keep running with adding print/logging error info.

I have tried to modify the __new__(...) function, but seems still got the same error:

import attr

class Book(Model):
    ...


    def __new__(cls, *args, **kwargs):
        acceptable_keys = [field.name for field in attr.fields(cls)]
        for key, value in kwargs.copy().items():
            if key not in acceptable_keys:
            kwargs.pop(key)  # remove unknown key
        return super().__new__(cls, *args, **kwargs)

OR, the second option is to allow that unknown allow_trial field.

Switch to class attributes instead of instance variables

The idea is to be able to define models like this:

class Article(Model):
    publications = models.ManyToManyField(Publication)
    headline = models.CharField()

instead of:

class Article(Model):
    def __init__(self, headline):
        self.publications = models.ManyToManyField(Publication)
        self.headline = headline

Arguments for this change:

• Class variables will allow reobject to inspect them at load-time and infer model relationships.
• Gives the ability to perform instantiation-time field validation.
• Gives the ability to implement post-init hooks => reobject signals?
• Closer to Django syntax.

Arguments against this change:

• "Plug and play" functionality is lost. One needs to refactor her codebase to use reobject.
• reobject classes will hence forth be special classes.

Resources:
Take inspiration from the attrs.

Technical considerations:
Will involve a major rework of reobject.model. I am considering adopting the following syntax:

class Article(metaclass=Model):
    publications = models.ManyToManyField(Publication)

The Model class will simply proxy the attribute generation process to attrs. See: http://www.attrs.org/en/stable/api.html#attr.make_class

Static type checking

• Since we are purely Python 3, it would be convenient and nice to have typing information at least in the interfaces which are publicly exposed.
• Add mypy to test-requirements.txt and add static type checks to Travis CI builds.

Performance benchmarks

Parameters:

• Memory
• Program size
• CPU
• Time

See http://benchmarksgame.alioth.debian.org/how-programs-are-measured.html for inspiration.

To Do: Update Readme with performance results.

Implement LimitedQuerySet class

The result returned by some QuerySet methods do not allow chaining with other methods due to their form. For example, the result of a value_list will not allow operations like order_by.

Any QuerySet method which returns a result where the items are not Model instances qualify to return instances of type LimitedQuerySet.

Example use case:

>>> Foo.objects.filter(...).values_list('pk', 'name', 'idx')
[(123, 'A', 0,), (124, 'B', 1,), ...]  # LimitedQuerySet

>>> Foo.objects.filter(...).values_list('pk', 'name', 'idx').map(lambda c: hash(c))
[-8396369143964016954, 3966759431255512253, ...]  # LimitedQuerySet