cos-archives/modular-odm

A database-agnostic Object-Document Mapper for Python.

modular-odm's Issues

Remove implicit prefixing

The "db_" prefix added when setting PickleStorage is unexpected behavior. I'd prefer specifying my db names explicitly, or at least have the prefix default to None or "".

Remove Flask as a hard dependency?

In the interest of keeping the package lightweight, I think Flask can be remove as a hard dependency if the only thing that depends on it is the FlaskStoredObject. It would make more sense to release Flask integration as a Flask extension (a la Flask-SQLAlchemy and Flask-Peewee).

Allow foreignfield definition by class

E.g.

class User(StoredObject): pass
class Project(StoredObject):
    creator = fields.ForeignField(User)

h/t @lyndsysimon

The url with either "." or "$" will raise an error in datebase

Raise better errors on forbidden characters in Mongo

MongoDB fields can't include "."s or begin with "$"s; raise appropriate exceptions when this happens.

Simpler querying of database

A simple query of the database would be good. Right now I'm left with

from modularodm.query.querydialect import DefaultQueryDialect as Q
stuff = Stuff.find(Q('label', 'eq', 'foo'))

I would prefer to just add the field and what I'm searching by as arguments.

Different exceptions raised on inserting a duplicate key, depending on backend

If a duplicate key is inserted into the database, different exceptions may be raised, depending on the backend used:

• PickleStorage and EphemeralStorage raise KeyExistsException
• MongoStorage raises DuplicateKeyError

I think it would be better if a single error type were raised, so a user can safely change the storage backend without having to change their error handling code.

Easier import of DefaultQueryDialect

Instead of:

from modularodm.query.querydialect import DefaultQueryDialect as Q

Why not just:

from modularodm import Q

StoredObjects can be defined, instantiated and saved with a non-required primary key

Simplified case - no error is generated by the following:

from modularodm import fields, StoredObject
from modularodm.storage.ephemeralstorage import EphemeralStorage

storage = EphemeralStorage()

class Foo(StoredObject):
    _id = IntegerField(primary=True, required=False)

Foo.set_storage(storage)

foo = Foo()
foo.save()

foo.to_storage()
# {'_id': None, '_version': 1}

storage
# {None: {'_id': None, '_version': 1}}

Fix changes to primary keys

Changing the primary key of a StoredObject will currently break relationships. Changing the primary key should either raise an exception or find and modify all relationships with the changed record.

Fix failing offset and limit tests

Error Raised When No Match is Found with find_one

I would expect that None would be returned instead, as of now an error is raised.

Assignment of non-existent attributes does not raise an AttributeError

When assigning a value to an attribute of a subclass of StoredObject, an exception is not raised when an arbitrary attribute is assigned.

Simplified case:

from modularodm.StoredObject import StoredObject

from modularodm.fields.IntegerField import IntegerField

class Foo(StoredObject):
    _id = IntegerField()

foo = Foo()

# this line should raise an AttributeError
foo.not_a_real_attribute = 1

Offset tests failing

There are currently 18 tests that are failing in test/queries/test_simple_queries.py. All of them pertain to sorting and offsetts.

No license

Let's decide on a license!

@jmcarp and @JeffSpies , since you are the maintainers, you have final say on this.

Method to update multiple fields

Instead of

person.firstname = 'Fred'
person.lastname = 'Mercury'

this:

person.update_fields(firstname='Fred', lastname='Mercury')

Loop on attribute assignment in subclass before `StoreObject`'s __init__ is called.

class Foo(StoredObject):
    _id = IntegerField()
    att = StringField()

    def __init__(self):
        self.att = 'foo'

This raises a RuntimeError: maximum recursion depth exceeded - it should at least tell the user to call StoredObject.__init__() before item assignment.

Allow backreference access by key

In addition to schema__key and schema__key__field, allow retrieval of backreferences (probably as ForeignList) by key.

Add JSON backend

Installation instructions out of date

The installation instructions suggest to:

pip install modular-odm

but that gives an old version of the software. Either the pypi setup needs changing or the instructions do.

Add documentation for the following

• Backref access
• sorting

Working with mongo and modular-odm

A note that wasn't clear from the examples:

If you are using Flask, In order to use PyMongo and also include your db logic in a models file, you must use the app context to initialize the database, so:

with app.app_context():
    Text.set_storage(storage.MongoStorage(mongo.db, 'text'))

Might be obvious to more experienced uses of Flask, but wasn't to me initially :).

Reserved characters in DictionaryField keys raises an Exception with MongoDB backend

If you create an instance of a subclass of StorageObject with MongoDB as the backend, then add a key containing a period to a DictionaryField an exception is raised upon attempting to save.

The specific fieldname that generated brought this to light for me was piwik.create_user. Changing it to piwik:create_user resolved the issue.

A partial traceback is as follows:

  File "/Users/lyndsy/src/osf/framework/guid/model.py", line 54, in save
    return super(GuidStoredObject, self).save(*args, **kwargs)
  File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storedobject.py", line 98, in wrapped
    return func(*args, **kwargs)
  File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storedobject.py", line 72, in wrapped
    return func(this, *args, **kwargs)
  File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storedobject.py", line 809, in save
    self.update_one(self, storage_data=storage_data, saved=True, inmem=True)
  File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storedobject.py", line 98, in wrapped
    return func(*args, **kwargs)
  File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storedobject.py", line 1036, in update_one
    storage_data,
  File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storage/base.py", line 85, in wrapped
    out = func(this, *args, **kwargs)
  File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storage/mongostorage.py", line 175, in update
    multi=True
  File "/Users/lyndsy/.virtualenvs/osf/lib/python2.7/site-packages/pymongo/collection.py", line 479, in update
    check_keys, self.__uuid_subtype), safe)
  File "/Users/lyndsy/.virtualenvs/osf/lib/python2.7/site-packages/pymongo/mongo_client.py", line 920, in _send_message
    rv = self.__check_response_to_last_error(response)
  File "/Users/lyndsy/.virtualenvs/osf/lib/python2.7/site-packages/pymongo/mongo_client.py", line 863, in __check_response_to_last_error
    raise OperationFailure(details["err"], details["code"])
OperationFailure: not okForStorage

How to extend a Collection

How would I go about adding functionality at the collection level using modular-odm?

For example, if I would like to map reduce a collection:

http://api.mongodb.org/python/2.0/api/pymongo/collection.html#pymongo.collection.Collection.map_reduce

make datetimefields timezone-aware

mydatefield = fields.DateTimeField(tz=pytz.UTC)

Add Mongo backend

Add QueueField

N-back queue.

[request] Change redundant names of validators

Instead of:

username = fields.StringField(validate=[validators.MinLengthValidator(8)])

Do this:

username = fields.StringField(validate=[validators.MinLength(8)])

You could just add aliases to retain backwards compatibility.

I'd say the same for fields: e.g. fields.String instead of fields.StringField.

list=True + default value

What is the expected behavior for fields with list set to True and a default value? For example:

class MySchema(StoredObject):
    my_field = StringField(default="lol", list=True)

my_instance = MySchema()
my_instance.my_field  # -> []

I could imagine users expecting my_instance.my_field to include ['lol'], rather than []. If the current behavior is correct, maybe we should throw a warning if users set list=True and provide a default value, since that default will probably never be used.

Document compound queries

If I have two items in a collection, one with a category of 'folder' and the other with a category of 'dashboard', I use a find_one query such as

print "What happens with an exclusive and without the overriding"
no_folder = Folder.find_one(Q('category', 'eq', 'folder') and Q('category', 'eq', 'dashboard'))
print no_folder.name

and instead of finding no items (as I would expect), I get the Dashboard item. If I were to switch the search terms,

print "What happens with an exclusive and without the overriding"
no_folder = Folder.find_one(Q('category', 'eq', 'dashboard') and Q('category', 'eq', 'folder'))
print no_folder.name

I get the folder item.

Saving a migrated error results in DuplicateKeyError

class V1(StoredObject):
    _id = fields.StringField(primary=True, index=True)
    my_string = fields.StringField()
    my_float = fields.FloatField()
    my_number = fields.FloatField()
    my_null = fields.StringField(required=False)
    _meta = {
        'version': 1,
        'optimistic': True
    }
V1.set_storage(_storage)

class V2(StoredObject):
    _id = fields.StringField(primary=True, index=True)
    my_string = fields.StringField()
    my_int = fields.IntegerField(default=5)
    my_number = fields.IntegerField()

    @classmethod
    def _migrate(cls, old, new):
        new.my_string = old.my_string + 'yo'
        if old.my_number:
            new.my_number = int(old.my_number)

    _meta = {
        'version_of': V1,
        'version': 2,
        'optimistic': True
    }
V2.set_storage(_storage)

record = self.V1(my_string='hi', my_float=1.2, my_number=3.4)
record.save()
migrated_record = self.V2.load(self.record._primary_key)
migrated_record.save()

Results in the Traceback:

   Traceback (most recent call last):
    tests/test_migration.py line 82 in test_migrate_all
      self.migrated_record.save()
    modularodm/storedobject.py line 98 in wrapped
      return func(*args, **kwargs)
    modularodm/storedobject.py line 72 in wrapped
      return func(this, *args, **kwargs)
    modularodm/storedobject.py line 754 in save
      self.insert(self._primary_key, storage_data)
    modularodm/storedobject.py line 98 in wrapped
      return func(*args, **kwargs)
    modularodm/storedobject.py line 870 in insert
      cls._storage[0].insert(cls._primary_name, cls._pk_to_storage(key), val)
    modularodm/storage/base.py line 87 in wrapped
      out = func(this, *args, **kwargs)
    modularodm/storage/mongostorage.py line 157 in insert
      self.store.insert(value)
    build/bdist.macosx-10.7-x86_64/egg/pymongo/collection.py line 362 in insert
      self.database.connection)
    build/bdist.macosx-10.7-x86_64/egg/pymongo/mongo_client.py line 969 in _send_message
      rv = self.__check_response_to_last_error(response)
    build/bdist.macosx-10.7-x86_64/egg/pymongo/mongo_client.py line 909 in __check_response_to_last_error
      raise DuplicateKeyError(details["err"], details["code"])
   DuplicateKeyError: E11000 duplicate key error index: modm_test.63e3d2dd.$_id_  dup key: { : "tcyf6" }

Fix __repr__ on Storage classes

MongoStorage::repr raises TypeError:

>>> print storage.MongoStorage(db, 'foo')
TypeError: __repr__ returned non-string (type Cursor)

Scripted creation of projects causes temporary backref fragility

While running a script such as OSF's create_fakes.py, if anyone creates a project by another means (running the script in a separate process, creating a project via the web interface, or similar), some of those projects will not have the node__contributed backref created. For example, if a user created a project while a script was generating fakes, the user created project would not have backrefs. If you ran the same script to generate 10 projects in two different processes, of the 20 projects created, only 10 would have the backrefs.

Remove removes all objects

Assumptions we have a stored object called tag

len(tag.find())  # 20
t = tag.find()[0]
t.remove()
len(tag.find()) # 0

The expected behavior would be that only t is removed from the backend

_id not set upon save

The _id of an object is not getting set upon save. Other fields save correctly. I am using MongoDB for the storage backend.

from pymongo import MongoClient
from modularodm import StoredObject, storage, fields

class Comment(StoredObject):
    _id = fields.StringField(primary=True, index=True)
    text = fields.StringField(required=True)

client = MongoClient()
db = client['testdb']
client.drop_database(db)
Comment.set_storage(storage.MongoStorage(db, "comment"))

c = Comment(text="foo")
c.save()
print(c.text) # => "foo"
print(c._id) # => None

This is confirmed in the mongo shell:

> db.comment.find()[0]
{ "_id" : null, "text" : "foo" }

MultipleBackendMeta returns None -> breaks nosetests

When running the tests with nosetests, most of the tests run except for the tests in test_lazy_load.py. I get the following error:

 ERROR: Failure: AttributeError ('NoneType' object has no attribute '__module__')

   Traceback (most recent call last):
    /Users/sloria1/Envs/modular-odm/lib/python2.7/site-packages/nose/loader.py line 518 in makeTest
      return self._makeTest(obj, parent)
    /Users/sloria1/Envs/modular-odm/lib/python2.7/site-packages/nose/loader.py line 563 in _makeTest
      obj = transplant_class(obj, parent.__name__)
    /Users/sloria1/Envs/modular-odm/lib/python2.7/site-packages/nose/util.py line 637 in transplant_class
      C.__module__ = module
   AttributeError: 'NoneType' object has no attribute '__module__'

I believe this is happening because MultipleBackendMeta.new is returning None.

I'm not sure why only that particular test module is affected. Can someone help me understand what MultipleBackendMeta is supposed to be doing so I can debug this?

Add BDB backend

Incorrect indents in migration print statements

Triple-quotes strings preserve all whitespace characters, so if you do:

def print_foo():
    if bar:
        if baz:
            print '''Lorem ipsum blah blah blah
                     indent shmindent
                    '''

The text will be indented when it is printed to stdout.

Better to use the new print() function for line continuations:

            print("Lorem ipsum blah blah blah"
                    " indent shmindent")

which has the added bonus of being py2/3 compatible.

Make schemas subclassable

Object comparison "gotcha"

@jmcarp and I discovered today that comparing two Stored objects may not always return the expected result when the objects have a DateTime field. This is due to rounding error and the way Mongo stores dates.

# Assume User has a date_created field
user = User('Joe')
user.save()

same_user = User.load(user._id)
user == same_user  # False 
user.date_registered == same_user.date_registered  # False
user._primary_key == same_user._primary_key  # True

Just something to watch out for.

Add blocked words to optimistic insert

No cusses allowed.

pip fails on install

On Ubuntu 12.04, installing osf requirements or running:

pip install modular-odm

Raises the following error:

AttributeError: 'NoneType' object has no attribute 'skip_requirements_regex'

Full traceback:

Downloading/unpacking modular-odm
Running setup.py egg_info for package modular-odm
Traceback (most recent call last):
File "<string>", line 14, in <module>
File "/home/austin/code/OpenScienceFramework/osf/build/modular-odm/setup.py", line 13, in <module>
for req in parse_requirements('requirements.txt')
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1200, in parse_requirements
skip_regex = options.skip_requirements_regex
AttributeError: 'NoneType' object has no attribute 'skip_requirements_regex'
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

File "<string>", line 14, in <module>

File "/home/austin/code/OpenScienceFramework/osf/build/modular-odm/setup.py", line 13, in <module>

for req in parse_requirements('requirements.txt')

File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1200, in parse_requirements

skip_regex = options.skip_requirements_regex

AttributeError: 'NoneType' object has no attribute 'skip_requirements_regex'

----------------------------------------
Command python setup.py egg_info failed with error code 1

Add unique constraint to fields

E.g. myfield = fields.StringField(unique=True)

Auto-expand optimistic insert

After k failed insertions, increase length of optimistically generated IDs.

Expose StoredObject#validate

[feature] Easier setup of storage backend

It gets repetitive to set the the storage for all StoredObject classes, especially when the common use case is to set all models to the same storage type. It would be great to have a convenience method to make this more concise.

I imagine it like so:

store = storage.MongoStorage(db)

class BaseModel(StoredObject):
    '''All classes that inherit from me share the same storage backend.'''
    _storage = store

class User(BaseModel):
    _id = fields.StringField(primary=True)
    _meta = {"optimistic": True}

class Comment(BaseModel):
    _id = fields.StringField(primary=True)
    user = fields.ForeignField(User, backref="comments")
    _meta = {"optimistic": True}

store.set_all()

Invalid fields make hard-to-debug errors

While bouncing around on different branches, I inadvertently got a privatelink added to one of my nodes. This caused a very deep recursion when it couldn't set the field because my new branch did not have a setter for that field. It would be better if the ODM recognized this situation and reported an error about the field, as the current failure mode is very difficult to debug.

Alternate syntax for ListField syntax

Here's our exiting syntax for creating a field that contains a list of integers, with the list's length required to be at least 5 items:

class Foo(StoredObject):
    field = IntegerField(
        list=True,
        list_validate=[MinLengthValidator(5), ]         
)

Here's my proposed syntax:

class Foo(StoredObject)
    field = ListField(
        IntegerField(),
        validate=[MinLengthValidator(5), ],
    ),

This would allow a few things - first, it would rid us of separate validate and list_validate parameters, as the validator could be passed directly to the list field.

Second, a type(Foo.field) would return something like <ListField(IntegerField)>, instead of <IntegerField> today.

I'm sure there are other implications, but I just wanted to get the thought down in writing at this point.

Specify minimum (not exact) version numbers for requirements

c03f267 pins exact version requirements, which may lead to version conflicts when installing into an environment with slightly different requirements.

Add embedded document field

See MongoEngine: https://mongoengine-odm.readthedocs.org/en/latest/guide/defining-documents.html