cos-archives / modular-odm Goto Github PK
View Code? Open in Web Editor NEWA database-agnostic Object-Document Mapper for Python.
License: Apache License 2.0
A database-agnostic Object-Document Mapper for Python.
License: Apache License 2.0
The "db_" prefix added when setting PickleStorage is unexpected behavior. I'd prefer specifying my db names explicitly, or at least have the prefix default to None
or ""
.
In the interest of keeping the package lightweight, I think Flask can be remove as a hard dependency if the only thing that depends on it is the FlaskStoredObject. It would make more sense to release Flask integration as a Flask extension (a la Flask-SQLAlchemy and Flask-Peewee).
E.g.
class User(StoredObject): pass
class Project(StoredObject):
creator = fields.ForeignField(User)
h/t @lyndsysimon
MongoDB fields can't include "."s or begin with "$"s; raise appropriate exceptions when this happens.
A simple query of the database would be good. Right now I'm left with
from modularodm.query.querydialect import DefaultQueryDialect as Q
stuff = Stuff.find(Q('label', 'eq', 'foo'))
I would prefer to just add the field and what I'm searching by as arguments.
If a duplicate key is inserted into the database, different exceptions may be raised, depending on the backend used:
KeyExistsException
DuplicateKeyError
I think it would be better if a single error type were raised, so a user can safely change the storage backend without having to change their error handling code.
Instead of:
from modularodm.query.querydialect import DefaultQueryDialect as Q
Why not just:
from modularodm import Q
Simplified case - no error is generated by the following:
from modularodm import fields, StoredObject
from modularodm.storage.ephemeralstorage import EphemeralStorage
storage = EphemeralStorage()
class Foo(StoredObject):
_id = IntegerField(primary=True, required=False)
Foo.set_storage(storage)
foo = Foo()
foo.save()
foo.to_storage()
# {'_id': None, '_version': 1}
storage
# {None: {'_id': None, '_version': 1}}
Changing the primary key of a StoredObject will currently break relationships. Changing the primary key should either raise an exception or find and modify all relationships with the changed record.
I would expect that None would be returned instead, as of now an error is raised.
When assigning a value to an attribute of a subclass of StoredObject, an exception is not raised when an arbitrary attribute is assigned.
Simplified case:
from modularodm.StoredObject import StoredObject
from modularodm.fields.IntegerField import IntegerField
class Foo(StoredObject):
_id = IntegerField()
foo = Foo()
# this line should raise an AttributeError
foo.not_a_real_attribute = 1
There are currently 18 tests that are failing in test/queries/test_simple_queries.py. All of them pertain to sorting and offsetts.
Let's decide on a license!
@jmcarp and @JeffSpies , since you are the maintainers, you have final say on this.
Instead of
person.firstname = 'Fred'
person.lastname = 'Mercury'
this:
person.update_fields(firstname='Fred', lastname='Mercury')
class Foo(StoredObject):
_id = IntegerField()
att = StringField()
def __init__(self):
self.att = 'foo'
This raises a RuntimeError: maximum recursion depth exceeded
- it should at least tell the user to call StoredObject.__init__()
before item assignment.
In addition to schema__key and schema__key__field, allow retrieval of backreferences (probably as ForeignList) by key.
The installation instructions suggest to:
pip install modular-odm
but that gives an old version of the software. Either the pypi setup needs changing or the instructions do.
A note that wasn't clear from the examples:
If you are using Flask, In order to use PyMongo and also include your db logic in a models file, you must use the app context to initialize the database, so:
with app.app_context():
Text.set_storage(storage.MongoStorage(mongo.db, 'text'))
Might be obvious to more experienced uses of Flask, but wasn't to me initially :).
If you create an instance of a subclass of StorageObject
with MongoDB as the backend, then add a key containing a period to a DictionaryField
an exception is raised upon attempting to save.
The specific fieldname that generated brought this to light for me was piwik.create_user
. Changing it to piwik:create_user
resolved the issue.
A partial traceback is as follows:
File "/Users/lyndsy/src/osf/framework/guid/model.py", line 54, in save
return super(GuidStoredObject, self).save(*args, **kwargs)
File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storedobject.py", line 98, in wrapped
return func(*args, **kwargs)
File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storedobject.py", line 72, in wrapped
return func(this, *args, **kwargs)
File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storedobject.py", line 809, in save
self.update_one(self, storage_data=storage_data, saved=True, inmem=True)
File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storedobject.py", line 98, in wrapped
return func(*args, **kwargs)
File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storedobject.py", line 1036, in update_one
storage_data,
File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storage/base.py", line 85, in wrapped
out = func(this, *args, **kwargs)
File "/Users/lyndsy/.virtualenvs/osf/src/modular-odm/modularodm/storage/mongostorage.py", line 175, in update
multi=True
File "/Users/lyndsy/.virtualenvs/osf/lib/python2.7/site-packages/pymongo/collection.py", line 479, in update
check_keys, self.__uuid_subtype), safe)
File "/Users/lyndsy/.virtualenvs/osf/lib/python2.7/site-packages/pymongo/mongo_client.py", line 920, in _send_message
rv = self.__check_response_to_last_error(response)
File "/Users/lyndsy/.virtualenvs/osf/lib/python2.7/site-packages/pymongo/mongo_client.py", line 863, in __check_response_to_last_error
raise OperationFailure(details["err"], details["code"])
OperationFailure: not okForStorage
How would I go about adding functionality at the collection level using modular-odm?
For example, if I would like to map reduce a collection:
mydatefield = fields.DateTimeField(tz=pytz.UTC)
N-back queue.
Instead of:
username = fields.StringField(validate=[validators.MinLengthValidator(8)])
Do this:
username = fields.StringField(validate=[validators.MinLength(8)])
You could just add aliases to retain backwards compatibility.
I'd say the same for fields: e.g. fields.String
instead of fields.StringField
.
What is the expected behavior for fields with list set to True and a default value? For example:
class MySchema(StoredObject):
my_field = StringField(default="lol", list=True)
my_instance = MySchema()
my_instance.my_field # -> []
I could imagine users expecting my_instance.my_field to include ['lol'], rather than []. If the current behavior is correct, maybe we should throw a warning if users set list=True and provide a default value, since that default will probably never be used.
If I have two items in a collection, one with a category of 'folder' and the other with a category of 'dashboard', I use a find_one query such as
print "What happens with an exclusive and without the overriding"
no_folder = Folder.find_one(Q('category', 'eq', 'folder') and Q('category', 'eq', 'dashboard'))
print no_folder.name
and instead of finding no items (as I would expect), I get the Dashboard item. If I were to switch the search terms,
print "What happens with an exclusive and without the overriding"
no_folder = Folder.find_one(Q('category', 'eq', 'dashboard') and Q('category', 'eq', 'folder'))
print no_folder.name
I get the folder item.
class V1(StoredObject):
_id = fields.StringField(primary=True, index=True)
my_string = fields.StringField()
my_float = fields.FloatField()
my_number = fields.FloatField()
my_null = fields.StringField(required=False)
_meta = {
'version': 1,
'optimistic': True
}
V1.set_storage(_storage)
class V2(StoredObject):
_id = fields.StringField(primary=True, index=True)
my_string = fields.StringField()
my_int = fields.IntegerField(default=5)
my_number = fields.IntegerField()
@classmethod
def _migrate(cls, old, new):
new.my_string = old.my_string + 'yo'
if old.my_number:
new.my_number = int(old.my_number)
_meta = {
'version_of': V1,
'version': 2,
'optimistic': True
}
V2.set_storage(_storage)
record = self.V1(my_string='hi', my_float=1.2, my_number=3.4)
record.save()
migrated_record = self.V2.load(self.record._primary_key)
migrated_record.save()
Results in the Traceback:
Traceback (most recent call last):
tests/test_migration.py line 82 in test_migrate_all
self.migrated_record.save()
modularodm/storedobject.py line 98 in wrapped
return func(*args, **kwargs)
modularodm/storedobject.py line 72 in wrapped
return func(this, *args, **kwargs)
modularodm/storedobject.py line 754 in save
self.insert(self._primary_key, storage_data)
modularodm/storedobject.py line 98 in wrapped
return func(*args, **kwargs)
modularodm/storedobject.py line 870 in insert
cls._storage[0].insert(cls._primary_name, cls._pk_to_storage(key), val)
modularodm/storage/base.py line 87 in wrapped
out = func(this, *args, **kwargs)
modularodm/storage/mongostorage.py line 157 in insert
self.store.insert(value)
build/bdist.macosx-10.7-x86_64/egg/pymongo/collection.py line 362 in insert
self.database.connection)
build/bdist.macosx-10.7-x86_64/egg/pymongo/mongo_client.py line 969 in _send_message
rv = self.__check_response_to_last_error(response)
build/bdist.macosx-10.7-x86_64/egg/pymongo/mongo_client.py line 909 in __check_response_to_last_error
raise DuplicateKeyError(details["err"], details["code"])
DuplicateKeyError: E11000 duplicate key error index: modm_test.63e3d2dd.$_id_ dup key: { : "tcyf6" }
MongoStorage::repr raises TypeError:
>>> print storage.MongoStorage(db, 'foo')
TypeError: __repr__ returned non-string (type Cursor)
While running a script such as OSF's create_fakes.py, if anyone creates a project by another means (running the script in a separate process, creating a project via the web interface, or similar), some of those projects will not have the node__contributed backref created. For example, if a user created a project while a script was generating fakes, the user created project would not have backrefs. If you ran the same script to generate 10 projects in two different processes, of the 20 projects created, only 10 would have the backrefs.
Assumptions we have a stored object called tag
len(tag.find()) # 20
t = tag.find()[0]
t.remove()
len(tag.find()) # 0
The expected behavior would be that only t is removed from the backend
The _id
of an object is not getting set upon save. Other fields save correctly. I am using MongoDB for the storage backend.
from pymongo import MongoClient
from modularodm import StoredObject, storage, fields
class Comment(StoredObject):
_id = fields.StringField(primary=True, index=True)
text = fields.StringField(required=True)
client = MongoClient()
db = client['testdb']
client.drop_database(db)
Comment.set_storage(storage.MongoStorage(db, "comment"))
c = Comment(text="foo")
c.save()
print(c.text) # => "foo"
print(c._id) # => None
This is confirmed in the mongo shell:
> db.comment.find()[0]
{ "_id" : null, "text" : "foo" }
When running the tests with nosetests
, most of the tests run except for the tests in test_lazy_load.py
. I get the following error:
ERROR: Failure: AttributeError ('NoneType' object has no attribute '__module__')
Traceback (most recent call last):
/Users/sloria1/Envs/modular-odm/lib/python2.7/site-packages/nose/loader.py line 518 in makeTest
return self._makeTest(obj, parent)
/Users/sloria1/Envs/modular-odm/lib/python2.7/site-packages/nose/loader.py line 563 in _makeTest
obj = transplant_class(obj, parent.__name__)
/Users/sloria1/Envs/modular-odm/lib/python2.7/site-packages/nose/util.py line 637 in transplant_class
C.__module__ = module
AttributeError: 'NoneType' object has no attribute '__module__'
I believe this is happening because MultipleBackendMeta.new is returning None
.
I'm not sure why only that particular test module is affected. Can someone help me understand what MultipleBackendMeta
is supposed to be doing so I can debug this?
Triple-quotes strings preserve all whitespace characters, so if you do:
def print_foo():
if bar:
if baz:
print '''Lorem ipsum blah blah blah
indent shmindent
'''
The text will be indented when it is printed to stdout.
Better to use the new print()
function for line continuations:
print("Lorem ipsum blah blah blah"
" indent shmindent")
which has the added bonus of being py2/3 compatible.
@jmcarp and I discovered today that comparing two Stored objects may not always return the expected result when the objects have a DateTime field. This is due to rounding error and the way Mongo stores dates.
# Assume User has a date_created field
user = User('Joe')
user.save()
same_user = User.load(user._id)
user == same_user # False
user.date_registered == same_user.date_registered # False
user._primary_key == same_user._primary_key # True
Just something to watch out for.
No cusses allowed.
On Ubuntu 12.04, installing osf requirements or running:
pip install modular-odm
Raises the following error:
AttributeError: 'NoneType' object has no attribute 'skip_requirements_regex'
Full traceback:
Downloading/unpacking modular-odm
Running setup.py egg_info for package modular-odm
Traceback (most recent call last):
File "<string>", line 14, in <module>
File "/home/austin/code/OpenScienceFramework/osf/build/modular-odm/setup.py", line 13, in <module>
for req in parse_requirements('requirements.txt')
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1200, in parse_requirements
skip_regex = options.skip_requirements_regex
AttributeError: 'NoneType' object has no attribute 'skip_requirements_regex'
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 14, in <module>
File "/home/austin/code/OpenScienceFramework/osf/build/modular-odm/setup.py", line 13, in <module>
for req in parse_requirements('requirements.txt')
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1200, in parse_requirements
skip_regex = options.skip_requirements_regex
AttributeError: 'NoneType' object has no attribute 'skip_requirements_regex'
----------------------------------------
Command python setup.py egg_info failed with error code 1
E.g. myfield = fields.StringField(unique=True)
After k failed insertions, increase length of optimistically generated IDs.
It gets repetitive to set the the storage for all StoredObject classes, especially when the common use case is to set all models to the same storage type. It would be great to have a convenience method to make this more concise.
I imagine it like so:
store = storage.MongoStorage(db)
class BaseModel(StoredObject):
'''All classes that inherit from me share the same storage backend.'''
_storage = store
class User(BaseModel):
_id = fields.StringField(primary=True)
_meta = {"optimistic": True}
class Comment(BaseModel):
_id = fields.StringField(primary=True)
user = fields.ForeignField(User, backref="comments")
_meta = {"optimistic": True}
store.set_all()
While bouncing around on different branches, I inadvertently got a privatelink added to one of my nodes. This caused a very deep recursion when it couldn't set the field because my new branch did not have a setter for that field. It would be better if the ODM recognized this situation and reported an error about the field, as the current failure mode is very difficult to debug.
Here's our exiting syntax for creating a field that contains a list of integers, with the list's length required to be at least 5 items:
class Foo(StoredObject):
field = IntegerField(
list=True,
list_validate=[MinLengthValidator(5), ]
)
Here's my proposed syntax:
class Foo(StoredObject)
field = ListField(
IntegerField(),
validate=[MinLengthValidator(5), ],
),
This would allow a few things - first, it would rid us of separate validate
and list_validate
parameters, as the validator could be passed directly to the list field.
Second, a type(Foo.field)
would return something like <ListField(IntegerField)>
, instead of <IntegerField>
today.
I'm sure there are other implications, but I just wanted to get the thought down in writing at this point.
c03f267 pins exact version requirements, which may lead to version conflicts when installing into an environment with slightly different requirements.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.