Add dynamic documents

[hmarr]

Add dynamic documents

[joshourisman]

Has there been any progress on this by any chance? I'm working on a project that essentially depends on this ability which is why we're planning on using MongoDB in the first place. I'd prefer to be able to use mongoengine rather than pymongo directly, so I'm going to give it a go of implementing this myself, but don't want to duplicate effort if anyone else has already started on it.

[hmarr]

I'm not aware of anyone working on this, so that'd be great if you're going to have a go at it. I guess the main issue is that we can't just assume that any attribute set on a document should be stored in the database, as there are quite often other attributes present on a document. Perhaps this could be resolved by using a method to set dynamic fields, or maybe dictionary-style access. Also, unless the type could some how be specified, all the conversions between Python and MongoDB types (such as on custom field types) would be skipped. I'd be interested to hear your thoughts.

[joshourisman]

I was thinking of using a method to set dynamic fields or alternatively inspecting the object on save() for attributes that are mongoengine fields but aren't included in the known fields (or possibly to check for a dynamic_fields dict and validate its attributes). In either case, I would want it to require actually using a BaseField subclass to avoid the issue of ambiguous datatypes.

[hmarr]

Ah right, one possible problem is that field values aren't stored in the field objects, they are stored in the _data dictionary on a Document. Also, even if Fields are used when dynamic values are inserted into the database, how will we know which field to use when it comes out (for conversion back to Python types)?
Cheers,
Harry

[joshourisman]

Ah, very good point. First thing that comes to mind is to perhaps store a tuple with a BaseField subclass name and the mongo value of the data, rather than just the data, although at that point you might as well create a TupleField class and store your 'dynamic' fields in a ListField of TupleFields. Second thought is to perhaps specify the field type in the field name; it would be easy enough to look at the keys in the dict returned by mongo to determine whether a field returned by the database is actually defined in the model and then extract the class of the field from the name if it's not.

Actually, I might try and do both, the TupleField idea could be useful in addition to the dynamic fields.

[joshourisman]

Would need to be at least a 3-tuple so we could store the BaseField subclass, any arguments that get passed to init(), and the actual data.

[hmarr]

That could work. What's your use case for this? We need to decide how "dynamic" documents need to be, and example use cases for dynamic documents may help figure that out. Another possibility is just to say that there's often little point in trying to validate schema-free documents, and just pass to the database exactly what we're given, and let PyMongo handle the type conversions for these extra attributes - sort of like DictFields..

[joshourisman]

Our use case is essentially free-form forms on top of Django. We want users to be able to define forms that use an arbitrary combination of pre-defined form fields, so we have no idea what any given form will look like or what number of what kind of fields it might have.

Skipping the validation stage actually would make a lot of sense for us since we can validate the data in Python before we even think about saving it to the database. That was essentially my original plan before I discovered mongoengine and was planning on working directly with PyMongo.

[hmarr]

Ok cool, so I guess a simple method and/or dictionary-style access would be the best bet for adding the data to the document. Maybe just adding BaseFields for each of the unrecognised fields that come back from the DB would work.

[joshourisman]

I've completed a basic implementation of dynamic Documents in my fork. It uses a special create_dynamic_field() method to create the field initially, but afterwords works transparently using the normal Document API by also storing an internal list of the dynamic fields in the database. I've also included tests for the new functionality that all pass (seemingly validly).

Changeset is joshourisman/mongoengine@d705fda8980134c5630f76b95eb0cb2f7e231f2f

[joshourisman]

Ok, appears there's still some issues in my code... It's not properly storing and retrieving the list of dynamic fields, which means it's not properly storing and retrieving the actual dynamic fields, though it appears to work when everything's running in a single session, such as when running tests.

[joshourisman]

Alright, now it seems to actually be working! I'm not entirely sure what was going wrong before, but it was failing to add the data for the dynamic fields to the _data dictionary even though everything else seemed to be working fine. I ended up just setting that manually in the init, though there may be a better way to do it...

Changeset is joshourisman/mongoengine@af7bbd3aca4e0b306c74ebe27ba74f354254b40f

[rafa-munoz]

Hi joshourisman, I forked your project to merge it with trunk :)

Here's the pull request:

https://github.com/hmarr/mongoengine/issues#issue/112

[rozza]

I should have closed this one - its done