pylons/colander

A serialization/deserialization/validation library for strings, mappings and lists.

Pylons

Pylons is a rapid web application development framework.

Note

Pylons has merged with repoze.bfg, and is now in maintenance-only mode. It's highly recommended that new projects start with the new merged web framework, pyramid.

Install

Read the online Installation instructions.

If you want to install from source you can run the following command:

This will display a message and download setuptools if the module is not already installed. It will then install Pylons and all its dependencies. You may need root privileges to install setuptools.

Testing

To test the source distribution run the following command:

This will install additional dependencies needed for the tests. As above, you may need root privileges.

Documentation

Read the complete Pylons web framework documentation.

Definitive Guide to Pylons is a book about Pylons published by Apress, written by James Gardner, with free HTML rendering.

Generating documentation requires Sphinx:

Then to build the documentation use the commands:

colander's Issues

Can't define declarative schema with non python identifiers as name

Here's an example

        class FnordSchema(colander.Schema):
            fnord = colander.SchemaNode(
                colander.Sequence(), 
                colander.SchemaNode(colander.Integer(), name='integer in sequence'), 
                name="fnord[]"))

Instead I have to use this workaround:

        class FnordSchema(colander.Schema):
        schema = FnordSchema()
        schema.add(colander.SchemaNode(
                colander.Sequence(), 
                colander.SchemaNode(colander.Integer(), name='integer in sequence'), 
                name="fnord[]"))

Ideally the parsed value would be available under the name 'fnord' instead of 'fnord[]', but that would be an exciter. :-)

PO files not compiled in 1.0a2?

I noticed that the German translations for colander validation error messages displayed in my application do not match the translations contained in colander.po. Especially, some translation mistakes that seem to have been fixed in the PO files still appear in the application.

Perhaps the PO files have not been compiled to MO files before release? (I obtained colander 1.0a2 from pypi using pip install.)

Translation strings not extracted in .pot file then not translated

On version 1.0a2, it seems that few messages are not extracted in the .pot file and then are not translated.
For exemple, the message 'Must be a URL' on line 413 does not appear in the colander.pot file.

Email validator accepts bad email addresses.

Examples of invalid addresses accepted by validator:
[email protected]..pl
[email protected]

Better regular expression: (?i)^[A-Z0-9._%+-]+@[A-Z0-9]+([.-][A-Z0-9]+)*.[A-Z]{2,4}$

Recursive schemas are currently not possible declaratively

When a schema, let's call it SchemaA, contains a sequence, where the child schema is SchemaA, one of the classes(either SchemaA, or the sequence schema) will not be found, as it hasn't been defined yet. An example of this is the following:

class CommentSchema(MappingSchema):
    solution_id = SchemaNode(Int(),  missing=None)
    parent_id = SchemaNode(Int(),  missing=None)
    replies = CommentsSchema()
    revisions = CommentRevisionsSchema(missing=None)
    votes = CommentVotesSchema(missing=None)

class CommentsSchema(SequenceSchema):
     comment = CommentSchema()

I have written a public api, to deal with this issue, that would be used like so:

CommentSchema.add_node(CommentsSchema(missing=None, name='replies'))

class CommentSchema(MappingSchema):
    solution_id = SchemaNode(Int(), missing=None)
    parent_id = SchemaNode(Int(), missing=None)
    revisions = CommentRevisionsSchema(missing=None)
    votes = CommentVotesSchema(missing=None)

class CommentsSchema(SequenceSchema):
     comment = CommentSchema()

CommentSchema.add_node(CommentsSchema(missing=None, name='replies'))

The problem is, is that at the point where CommentSchema is instantiated on CommentsSchema, CommentSchema does not yet have the replies attribute, so this results in the Schema structure to not be fully recursive, as it just goes down one level.

The new support for default=drop just doesn't work

Just copy-pasting the example from CHANGES.txt:

import colander

class What(colander.MappingSchema):
    thing = colander.SchemaNode(colander.String(), default=colander.drop,
        missing=None)

result = What().serialize({}) # no "thing" in mapping
assert result == {}

Result is actually:

{'thing': u'<colander._drop object at 0x1f26190>'}

(ie. the drop object cast to a string)

How do I use `accept_scalar`?

From the docs, I can't figure out how the accept_scalar argument can be used in my schemas.

I'd like something like this:

from colander import Date
from colander import SequenceSchema
class DatesSchema(SequenceSchema):
    date = Date()

DatesSchema().deserialize(['2013-02-14', '2013-02-15'])
DatesSchema().deserialize('2013-02-14') # can't get this one to work.

If it's any help, I'm trying to use Colander to validate an URL query string and I can't determine before passing the results of urlparse.parse_qs() to Colander if the passed in value should be a list or not. Query arguments with a single value can either be treated as 1-item lists or scalars. If I set them as 1-item lists, then all scalar values don't work. If I set them as scalars, then I can't pass in a 1-item list (it's converted as a scalar and rejected by Colander).

For example:

• /foo?dates=2013-02-14,2013-02-15 # I got this to work
• /foo?dates=2013-02-14 # This ends up as a scalar and it's rejected.

Slightly incorrect German translations

The German colander.po has:

msgid "Shorter than minimum length ${min}"
msgstr "Kürzer als erlaubt, die Mindestlänge beträgt ${min} Zeichen"

msgid "Longer than maximum length ${max}"
msgstr "Länger als erlaubt, die Maximallänge beträgt ${max} Zeichen"

The English version is non-specific as to whose length is too short/long. However, the German version specifies "Zeichen" (characters), which is incorrect if the Range validator is used for sequences etc.

The simples solution would be to simply leave out the word "Zeichen" completely.

Deferred Declarative Properties

This is actually kind of a big feature, as it would make it possible to support all possible types of relationships between schemas declaratively. I've been looking into how to implement it, but can't find any obvious ways. Going to continue to try, but just wanted to make it known.

Allow deferred default to execute per schema in sequence

The default using a deferred is only created once during schema initialization. This isn't that great when using SchemaNode's in a Sequence to provide defaults since they all get the same default. In my case I need each one to have a random int applied, and they all get the same int.

It'd be great if there was some way to indicate a deferred default should run for each object in the sequence.

IndexError: list index out of range in Colander 0.8.1

Getting the following error with 0.8.1

Module haufe.adb.main.deform_util, line 173, in render_form
Module deform.field, line 419, in render
Module colander, line 1296, in serialize
Module colander, line 452, in serialize
Module colander, line 423, in _impl
Module colander, line 450, in callback
Module colander, line 1296, in serialize
Module colander, line 452, in serialize
Module colander, line 423, in _impl
Module colander, line 450, in callback
Module colander, line 1296, in serialize
Module colander, line 654, in serialize
Module colander, line 615, in _impl
IndexError: list index out of range

This happens because one particular node has no children:

(Pdb) print node
<colander.SchemaNode object at 46912531444688 (named geschaeftsfelder)>
(Pdb) print node.children
[]

the node is defined as

geschaeftsfelder = ListOfIntegers(
    title=u'Redaktion(en)',
    missing='',
    widget=selection_geschaeftsfelder,
    ro_widget=MultiSelectWidget(),
    view_roles=[u'AutorenRed', u'AutorenFibu'],
    edit_roles=[],
    create_roles=[u'AutorenRed'],
)

and ListOfIntegers defined as

class ListOfIntegers(colander.SequenceSchema):
item = colander.SchemaNode(
colander.Int(),
)

Why does Colander expect that a node must have at least one child:

613 for num, subval in enumerate(value):$
614 try:$
615 result.append(callback(node.children[0], subval))$

?

colander installs stray .po & .pot files

--- /usr/lib64/python2.7/site-packages/colander/locale/
>>> /usr/lib64/python2.7/site-packages/colander/locale/colander.pot
--- /usr/lib64/python2.7/site-packages/colander/locale/cs/
--- /usr/lib64/python2.7/site-packages/colander/locale/cs/LC_MESSAGES/
>>> /usr/lib64/python2.7/site-packages/colander/locale/cs/LC_MESSAGES/colander.mo
>>> /usr/lib64/python2.7/site-packages/colander/locale/cs/LC_MESSAGES/colander.po
--- /usr/lib64/python2.7/site-packages/colander/locale/de_DE/
--- /usr/lib64/python2.7/site-packages/colander/locale/de_DE/LC_MESSAGES/
>>> /usr/lib64/python2.7/site-packages/colander/locale/de_DE/LC_MESSAGES/colander.mo
>>> /usr/lib64/python2.7/site-packages/colander/locale/de_DE/LC_MESSAGES/colander.po

etc.

The .po and .pot files are merely sources for messages and are not used by anything in runtime. Therefore, they shouldn't be installed.

Documentation bug in "Serialization Combinations" table

In the table available here:
http://docs.pylonsproject.org/projects/colander/en/latest/null.html#serialization-combinations
The first and last lines of the table are contradictory ("value serialized" versus "null serialized").

Add support for json schema

It would be great if colander models would be able to generate json schema. This way it would be possible to use any kind of form library that supports json schema to render forms, making it more standard and compatible.

collander.All validator do not work with deferred validators

It throws exception
AttributeError: 'int' object has no attribute 'get'

Add to_json function, that would eventually output json

As we already know result from serializer resemble json, but not really(every element has eventually string type). I think a function is needed that would serialize a data and output json, because that's what you need in REST services.

@plamut is working on json schema in #112, but there's not even a proper function that would output json. I would do it, but i don't know what's the proper way, because serializer does not give me type information, so output from it is not very usefull.
As far as i understand a parallel implementation(to serializtion) would be needed, but wouldn't it be just simpler for serializer to output json and preserve type information this way, or are there any other issues i'm not aware of?

TypeError at deserializing malformed Date and DateTime

Version 1.0a2 is ok, it raises Invalid.

>>>colander.SchemaNode(colander.DateTime()).deserialize('2013/05/31')
Traceback (most recent call last):
 ...
 ...
colander.Invalid: {'': u'Invalid date'}

Version 1.0a3 crashes, and raises TypeError:

>>> colander.SchemaNode(colander.DateTime()).deserialize('2013/05/31')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mathieu/Code/daybed/src/local/lib/python2.7/site-packages/colander/__init__.py", line 1864, in deserialize
    appstruct = self.typ.deserialize(self, cstruct)
  File "/home/mathieu/Code/daybed/src/local/lib/python2.7/site-packages/colander/__init__.py", line 1492, in deserialize
    cstruct, default_timezone=self.default_tzinfo)
  File "/home/mathieu/Code/daybed/src/local/lib/python2.7/site-packages/colander/iso8601.py", line 137, in parse_date
    return datetime(int(groups["year"]), int(groups["month"]), int(groups["day"]),
TypeError: int() argument must be a string or a number, not 'NoneType'

Serialization of `None` is now very inconsistent

The serialization of None is very inconsistent since 1.0b1, because #513d860 reverted part of the special cases for None:

• String: "None"
• Boolean: False
• Date, Time, DateTime: null
• Integer, Float, Decimal, Money: error
• GlobalObject: error
• Mapping, Tuple, Sequence: error
• Set, List: None

In our schemas, we have definitions like:

result = SchemaNode(String(),
    title="Result",
    validator=OneOf([ None, "success", "failure" ]), missing=None,
)

(ie. result can be undefined, "success" or "failure") that are broken because None is now serialized to "None" instead of passing through as null

Any idea how to move forward from here, and what is the correct behavior?

colander.deferred loses wrapped object's __doc__ etc

For sphinx autodoc docs, colander.deferred should try to retain as much info about the wrapped object as possible so it can be rendered.

What about deferred SchemaTypes?

Hey guys. I've had this idea on my head for some time. What do you think about supporting deferred schema types?

I mean, colander atually supports them, all you have to do is inherit from colander.deferred and override __call__.

But since SchemaType.bind binds the attributes in an undefined order, something that depends on the type of the node can be binded before the type itself and break.

So I concluded that if the type was guaranteed to always be binded before anything else, It would be fine to have deferred schema types.

I think this could be useful while writing colander extensions, since sometimes is better to let the clients use the dialect colander.SchemaNode(CustomType()) than forcing them to use a custom node mylib.SchemaNode().

What do you think?

Message argument to validators

See http://docs.pylonsproject.org/projects/colander/en/latest/api.html#validators

Currently some require msg and some message. Would be nice to go for one and deprecate the other.

Serializing datetime data from UTC to localtime trouble

I have set colander.DateTime to use a timezone (offset +2 and day light savings). I did this by subclassing th tzinfo in python.

When i deserialize a datetime string of "2013-08-16 07:00:00" it creates a datetime object of datetime.datetime(2013, 8, 16, 7, 0, tzinfo=TimeZone object at...) which prints out as "2013-08-16T07:00:00+03:00".

Data is then stored in MongoDB as ISODate("2013-08-16T04: 00: 00.0Z")-object. Which in my mind seems right, so all the datetimes in database are in UTC.

When i try to serialize the saved datetime back to string the database hands the appstruct to colander in the form of datetime.datetime(2013, 8, 16, 4, 0) without timezone info. So I would assume it should be handled as UTC, right? But when the data is passed to colander to serialize, it adds the timezone and the result is
"2013-08-16T04:00:00+03:00".

Am I missing something here or is this a bug?

Here's my code, in case needed.

class TimeZone(tzinfo):
    def __init__(self, offset, isdst, name):
        self.offset = offset
        self.isdst = isdst
        self.name = name
    def utcoffset(self, dt):
        return timedelta(hours=self.offset) + self.dst(dt)
    def dst(self, dt):
        return timedelta(hours=1) if self.isdst else timedelta(0)
    def tzname(self, dt):
        return self.name

timezone = TimeZone(+2, True, "Finland/Helsinki")

class Data(colander.MappingSchema):
    time = colander.SchemaNode(
        colander.DateTime(timezone)
        )

Documentation fails to build

$ make html
mkdir -p _build/html _build/doctrees
sphinx-build -b html -d _build/doctrees  -W . _build/html
Running Sphinx v1.1.2
Cloning into '_themes'...
remote: Counting objects: 315, done.
remote: Compressing objects: 100% (239/239), done.
remote: Total 315 (delta 146), reused 199 (delta 74)
Receiving objects: 100% (315/315), 455.44 KiB | 264 KiB/s, done.
Resolving deltas: 100% (146/146), done.
running test
running egg_info
creating colander.egg-info
writing requirements to colander.egg-info/requires.txt
writing colander.egg-info/PKG-INFO
writing top-level names to colander.egg-info/top_level.txt
writing dependency_links to colander.egg-info/dependency_links.txt
writing manifest file 'colander.egg-info/SOURCES.txt'
reading manifest file 'colander.egg-info/SOURCES.txt'
writing manifest file 'colander.egg-info/SOURCES.txt'
running build_ext
..................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 290 tests in 0.123s

OK
loading pickled environment... not yet created
building [html]: targets for 10 source files that are out of date
updating environment: 10 added, 0 changed, 0 removed
reading sources... [100%] null                                                                                                                               

Warning, treated as error:
/tmp/colander/docs/null.rst:191: ERROR: Error in "note" directive:
invalid option block.

.. note:: ``<missing>`` in the above table represents the circumstance
   in which a key present in a :class:`colander.MappingSchema` is not
   present in a mapping passed to its
   :meth:`colander.SchemaNode.serialize` method.  In reality,
   ``<missing>`` means exactly the same thing as
   :attr:`colanderr.null`, because the :class:`colander.Mapping` type
   does the equivalent of ``mapping.get(keyname, colander.null)`` to
   find a subvalue during serialization.

make: *** [html] Error 1

Unable to set Mapping's "uknown" attribute with MappingSchema

When created imperatively, I can set a Mapping's "unknown" attribute:

my_mapping = SchemaNode(Mapping(unknown='raise'))
my_mapping.add(...)

When created declaratively, there's no way to do this, though:

class MyMapping(MappingSchema):  # :(
    ...

The same applies to Sequence's "scalar" attribute and SequenceSchema.

Invalid date exception is raised when datetime.now() is passed to deserialize

if date string is passed, deserialize() works fine. However passing datetime.now() raises Invalid date exception.

import colander
import datetime
class Schema(colander.MappingSchema):
... dt = colander.SchemaNode(colander.DateTime())
...
sch = Schema()
val = sch.deserialize({'dt': '2011-08-29 11:30:44.212639'})
val
{'dt': datetime.datetime(2011, 8, 29, 11, 30, 44, 212639, tzinfo=<iso8601.iso8601.Utc object at 0x237a3d0>)}
val = sch.deserialize({'dt': datetime.datetime.now()})
Traceback (most recent call last):
File "", line 1, in
raise error
Invalid: {'dt': u'Invalid date'}

Enable multi-field / field comparison validation

It would be great to be able to add a validation that compares the value of two fields. For example, checking that two password fields are equal, or requiring phone only if email is null.

False positive validation for strings

I doubt that I am the first one to see this, so it might be a feature. But it seems like a bug to me. Apologies if I am missing something.

IMO the call to deserialize should fail in this example below. But instead the "name" dictionary of the input is converted to a string:

import colander 

class BizCard(colander.MappingSchema):
    name = colander.SchemaNode(colander.String())

json_data = {
    "name" : {
            "first": "John",
            "last": "Doe"
        }
}

schema = BizCard()
schema.deserialize(json_data)

(tested against version 1.0a2)

Colander serializes numbers & bools as strings.

When I use the Bool and Int types in my schemas like so:

class TestSchema(MappingSchema):
    interested = SchemaNode(Boolean(), missing=False, default=False)

And I use it to serialize some data like so:

schema = TestSchema()
data = schema.serialize(dict(interested=True))

Then data['interested']' is"true"` (a string).

Now, if I json.dumps(data), I get strings in my JSON data instead of the boolean I requested. This is really annoying because there's no way to fix this reliably after the fact! At best, I can supply a custom JSONEncoder, but I can still accidentally convert strings that shouldn't be converted.

The same problem exists with numbers.

Design choices about deserializing empty string as colander.null?

I was wondering why, in the deserialize method of class colander.String, the empty string is considered as colander.null?

I think that behavior looses precious information. I am using colander.String in combination with Deform and let’s say I have a simple optional text input.

In case an empty string is submitted by the user, then I would like to empty the corresponding string in my business data.

In case no text input is submitted by the user, either because there is a bug which failed to generate this HTML input element or because the user is malicious and he deliberately removed the HTML input element, then I would like not to alter the corresponding string in my business data.

However, as colander.String is considering both cases the same way then I cannot discriminate those 2 cases and I have to choose the "less bad" action to perform in case of a colander.null and it results in one of those disadvantages:

1. Take the risk to erase business data in case my code is buggy.
2. Prevent the user to empty the string

So I was wondering what were the reasons for that design choice?

Documentation outdated concerning schema_order

Apparently, schema_order has been "replaced" by something like insert_before (See 9962e03). However, the documentation is still mentioning schema_order.

Inconsistent parameter names

Function accepts message, but most other classes seem to accept msg.

String() serializes empty string to colander.null

>>> colander.String().serialize(node, '')    # buggy behavior
<colander.null>
>>> colander.String().serialize(node, None)  # expected behavior
<colander.null>

This seems to be a "regression" from #45

While it makes sense that there is no obvious string representation of None, it seems intuitive to me that serializing a string would result in a string.

With the current behavior, it is not possible to have a String type SchemaNode with a default of empty string, e.g.:

class Foo(colander.MappingSchema):
    bar = colander.SchemaNode(colander.String(), default='')

would give this:

>>> Foo().serialize({})
{'bar': <colander.null>}

Thank you for your time and clarification.

Can't deserialize empty string

string type treats empty string as missing value

node = colander.SchemaNode(
colander.String(),
name = 'str',
default = '')

node.deserialize('')
^^^^^^^^^^^^^^^^^ raises required exception

Support multiple validators, but stop at first exception

It would be handy if there would be something like All that would stop at the first validation error.

Some validations are quick & easy, like Email, but some can be pretty expensive, like checking if the email address is already in use. There's no need for that check, if the email is not valid in the first place.

I'm happy to attach a pull request, if somebody would be smart enough to suggest a name.

An alternative would be to support a list or tuple of validators as the validator argument to SchemaNode and stop at the first failure by default.

Or maybe I'm just missing the obvious way to do this, in which case the docs should be updated (which I'm happy to do once somebody explains me how to do what I want).

SchemaNode constructor doesn't accept typ as a keyword argument

Constructing SchemaNodes as follows:

colander.SchemaNode(typ=colander.Integer)

Leads to a "NotImplementedError: SchemaNode construction without a typ argument ...". Ideally, the SchemaNode constructor would have code somewhat like this:

if arg:
    self.typ = arg[0]
    _add_node_children(self, arg[1:])
elif 'typ' in kw:
    self.typ = kw['typ']
else:
    self.typ = self.schema_type()

That would allow constructing SchemaNodes from dictionaries of arbitrary keyword arguments.

Translation domains of messages passed to validators

Take a look at __call__ methods of Range validator:

if self.min is not None:
    if value < self.min:
        min_err = _(self.min_err, mapping={'val':value, 'min':self.min})
        raise Invalid(node, min_err)

Shoudn't there be:

        min_err = translationstring.TranslationString(self.min_err, 
                domain=self.min_err.domain, 
                default=self.min_err.default,
                mapping={'val':value, 'min':self.min})

How should the default keyword on SchemaNode be used?

I am trying to specify an optional field, but it raises colander.Invalid when the value is empty, even if I passed a default value to the constructor.

Code: http://pastebin.com/VBuBy32X

Unbound deferred validators are ignored?

On IRC this morning, subsu pointed out to me that unbound deferred validators are silently ignored when deserializing. Indeed, this behavior is documented.

This seems completely backwards to me. The validators are supposed to make particular assertions about the integrity of the unserialized data. If I forget to bind one, it is a programming error, and I want an exception thrown. I certainly don't want untrusted input to be passed through unvalidated. This is freaking me out, man!

Is this really what was intended? (Why?) Anyhow, I propose either: changing this behavior (which might be problematic, given that it is documented and all); or adding a second deferred (sub)class (compulsory_deferred or some such) for which errors will be thrown if an attempt is made to use them before they are bound.

Test failure with Python 3.3

Colander 0.9.9 introduced test_serialize_quantize_no_rounding(), which fails with Python 3.3. It passes with older versions of Python.

$ python3.3 setup.py test
...
======================================================================
ERROR: test_serialize_quantize_no_rounding (colander.tests.test_colander.TestDecimal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/colander-0.9.9/colander/__init__.py", line 1035, in serialize
    return str(self.num(appstruct))
  File "/tmp/colander-0.9.9/colander/__init__.py", line 1109, in num
    result = result.quantize(self.quant, self.rounding)
TypeError: an integer is required

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/colander-0.9.9/colander/tests/test_colander.py", line 1371, in test_serialize_quantize_no_rounding
    result = typ.serialize(node, val)
  File "/tmp/colander-0.9.9/colander/__init__.py", line 1039, in serialize
    mapping={'val':appstruct}),
colander.Invalid: {'': '".000001" is not a number'}

----------------------------------------------------------------------
Ran 335 tests in 0.195s

FAILED (errors=1)

Validation for colander.Mapping() too liberal

In colander/init.py:405-412 (class Mapping(SchemaType)) there's this validation procedure:

def _validate(self, node, value):
    try:
        return dict(value)
    except Exception, e:
        raise Invalid(…)

The problem is that dict.init will convert to dictionary almost anything:

|  dict(iterable) -> new dictionary initialized as if via:
|      d = {}
|      for k, v in iterable:
|          d[k] = v

Which means that all of these validate correctly:

• dict( "" )
• dict( [] )
• dict( () )

This is inconsistent with, for example, behavior of colander.String.
Zero-value list will result in colander.null:

In colander/init.py:793-759 (class String(SchemaType))

def deserialize(self, node, cstruct):
    if not cstruct:
        return null

And any([bool( [] ), bool( {} ), bool( "" ), bool( set() ), bool( frozenset() )]) => False

I'd like to propose to either check explicitly for dict-like functionality (setitem, getitem), or to at least make the behavior consistent with colander.String (“if not cstruct” instead of “if cstruct is null” in colander/init.py:456).

Invalid.asdict() broken for colander.All()

Invalid.asdict() fails to handle the list of messages returned by colander.All().

It should check if if hasattr(exc.msg, '__iter__') and do an extend instead of append or something similar.

File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 324, in getMessage
    msg = str(self.msg)
File "/Users/example/.virtualenvs/project/lib/python2.7/site-packages/colander-0.9.1-py2.7.egg/colander/__init__.py", line 157, in __str__
    return pprint.pformat(self.asdict())
File "/Users/example/.virtualenvs/project/lib/python2.7/site-packages/colander-0.9.1-py2.7.egg/colander/__init__.py", line 151, in asdict
    errors['.'.join(keyparts)] = '; '.join(interpolate(msgs))
TypeError: sequence item 0: expected string, list found

Allow for missing values

It is not clear why colander strictly enforces the presence of non-required values. There should at least be an 'optional' parameter which would enable validation of schemata with non-required nodes without forcing undesired default or sentinel values.

conditional validation

How can I achieve conditional validation?

Say I have two fields field_a and field_b, field_b is optional by default but it becomes required when field_a has certain values.

I've been looking at the existing validators and it seems it only receives the node that it should validate, how can I access other nodes within the same schema from a validator?

autobind feature?

The binding feature seems like a great feature. However, it means that you have to know beforehand to call bind() before [de]serialize(). It would be nice if you could set an autobind variable on a SchemaNode so that [de]serialize() would automatically call bind() for you.

The specific use-case I have in mind is https://github.com/stefanofontanelli/ColanderAlchemy/ where the schema is automatically generated and it's possible the schema may have deferred elements. Currently the code never uses deferred elements because the end user will likely not remember (or know) to call bind() before [de]serialize().

Cannot prepare based on schema binding.

There is no support for deferred preparers, which would seem to be the only way to transform values during serialization if some bound value must be used.

Edit: the Colander: Is preparer supported in schema binding? thread served as inspiration for the fix.

apply a validator to a schema

hi,

first, thanks for that great library!

i wonder if there is a solution to apply a validator to a schema before it is used. e.g.::

class LocationSchema(colander.Schema):
    validator = something
    ...

this doesn't work, because all constructor arguments of SchemaNode eliminate the class attributes::

def __init__(self, typ, *children, **kw):
    self.typ = typ
    self.preparer = kw.pop('preparer', None)
    self.validator = kw.pop('validator', None)
    ... 

but in my opinion there should be a way. imagine having a schema, inherited schemas and each instanciation needs the validator applied. if that is missing, the validation is dangerously wrong.

do you have an idea?

best, andi

documentation: error in example

http://docs.pylonsproject.org/projects/colander/en/latest/basics.html#defining-a-schema-imperatively
In the 2nd example on line 7 it says:
friend.add(colander.SchemaNode(colander.String()), name='name')
But it should say:
friend.add(colander.SchemaNode(colander.String(), name='name'))

flatten of mapping produces dot-prefixed keys

import colander
import pprint

class Friend(colander.TupleSchema):
    rank = colander.SchemaNode(colander.Int(),
                               validator=colander.Range(0, 9999))
    name = colander.SchemaNode(colander.String())

class Phone(colander.MappingSchema):
    location = colander.SchemaNode(colander.String(),
                                  validator=colander.OneOf(['home', 'work']))
    number = colander.SchemaNode(colander.String())

class Friends(colander.SequenceSchema):
    friend = Friend()

class Phones(colander.SequenceSchema):
    phone = Phone()

class Person(colander.MappingSchema):
    name = colander.SchemaNode(colander.String())
    age = colander.SchemaNode(colander.Int(),
                             validator=colander.Range(0, 200))
    friends = Friends()
    phones = Phones()

schema = Person()
appstruct = {
  'name':'keith',
  'age':20,
  'friends':[(1, 'jim'),(2, 'bob'), (3, 'joe'), (4, 'fred')],
  'phones':[{'location':'home', 'number':'555-1212'},
            {'location':'work', 'number':'555-8989'},],
  }

pprint.pprint(schema.flatten(appstruct))

# produces:

# {'.age': 20,
#  '.friends.0.name': 'jim',
#  '.friends.0.rank': 1,
#  '.friends.1.name': 'bob',
#  '.friends.1.rank': 2,
#  '.friends.2.name': 'joe',
#  '.friends.2.rank': 3,
#  '.friends.3.name': 'fred',
#  '.friends.3.rank': 4,
#  '.name': 'keith',
#  '.phones.0.location': 'home',
#  '.phones.0.number': '555-1212',
#  '.phones.1.location': 'work',
#  '.phones.1.number': '555-8989'}

# should produce?:

# {'age': 20,
#  'friends.0.name': 'jim',
#  'friends.0.rank': 1,
#  'friends.1.name': 'bob',
#  'friends.1.rank': 2,
#  'friends.2.name': 'joe',
#  'friends.2.rank': 3,
#  'friends.3.name': 'fred',
#  'friends.3.rank': 4,
#  'name': 'keith',
#  'phones.0.location': 'home',
#  'phones.0.number': '555-1212',
#  'phones.1.location': 'work',
#  'phones.1.number': '555-8989'}

I think this is the right fix, but it'd be nice to get some confirmation from @chrisrossi:

https://github.com/Pylons/colander/compare/fix.rootflatten

Hard to use unknown='preserve' in mapping schema

To allow preservation of unknown keys when deserializing with a MappingSchema, I need to write this:

class MySchema(colander.MappingSchema):
    some_key = colander.SchemaNode(colander.String())

    # Force colander to preserve unknown fields.
    def __init__(self, *args, **kwds):
        typ = colander.Mapping(unknown='preserve')
        super(MySchema, self).__init__(typ, *args, **kwds)

I would really prefer having a simple way to do this, such as:

class MySchema(colander.MappingSchema):
    unknown = 'preserve'
    some_key = colander.SchemaNode(colander.String())

Or even

MySchema(unknown='preserve').deserialize(...)

Deferred validation of sequence raises `IndexError`.

I'm following the RangedIntSchemaNode example under Subclassing SchemaNode and it works (provided you add a schema_type class attribute), but I can't get it to work for sequences.

For example:

import colander

class SortedSequenceSchemaNode(colander.SchemaNode):
    default = []
    title = 'Sorted sequence'
    schema_type = colander.Sequence

    @colander.deferred
    def validator(self, kw):
        sorting_key = kw.get('sorting_key', None)
        def validate_sorted_sequence(node, cstruct):
            # Check edge cases.
            if cstruct is None or cstruct is colander.null:
                return colander.null
            if not iterable(cstruct):
                raise colander.Invalid('Expecting a sequence, not %r.' % cstruct)
            # Sort items in sequence.
            return sorted(cstruct, key=sorting_key)
        return validate_sorted_sequence


class TestSchema(colander.MappingSchema):
    stuff = SortedSequenceSchemaNode(missing=[])


data = {'stuff': ['A', 'b', 'C', 'd']}
data = TestSchema().bind(sorting_key=str.lower).deserialize(data)
print data['stuff']

When I execute this, I get

IndexError: list index out of range

in colander/__init__.py, line 858:

result.append(callback(node.children[0], subval))

I'm using Colander 1.0a2.