Allow custom escaping functions when generating LDAP filters

[twisted]

Allow custom escaping functions when generating LDAP filters

[cwaldbieser]

There are a couple "ah-ha!" moments I had when reviewing the RFCs and the code.

1. The syntax for LDAP filters matching GUIDs is not really special. What makes creating the filter so non-intuitive is that GUIDs are stored as binary data. The string representations we commonly work with are not how the data is stored, and that is why trying to filter on the common string representation doesn't work.
2. RFC 4515 explains how escaping in LDAP filters works. In particular, on page 4, the characters that are normally escaped. It further goes on to say:

Other octets that are part of the <normal> set may be escaped using this mechanism, for example, non-printing ASCII characters.

And later:

As indicated by the <valueencoding> rule, implementations MUST escape all octets greater than 0x7F that are not part of a valid UTF-8 encoding sequence when they generate a string representation of a search filter.

This tells us that binary values more or less need to be represented in the escaped format, which is what the Ping Identity article you referenced is describing. (More on the "more or less" later).

3. The pureldap.LDAPFilter_* objects have both str() representations and asTest() methods. The former is used when transmitting the BER sequences to an LDAP server. The latter is used when displaying textual representations of the data (during debugging, for example).

A code example may get to the heart of the matter:

>>> import ldaptor
>>> import ldaptor.ldapfilter as lf
>>> guid_chars = "90395F191AB51B4A9E9686C66CB18D11"
>>> chars = [c for c in guid_chars]
>>> chars
['9', '0', '3', '9', '5', 'F', '1', '9', '1', 'A', 'B', '5', '1', 'B', '4', 'A', '9', 'E', '9', '6', '8', '6', 'C', '6', '6', 'C', 'B', '1', '8', 'D', '1', '1']
>>> firsts = chars[::2]
>>> seconds = chars[1::2]
>>> pairs = zip(firsts, seconds)
>>> escaped_guid_str_rep = ''.join([r"\{0}{1}".format(a, b) for a, b in pairs])
>>> print(escaped_guid_str_rep)
\90\39\5F\19\1A\B5\1B\4A\9E\96\86\C6\6C\B1\8D\11
>>> filter_str = "(guid={0})".format(escaped_guid_str_rep)
>>> print(filter_str)
(guid=\90\39\5F\19\1A\B5\1B\4A\9E\96\86\C6\6C\B1\8D\11)
>>> fo = lf.parseFilter(filter_str)
>>> str(fo)
'\xa3\x18\x04\x04guid\x04\x10\x909_\x19\x1a\xb5\x1bJ\x9e\x96\x86\xc6l\xb1\x8d\x11'
>>> fo.asText()
'(guid=\x909_\x19\x1a\xb5\x1bJ\x9e\x96\x86\xc6l\xb1\x8d\x11)'
>>> chr(0x39)
'9'
>>> chr(0x5F)
'_'
>>> 

What we see here is that the filter string will be parsed and represented correctly both as bytes sent to the LDAP server and as text displayed during debugging. However, the textual representation used for debugging, while correct, seems a bit odd to us as human beings because it mixes escaped and non-escaped representations. All the bytes that must be escaped are, but the parts that may optionally be escaped in a representation are not.

This is actually a bit confusing to look at, but the sequences:
\90\39\5F\19\1A\B5\1B\4A\9E\96\86\C6\6C\B1\8D\11
and
\x909_\x19\x1a\xb5\x1bJ\x9e\x96\x86\xc6l\xb1\x8d\x11
are equivalent. One is just a lot more intuitive for a human being to parse.

Your proposed change would provide a way to customize the textual representation of filters, though I'd suggest that the naming ("custom escape functionality") is perhaps a bit misleading. "Custom LDAP filter textual representation" or "custom LDAP filter human-readable representation" both seem more accurate to me, though they are very wordy.

I wonder if it would be possible to push on the default case a bit. Do you think it might be possible to extend your idea so that the human-readable representation would automatically detect binary data in filters automatically represent all the octets with escape sequences? Maybe if say half of the characters in the filter assertion value were non-printable?

I'll have access to an AD instance tomorrow, so I should be able to test to make sure that filtering on GUIDs already works-- it is just the textual representations can be queer-looking for binary data.