vanrein/quick-der

Quick `n' Easy DER library

quick-der's Issues

use semantic versioning

Makes it easier to create a package

https://semver.org/

Add der_cmp_INTEGER()

As introduced in KXOVER. Be sure to test heavily; an ordered sequence of INTEGER values and testing all combinations (quadratic) to present the right answer should do.

Provide an ASN.1 compiler

The PATH and WALK notations are useful for manual construction of a DER parser, but they require at least an intuition about how DER works. It is possible to derive it from ASN.1 directly though, and a compiler would be really helpful.

Although we mentioned libtasn1 before, the focus has shifted to asn1ate because it appears to be more complete, and has a dynamic backend structure built upon a semantic model and even some global analysis (dependencies).

Having a compiler in Python is not problematic, and may even simplify dependencies. The purpose of this compiler is mainly intended for building environments, and inasfar as RFCs and other standards are mapped with it, we intend to prepare it for #include <quick-der/rfc5280.h> to make DER Quick... and Easy.

Add SPNEGO, RFC 4178

How could I forget :)

parallel make fails with high number of jobs

[ 62%] Building C object test/CMakeFiles/ldap.test.dir/ldapsearch.c.o
cd /home/gijs/arpa2/quick-der/obj-x86_64-linux-gnu/test && /usr/bin/x86_64-linux-gnu-gcc   -I/home/gijs/arpa2/quick-der/include -I/home/gijs/arpa2/quick-der/obj-x86_64-linux-gnu/rfc  -march=native -O2 -pipe -Wdate-time -D_FORTIFY_SOURCE=2    -std=gnu99 -o CMakeFiles/ldap.test.dir/ldapsearch.c.o   -c /home/gijs/arpa2/quick-der/test/ldapsearch.c
[ 62%] Built target rfc2578_asn1_h
/home/gijs/arpa2/quick-der/test/ldapsearch.c:23:31: fatal error: quick-der/rfc4511.h: No such file or directory
compilation terminated.

What we can der_unpack() we should also der_pack()

The Python code does clever things with substitute classes and native types. This is wonderful to work with, but we should also make it work when repacking into DER.

The Python code introduces None for non-existent entries; it even sets all fields to None when creating an empty instance of an ASN.1 type-named class. When the entries that are None conform to syntax, these should translate into proper DER-packed output.

On OS X shared libraries have .dylib extension

https://github.com/vanrein/quick-der/blob/master/contrib/cmake/Quick-DERConfig.cmake.in#L12

Failure matching EncryptedData of RFC4120

This contains a complex structure,

SEQUENCE {
   ... [0] ...,
   ... [1] ... OPTIONAL,
   ... [2] ...
}

which translates to parser structure

#define DER_PACK_rfc4120_EncryptedData \  
        DER_PACK_ENTER | DER_TAG_SEQUENCE, \  
        DER_PACK_ENTER | DER_TAG_CONTEXT(0), \  
        DER_PACK_rfc4120_Int32, \  
        DER_PACK_LEAVE, \  
        DER_PACK_OPTIONAL, \  
        DER_PACK_ENTER | DER_TAG_CONTEXT(1), \  
        DER_PACK_rfc4120_UInt32, \  
        DER_PACK_LEAVE, \  
        DER_PACK_ENTER | DER_TAG_CONTEXT(2), \  
        DER_PACK_STORE | DER_TAG_OCTETSTRING, \  
        DER_PACK_LEAVE, \  
        DER_PACK_LEAVE

The OPTIONAL followed by ENTER goes wrong when called with a structure with [0]...[2]... but not the optional [1]... part. What happens is that der_unpack_rec() is invoked with the contents of the [2] after a mismatch of the [1] syntax part. Bad, and indeed EBADMSG is reported to the caller! Note that the optional flags are passed properly into der_unpack_rec() though, so it is surprising in itself that it finds a reason to complain at all!

Bugs therefore are:

1. should not invoke der_unpack_rec() with the embedded type of one that mismatched;
2. should not report EBADMSG for a mismatch when optional flagging is set.

Handle IMPLICIT tagging

When working on LillyDAP (meaning "little LDAP"), I first ran into the use of IMPLICIT tags and this is not supported properly.

For instance, this is a SearchResultDone message:

30 0c 02 01 01 65 07 0a  01 00 04 00 04 00

Decoded with derdump, this becomes

SEQUENCE (OF): tag 0x30 #12 @0 ^0, Universal, Constructed
  INTEGER: tag 0x02 #1 @2 ^1, Universal, Primitive
     01 == 1
  [APPLICATION 5]: tag 0x65 #7 @5 ^1, Application, Constructed
    ENUMERATED: tag 0x0a #1 @7 ^2, Universal, Primitive
       00 == "."
    OCTETSTRING: tag 0x04 #0 @10 ^2, Universal, Primitive
    OCTETSTRING: tag 0x04 #0 @12 ^2, Universal, Primitive

Note the absense of the SEQUENCE (OF) immediately underneath the [APPLICATION 5] tag for SearchResultDone.

In conflict with this, Quick DER currently generates the following in rfc4511.h:

#define DER_PACK_rfc4511_LDAPResult \
        DER_PACK_ENTER | DER_TAG_SEQUENCE, \
        DER_PACK_STORE | DER_TAG_ENUMERATED, \
        DER_PACK_rfc4511_LDAPDN, \
        DER_PACK_rfc4511_LDAPString, \
        DER_PACK_OPTIONAL, \
        DER_PACK_ENTER | DER_TAG_CONTEXT(3), \
        DER_PACK_rfc4511_Referral, \
        DER_PACK_LEAVE, \
        DER_PACK_LEAVE

#define DER_PACK_rfc4511_SearchResultDone \
        DER_PACK_ENTER | DER_TAG_APPLICATION(5), \
        DER_PACK_rfc4511_LDAPResult, \
        DER_PACK_LEAVE

This assumes that a SEQUENCE tag will be found underneath [APPLICATION 5]. A suggested improvement could be:

#define DER_PACK_rfc4511_LDAPResult_implicit \
        DER_PACK_STORE | DER_TAG_ENUMERATED, \
        DER_PACK_rfc4511_LDAPDN, \
        DER_PACK_rfc4511_LDAPString, \
        DER_PACK_OPTIONAL, \
        DER_PACK_ENTER | DER_TAG_CONTEXT(3), \
        DER_PACK_rfc4511_Referral, \
        DER_PACK_LEAVE

#define DER_PACK_rfc4511_LDAPResult \
        DER_PACK_ENTER | DER_TAG_SEQUENCE, \
        DER_PACK_rfc4511_LDAPResult_implicit, \
        DER_PACK_LEAVE

#define DER_PACK_rfc4511_SearchResultDone \
        DER_PACK_ENTER | DER_TAG_APPLICATION(5), \
        DER_PACK_rfc4511_LDAPResult_implicit, \
        DER_PACK_LEAVE

Until this is fixed, LillyDAP must remain on hold :'-(

Elaborate testing framework

Ideally, we should generate "difficult" cases for all primitive types, as well as the possible combinations for each of the constructed types. This should help to automatically construct a set of tests to perform on each builds. Specific problems reported by users of Quick DER could also be manually added to the set of tests. Below is how I described this in Dutch.

Missing: primitive data handling

It would be valuable to have functions to read/write elementary types such as INTEGER, ENUMERATED and REAL, the various *STRING forms and as a special form, perhaps getting and setting individual flags or flag fields in a BIT STRING.

There does not seem as much need for OBJECT IDENTIFIER, which can usually be treated in its binary form.

Iteration is a nuisance

The C interface for iteration is mind-boggling. Even when it is sort-of okay to write...

if (der_iter_first (&crs, &iter) do {
   ...work on iter...
} while (der_iter_next (&iter));

...there still is confusion about whether to der_enter() and der_focus(), both inside and prior to the iteration. The cause is basically that the iteration uses .derlen that span to the end of iteration, which is not normally what we use. Or one might say that the .derptr has dived in one more level than .derlen.

A normal API for iteration would use a callback, and that seems quite doable here too; it requires more functions but simplifies the conceptual use of iteration. We might also improve on the current API, of course, for instance by taking the .derlen from the original cursor and thus require it to be presented in der_iter_next() too.

cmaking current master fails

( cd build && cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local -DCMAKE_PREFIX_PATH=/usr/local )
CMake Error at CMakeLists.txt:8 (find_package):
  Could not find a package configuration file provided by "ARPA2CM" with any
  of the following names:

    ARPA2CMConfig.cmake
    arpa2cm-config.cmake

  Add the installation prefix of "ARPA2CM" to CMAKE_PREFIX_PATH or set
  "ARPA2CM_DIR" to a directory containing one of the above files.  If
  "ARPA2CM" provides a separate development package or SDK, be sure it has
  been installed.


-- Configuring incomplete, errors occurred!
make: *** [build/CMakeCache.txt] Error 1

Consider return-type consistency of der_pack()

Apparently, der_pack() is the only function that does not return a success/failure value and sets errno accordingly. Instead, der_pack() returns a byte size. Consider better alignment.

The size output could be a size_t * output parameter. It may be set to 0 to indicate "please figure it out" and may then be validated upon 2nd call. The output may in both cases be an error code. (The error code could be retained with output parameter initialisation to a weird value (like 0 or 1) to indicate keeping errno across calls. This might simplify error handling — but at the expense of simplicity, so probably not a good idea).

It would seem to be possible to create a #define der_pack() with a parameter less, and map it to the function. That would effectively allow for backward compatibility and a deprecation path.

Unusual bitflag-value and unreachable code in derwalk()

In lib/der_unpack.c, function derwalk(), there's the following code:

				if (*walk & DER_PACK_STORE) {
					memset (outarray + (*outctr)++,

The DER_PACK_STORE is a weird "bit flag" since it has value 0x00 (see api.h), Possibly this is intended:
((*walk & DER_PACK_MATCHBITS) == DER_PACK_STORE), but it could just as easily be intended the other way around -- I can't quickly tell why that memset is there.

Generate hyperlinked ASN.1 (waiting for YOU)

This applies to the asn2quickder tool, which generates code (or anything else) from ASN.1 formats, such as the RFCs that we habitually include in this project.

Seeing that ASN.1 references symbols all over the place, it is useful to be able to map the notation to a hyperlinked document. HTML is a quick win, but a more general format such as TexInfo is probably better, since it maps to so many other forms.

I won't have the time for it, so you're welcome to pitch in what you've got... it's a great task for a beginning Python programmer, and very useful too :-)

[moved here from an old repo issue]

port extension to Python3

See https://github.com/vanrein/quick-der/tree/python3

Branch python3 created to do this work.

This is a low priority and welcomes submissions :)

make fails

Looks like something is missing:

$ make
...
make[1]: Entering directory '/usr/oms/src/quick-der/rfc'
PYTHONPATH='../tool/asn1ate' ../tool/asn1ate/asn1ate/asn2quickder.py rfc3280.asn1
/bin/sh: 1: ../tool/asn1ate/asn1ate/asn2quickder.py: not found
Makefile:26: recipe for target 'rfc3280.h' failed
make[1]: *** [rfc3280.h] Error 127
make[1]: Leaving directory '/usr/oms/src/quick-der/rfc'
Makefile:5: recipe for target 'all' failed
make: *** [all] Error 2

CMake EXPORT broken on POSIX

This commit was made for Windows compliance, but it breaks POSIX. It should be reverted or made more general.

5cd5934

Enable travis-ci

travis-ci is a continues integration service that runs a custom script for every commit or PR. You can for example compile the code and run a test suite. You can even configure it to upload code to for example pypi on a release. It is very useful and free for open source software. It also integrates very nicely with github.

an example:

https://github.com/gijzelaerr/pymonetdb
https://travis-ci.org/gijzelaerr/pymonetdb

Consider splitting into multiple (packages|repo)

The Python tooling built on top of quick-der is not necessary for using the library itself. Since building the scripts and getting the dependencies right for generating the (C and Python) bindings for ASN.1 syntax files is a pain in the butt, perhaps it should be split out of the QD library (repo|package). Generated (C) headers for the RFC headers could be checked in, since they change very little.

As an alternative, the configure script should at least try to find a suitable Python interpreter and check for dependency modules (e.g. six) before the build starts.

Support for sub-parsers for SEQUENCE/SET OF

Quick DER chooses to be simplistic in terms of memory management. Layers built around it, such as LillyDAP may well be smarter.

This means that the one thing Quick DER refuses, namely to parse SEQUENCE OF and SET OF, can be supported with some help of the outer layer. The issue now is to make Quick DER support such outer layers (and test it with LillyDAP).

We could map to a dernode union instead of dercursor for these structures, and initially der_unpack() into the same form as now namely the dercursor in the wire variant of the union; surrounding code in the smarter application can then be mapped to the info form with an array pointer derray and array length count dercnt. The array elements themselves are dernode values once more. It is the application's knowledge when to use the wire form and when to use info. It usually maps one way after der_unpack() and the other way before der_pack().

Finally, the ASN.1 compiler must produce useful definitions that guide the application to the indices and syntaxes of sub-elements -- which must be supportive of a recursive procedure because, for instance, there may be a SEQUENCE OF inside of a SEQUENCE OF. Such recursion would of course apply to each individual element, as the structure is repeating.

Once this is done, we shall consider adding generic code to Quick DER, using a callback for memory allocation.

port asn2quickder to Pythonic argument-handling

asn2quickder uses getopt.getopt(), which is fun for C programmers, but was replaced in Python-style by argparse ages ago. A low-priority job would be to replace the command-line-argument handling by something more modern (handling the test-cases collection is the biggest thing there).

SEQUENCE of OPTIONs packs to 0 bytes

Running der_pack() on the following produces 0 bytes when the two dercursor structures have their derptr==NULL -- but it should have packed to an empty sequence,0x30 0x00.

KIP-SASL-REQ ::= [PRIVATE 3] SEQUENCE {
   mech  [0] IA5String OPTIONAL,
   c2s   [1] OCTET STRING OPTIONAL
}

Since 0 is the error output from der_pack(), this is confusing.

Note the possible relation with #58 on the returned value.

Extension: DER streams

It may be useful to support a DER stream concept, holding sequences of DER structures containing DER messages. Except that it's BER, LDAP is an example of such a protocol.

A reading routine may start off with an initial read that determines the desired length (from the tag/length) and allow the caller to arrange for that before reading the remaining data. The initial call can be made with a fixed length buffer, or with one that has shown to be large enough in prior calls.

A similar buffer can be filled when packing to write back a response.

IMPLICIT tags via DER_PACK_ANY fail on ""

With the new support for IMPLICIT tags such as used by LDAP, many things now work better. But there are exceptions not yet solved. Take this LDAPMessage for instance,

SEQUENCE (OF): tag 0x30 #12 @0 ^0, Universal, Constructed
  INTEGER: tag 0x02 #1 @2 ^1, Universal, Primitive
     01 == 1
  [APPLICATION 0]: tag 0x60 #7 @5 ^1, Application, Constructed
    INTEGER: tag 0x02 #1 @7 ^2, Universal, Primitive
       03 == 3
    OCTETSTRING: tag 0x04 #0 @10 ^2, Universal, Primitive
    [0]: tag 0x80 #0 @12 ^2, Contextual, Primitive

Note how nothing is contained in the [0] tag on the last line. This is possible when the contained information is elementary, empty and implicitly tagged. This is indeed the case, if we look at the definitions:

       BindRequest ::= [APPLICATION 0] SEQUENCE {
             version                 INTEGER (1 ..  127),
             name                    LDAPDN,
             authentication          AuthenticationChoice }

        AuthenticationChoice ::= CHOICE {
             simple                  [0] OCTET STRING,
                                     -- 1 and 2 reserved
             sasl                    [3] SaslCredentials,
             ...  }

This however, translates to:

#define DER_PACK_rfc4511_BindRequest \
        DER_PACK_ENTER | DER_TAG_APPLICATION(0), \
        DER_PACK_STORE | DER_TAG_INTEGER, \
        DER_PACK_rfc4511_LDAPDN, \
        DER_PACK_rfc4511_AuthenticationChoice, \
        DER_PACK_LEAVE

#define DER_PACK_rfc4511_AuthenticationChoice \
        DER_PACK_CHOICE_BEGIN, \
        DER_PACK_ENTER | DER_TAG_CONTEXT(0), \
        DER_PACK_STORE | DER_PACK_ANY, \
        DER_PACK_LEAVE, \
        DER_PACK_ENTER | DER_TAG_CONTEXT(3), \
        DER_PIMP_rfc4511_SaslCredentials, \
        DER_PACK_LEAVE/* ...ASN.1 extensions... */, \
        DER_PACK_CHOICE_END

The problem lies in the DER_PACK_STORE | DER_PACK_ANY nested within the DER_PACK_ENTER | DER_TAG_CONTEXT(0) of DER_PACK_rfc4511_AuthenticationChoice. This does not seem to capture an empty string (and probably nothing that is not proper DER-encoded data, so header / length / value format.

What needs to be done, is replace these three lines

        DER_PACK_ENTER | DER_TAG_CONTEXT(0), \
        DER_PACK_STORE | DER_PACK_ANY, \
        DER_PACK_LEAVE, \

with the single line

        DER_PACK_STORE | DER_TAG_CONTEXT(0), \

This way, the structure is stored without any attempt to entering it -- which is the intention.

Dependency sort assignments in a module by default

Currently, this is done at various locations while I see no reason to not sort them by default after parsing the semantic model.

https://github.com/vanrein/quick-der/search?utf8=%E2%9C%93&q=dependency_sort&type=

der_unpack() modifies dercursor

is this expected? If so, why?

Missing: der_focus()

After a DER walk, we have a dercursor that incorporates all the remainder items in a structure. Often, we will only be interested in the first one. Using der_focus() we could zoom in on that one element. This can currently be established with der_header() but that is a bit obnoxious.

Python packaging not flawless

The Python packaging code does a few things funnily [though I'm not laughing about it]:

• The installation in the build directory does not introduce the quick_der packaging dir, so setting of PYTHON_PATH does not really work
• This has led to scripts in test/*.py that make funny assumptions about package names (they strip the quick_der package name)
• The generated scripts do not end up in python/quick_der/*.py where they ought to be
• The __init__.py and api.py are not installed alongside the generated scripts
• The setup.py file does not incorporate these files; we only seem to build and install _quickder.so with it and leave the rest to CMake

der_unpack() can use const for input cursor

We can use der_unpack(const dercursor *crs, ...)

When this is done, the calls from keehive can also be made with static const structures.

Unreachable code in der_enter()

lib/der_skipenter.c contains der_enter(), with unreachable code at the end of der_enter(). The if(tag == DER_TAG_BITSTRING) is not reached because of the returns in the if-else above it. The comment for the function explicitly mentions this special behavior, as does the comment in test/certio.c, so it might be important behavior. I can't think of a reason to enter a bit-string.

assumes python is in install prefix

at the moment it seems python code is installed in <PREFIX>lib/python2.7/site-packages/.

This probably works fine if you install in /usr/local, but fails in most other cases.

Probably better to do is:

• Get current Python from path, which can be optionally overriden using cmake
• Ask Python where its site-packages folder is.

This works in most cases, also with virtualenvs. Alternatively you could just split the python code out and safe frustration, don't use cmake for Python and have a pipable python package.

'NoneType' object has no attribute 'symbols_imported' using latest asn1ate

I've modified the python code to directly use asn1ate from github, see the pygijs branch:

https://github.com/vanrein/quick-der/tree/pygijs

When I run asn2quickder with the latest asn1nate i run into troubles:

/Users/gijs/Work/quick-der/python/.virtualenv/bin/python /Users/gijs/Work/quick-der/python/scripts/asn2quickder ../../doc-pkcs11/doc/RemotePKCS11.asn1
Traceback (most recent call last):
  File "/Users/gijs/Work/quick-der/python/scripts/asn2quickder", line 1580, in <module>
    for rm in dm.imports.symbols_imported.keys():
AttributeError: 'NoneType' object has no attribute 'symbols_imported'

Process finished with exit code 1

Python module doesn't install correctly when installed in developer mode

this works:

$ virtualenv -p python3 .venv
Running virtualenv with interpreter /usr/local/bin/python3
[...]
Installing setuptools, pip, wheel...done.
$ .venv/bin/pip install ../quick-der
Processing /Users/gijs/Work/quick-der
[..]
Successfully installed asn1ate-0.6.0 pyparsing-2.2.0 pyquickder-1.2.2 six-1.11.0
$  .venv/bin/python3 -c "import quick_der"

But installing quick_der in developer (-e) mode doesn't work:

$  .venv/bin/pip install -e ../quick-der
Obtaining file:///Users/gijs/Work/quick-der
[...]
Successfully installed asn1ate-0.6.0 pyparsing-2.2.0 pyquickder six-1.11.0
$  .venv/bin/python3 -c "import quick_der"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'quick_der'

This probably has to do with that the python module is not a direct subfolder of the setup.py script.

Consider adding (very) long tags

Current Quick DER refuses tags over 30, but semantics of an extension are not dramatic, in that tag 31 indicates that a length octet follows and then that many bytes.

This would make an interesting utility for storing

• magic codes in a tag of DER-encoded content, by containing as much as 16 bytes (for instance) describing a particular application's random number, or
• nested names structured like a (reverse) domain-nested name (but case sensitive)

The parser may be updated to treat tag 31 specially, without loosing much of its other functions; the WALK and PACK notations can be adapted to accommodate inlined exact matches. Only CHOICE would really suffer if it had to match this style of DER code. It may be simply enough to adapt der_header() to deliver tag 31 and have it skip ahead, and let the user program match the exact string that follows.

This may be so doable we should do it. Just add a hint below if you have an aplication that might convince us!

Worries about parsing of SET

While programming I never considered that elements of SET might occur in any order, and treated it just like SEQUENCE. This is likely to be a bug in the current logic of the parser.

asn2quickder python script doesn't work when installed

the ans1ate module is not installed

 python2.7 /usr/local/bin/asn2quickder
Traceback (most recent call last):
  File "/usr/local/bin/asn2quickder", line 34, in <module>
    from asn1ate import parser
ImportError: No module named asn1ate

cmake files installed in wrong location

they are installed in <PREFIX>/lib/cmake/Quick-DER but I'm quite sure that should be <PREFIX>/share/cmake/Modules

asn2quickder doesn't support AUTOMATIC TAGS

See tools/test/test01.asn1 vs. tools/test/test01a.asn1. Running asn2quickder on the first gives a meaningless error message, while the other one compiles successfully.

Either asn2quickder should do something sensible with AUTOMATIC TAGS (e.g. ignore them) or produce a better error message.

COMPONENTS OF is almost working

Got most of COMPONENTS OF working:

• generation of DER_PACK_ and DER_PIMP_ is working
• generation of DER_OVLY_ does not

The matter is how to get to the structure elements. Already added a default-False parameter naked that can be explicitly set to naked=True to unleash elements in a structure without the opening struct { and closing }.

Nested optional types encoded with zero length

This applies to TLS-KDH, just preceding the checkin of arpa2/tlspool@c78ab02

In Kerberos' Authenticator:

   Authenticator   ::= [APPLICATION 2] SEQUENCE  {
           authenticator-vno       [0] INTEGER (5),
           crealm                  [1] Realm,
           cname                   [2] PrincipalName,
           cksum                   [3] Checksum OPTIONAL,
           cusec                   [4] Microseconds,
           ctime                   [5] KerberosTime,
           subkey                  [6] EncryptionKey OPTIONAL,
           seq-number              [7] UInt32 OPTIONAL,
           authorization-data      [8] AuthorizationData OPTIONAL
   }

The parts [7] UInt32 OPTIONAL with no value of that kind in the corresponding dercursor, run through der_pack(), leads to the encoding a7 00. Similarly [8] AuthorizationData OPTIONAL is packed as a8 00. The result is unacceptable to der_unpack().

In other words, the OPTIONAL is not applied to the outside tag and only to the inner real type. This is wrong.

Constraints ignored by Quick DER

The present Quick DER ignores constraints, even when they might be meaningful to the parsing process. This leads to the (clear) situation where library users must handle constraint validation themselves.

There are a few ways of improving on this:

• Some constraints to SIZE() limit SEQUENCE size, which means it may be possible to parse it into a fixed-size array; this will be difficult to communicate, and lead to rather long parsing instructions, so it is of doubtful benefit. It also complicates the use of Quick DER.
• We may add code generators for parsing and unparsing structures from other structures or function parameters; when this is done, we should incorporate constraints because the abstraction of such dedicated parsers/unparsers would otherwise be broken