Fix regressions in support for bytes / unicode in certain APIs

[pyca]

Fix regressions in support for bytes / unicode in certain APIs

[exarkun]

Can you confirm that this is all based on Python 2.7 and not some other version?

[adiroiban]

Sorry for missing the details.

Yes. This is Python 2.7.3 on Ubuntu 12.04 x32.

[flavio]

I experience the same issue on openSUSE 13.1 and SUSE Linux Enterprise (both running python 2.7.6).

[exarkun]

This is going to be fixed for 0.15 - actually fixed, not just documented as broken.

[lvh]

What's the actual fix? I presume some of them really only ought to take bytes, but I could also appreciate that you want to have e.g. Context.set_cipher_list(u'ALL') work again as well, since it did in previous versions :)

[exarkun]

Any API which in the past accepted bytes without error should accept bytes without error.

Any API which in the past accepted unicode without error should accept unicode without error.

"want" is putting it rather strongly though.

[lvh]

Gotcha. So the resolution for code is: typecheck for unicode and encode ASCII if necessary. I'm inclined to raise warnings instead so we don't have to support it in perpetuity but I don't really care.

Solution for docs: document one seemingly appropriate type (and accidentally accept the other one, too), or document both as supported? I'd like to know for #126 :)

[exarkun]

Deprecating the nonsense case would be great. It doesn't need to be part of this change though (in the interest of getting it done).

I think document the appropriate type but if figuring that out is hard then maybe leave the docs alone?

[cslink]

Are there any updates on the progress of this issue? I'm currently trying to decide whether to try and add support for 0.14 or just support 0.13 and the upcoming 0.15.

[exarkun]

I don't have availability to work on this in the near future. Other community members haven't expressed any interest in having this fixed, but have expressed interest in having 0.15 out sooner rather than later. The practical outcome may be that 0.15 is released with these regressions and they just become part of the API.

[exarkun]

Other community members haven't expressed any interest in having this fixed

Sorry, I meant "in fixing this themselves". Clearly the people who commented on this issue, at least, had some interest in having it fixed.

[hynek]

I’m afraid the problem of this issue is that it has unclear scope and somehow daunting.

I don’t think it’s very difficult to go through the API and add some isinstance + encode("ascii") if unicode has been passed. But somehow there’s not a clear path forward. If there would, I’m personally volunteering to walk it/help directing others.

[mcepl]

What is so complicated on DeprecationWarning? Add all these isinstance + encode("utf8") things, but throw a warning as well. In next major release, remove both?

[abrahammartin]

Is 0.15 going to be release any time soon? 0.14 was released a year ago and there are a lot of bugfixes and new features added since then.

[dbrent-amazon]

I am having the same issue. http://stackoverflow.com/questions/28749543/pyopenssl-signature-verification-using-python-3

[exarkun]

Is 0.15 going to be release any time soon?

It's on my todo list. I had planned to do one over the holidays but got tied up with other (even less fun, I assure you) things. I have some airport time coming up during which I intend to give this another shot. Sorry about not getting to this more quickly.

[sgallagher]

I was just about to file this as a new bug when I stumbled upon this ticket. I'm working on a script to generate self-signed certificates and it's broken on Python 3. One thing I couldn't discern from the comments above: is there a workaround? Can I convert the strings to something else that would be accepted?

The following code works in Python 2

crypto.X509Extension("subjectKeyIdentifier",
                                 False,
                                 "hash",
                                 subject=ca_cert,
                                 issuer=ca_cert)

However, on Python 3, it results in the error:

File "/usr/lib/python3.4/site-packages/OpenSSL/crypto.py", line 493, in __init__
    extension = _lib.X509V3_EXT_nconf(_ffi.NULL, ctx, type_name, value)
TypeError: initializer for ctype 'char *' must be a bytes or list or tuple, not str

[adiroiban]

you can try using b"subjectKeyIdentifier" to force bytes... similar for b"hash"

[exarkun]

I was just about to file this as a new bug when I stumbled upon this ticket. I'm working on a script to generate self-signed certificates and it's broken on Python 3.

Also please note this is not a pyOpenSSL bug. This is a Python 2 / Python 3 incompatibility. Your program passes byte strings to the pyOpenSSL API on Python 2. However, your program passes unicode strings to the pyOpenSSL API on Python 3. The API supports byte strings. My inclination at this point is not to change pyOpenSSL to support this usage. Instead, if you want to port an application to Python 3, you have to fix your string usage.

[adiroiban]

At least set_cihpher_list was working when passed Unicode strings in Python2. In python2 I am explicitly using u'ALL' to make it an unicode string.

[exarkun]

At least set_cihpher_list was working when passed Unicode strings in Python2.

Of course. Python 2 will implicitly encode unicode to str for any number of reasons. This was never intended behavior.

This isn't quite the same thing that sgallagher's comment is about. His program passes bytes to a pyOpenSSL API on Python 2. He can reasonably expect that passing bytes to that API on Python 3 should work. It does not seem to me that he can reasonably expect that passing unicode to that API on Python 3 API should work.

[adiroiban]

I am +1 for updating the documentation to inform that only bytes are supported.
Later we can update separate API method to support Unicode and to write unit tests for that.

To fix this ticket I think that is ok to just add a bit warning in pyopenssl introduction that Unicode is not supported... and that Unicode which worked in py2.7 will break in py3.

[adiroiban]

I am only testing on Py 2.7

I have found another Unicode issue in crypto.X509Req() subject which ask for Unicode

Extensions are accepted with UTF-8

I have this code

csr = crypto.X509Req()
subject = csr.get_subject()
name = u\u20acuro-zone.com'
subject.organizationName = name.encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position X: ordinal not in range(128)

but will accept only accept Unicode ... not UTF-8 bytes.

Same story for commonName organizationalUnitName localityName stateOrProvinceName

countryName only accepts the hardcoded codes so we should not care about unicode here.

subject.emailAddress only accepts ascii chars and fail if I sent it Unicode with non-ascii characters... but also fails if I encode them with utf-8

I am working at a python script to generate a CSR . The code and test is here chevah/chevah-keycert#10

[exarkun]

What version of pyOpenSSL did you test this against? master@HEAD runs this code without error.

[adiroiban]

I was testing against released version 0.13 and 0.14.

I will give it a try with master. Looking forward for 0.15 release. Thanks!

[digz6666]

Is it fixed in version 12.6.0? https://pypi.python.org/pypi/pyOpenSSL

[hynek]

This seems to have been fixed years ago. @adiroiban can you confirm and close please?

[adiroiban]

Looks good. Thanks!
both Unicode and UTF-8 bytes are accepted

$ py
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from OpenSSL import crypto
>>> csr = crypto.X509Req()
>>> subject = csr.get_subject()
>>> name = 'u\u20acuro-zone.com'
>>> subject.organizationName = name.encode('utf-8')
>>> subject
<X509Name object '/O=u\u20acuro-zone.com'>
>>> subject.organizationName = name
>>> import OpenSSL
>>> OpenSSL.__version__
'16.2.0'