lib/openssl/hmac.rb:55:in `initialize': EVP_PKEY_new_mac_key: malloc failure (OpenSSL::HMACError) with -Werror flag

[junaruga]

lib/openssl/hmac.rb:55:in `initialize': EVP_PKEY_new_mac_key: malloc failure (OpenSSL::HMACError) with -Werror flag

[rhenium]

I reproduced it with RUBY_OPENSSL_EXTCFLAGS=-Werror rake compile. mkmf.log says:

have_func: checking for EVP_PKEY_new_raw_private_key(0, NULL, (unsigned char *)"", 0) in openssl/evp.h... -------------------- no

LD_LIBRARY_PATH=.:/opt/ruby/master/lib "gcc -o conftest -I/opt/ruby/master/include/ruby-3.3.0+0/x86_64-linux -I/opt/ruby/master/include/ruby-3.3.0+0/ruby/backward -I/opt/ruby/master/include/ruby-3.3.0+0 -I../../../../ext/openssl -I/opt/openssl/3.0-stable/include    -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wdeprecated-declarations -Wdiv-by-zero -Wduplicated-cond -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wwrite-strings -Wold-style-definition -Wimplicit-fallthrough=0 -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable -Wundef  -fPIC -Werror conftest.c  -L. -L/opt/ruby/master/lib -Wl,-rpath,/opt/ruby/master/lib -L/opt/openssl/3.0-stable/lib -Wl,-rpath,/opt/openssl/3.0-stable/lib -L. -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,--no-as-needed    -lssl -lcrypto  -Wl,-rpath,/opt/ruby/master/lib -L/opt/ruby/master/lib -lruby -lssl -lcrypto  -lm -lpthread  -lc"
conftest.c: In function ‘t’:
conftest.c:17:30: error: ‘s1’ may be used uninitialized [-Werror=maybe-uninitialized]
   17 | int t(void) { char s1[1024]; EVP_PKEY_new_raw_private_key(0, NULL, (unsigned char *)s1, 0); return 0; }
      |                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from conftest.c:3:
/opt/openssl/3.0-stable/include/openssl/evp.h:1810:11: note: by argument 3 of type ‘const unsigned char *’ to ‘EVP_PKEY_new_raw_private_key’ declared here
 1810 | EVP_PKEY *EVP_PKEY_new_raw_private_key(int type, ENGINE *e,
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
conftest.c:17:20: note: ‘s1’ declared here
   17 | int t(void) { char s1[1024]; EVP_PKEY_new_raw_private_key(0, NULL, (unsigned char *)s1, 0); return 0; }
      |                    ^~
At top level:
cc1: note: unrecognized command-line option ‘-Wno-self-assign’ may have been intended to silence earlier diagnostics
cc1: note: unrecognized command-line option ‘-Wno-parentheses-equality’ may have been intended to silence earlier diagnostics
cc1: note: unrecognized command-line option ‘-Wno-constant-logical-operand’ may have been intended to silence earlier diagnostics
cc1: all warnings being treated as errors
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <openssl/evp.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     int (* volatile tp)(void)=(int (*)(void))&t;
11:     printf("%d", (*tp)());
12:   }
13: 
14:   return !!argv[argc];
15: }
16: 
17: int t(void) { char s1[1024]; EVP_PKEY_new_raw_private_key(0, NULL, (unsigned char *)s1, 0); return 0; }
/* end */

The checking program used by have_func generally is not expected to be a perfectly valid program, so I think we should not apply -Werror to it. I think we can move the append_cflags logic in extconf.rb to after all checking code, right before create_makefile.

[junaruga]

I will check if this issue happens with Ruby 3.0.

[rhenium]

The EVP_PKEY_new_mac_key returns NULL
in the ossl_hmac_initialize in ext/openssl/ossl_hmac.c:107.

This is a known issue in OpenSSL 3.0 when the HMAC key is empty: openssl/openssl#13089

However, this code is supposed to be unreachable with OpenSSL >= 3.0 (#538).

[junaruga]

CC: @cwjenkins

Thank you for providing the info!

I checked the CI results, and the "ubuntu-22.04 (ruby) 3.1" case fails and "ubuntu-22.04 (ruby) 3.0" passes. So, I assumed that this is a Ruby version specific issue. Because it seemed that it was executed in the same environment.
https://github.com/junaruga/openssl/actions/runs/5125132987/jobs/9217902633

But it was a wrong assumption. I checked the used OpenSSL versions now. And I found the difference of used OpenSSL versions.

• "ubuntu-22.04 (ruby) 3.1": OpenSSL 3.0.2 is used. The issue happened.
  https://github.com/junaruga/openssl/actions/runs/5125132987/jobs/9217902633#step:8:19
• "ubuntu-22.04 (ruby) 3.0": OpenSSL 1.1.1t is used. The issue didn't happen.
  https://github.com/junaruga/openssl/actions/runs/5125132987/jobs/9217902535#step:8:19

[junaruga]

The checking program used by have_func generally is not expected to be a perfectly valid program, so I think we should not apply -Werror to it. I think we can move the append_cflags logic in extconf.rb to after all checking code, right before create_makefile.

I see. The error happened in the have_func. And I agree with your idea moving to the right before create_makefile. Do you like to send the pull-request by yourself?

[rhenium]

Already! Haha #633

[junaruga]

All right. Thanks. :)

[rhenium]

I checked the CI results, and the "ubuntu-22.04 (ruby) 3.1" case fails and "ubuntu-22.04 (ruby) 3.0" passes.

This is probably because the checking program was not producing a warning on Ruby 3.0.

A change in Ruby 3.1 (ruby/ruby@cb5a41c) replaced the "" in the argument with a char s1[1024]. I don't really understand the intention, however.

However, I think it will likely happen again that a new warning starts to appear in the checking program (e.g., by OpenSSL converting a function to a macro), I think the safest workaround is to not apply -Werror.

[junaruga]

This is probably because the checking program was not producing a warning on Ruby 3.0.
A change in Ruby 3.1 (ruby/ruby@cb5a41c) replaced the "" in the argument with a char s1[1024]. I don't really understand the intention, however.

Sorry, I don't understand what the change in Ruby 3.1 (ruby/ruby@cb5a41c) is related to about the CI results I provided above.

In the "ubuntu-22.04 (ruby) 3.1" case, the rake compile with -Werror passed without the compiler warnings, and the rake test got the 1 error and 7 pendings that always happen with the -Werror in the case. In my understanding, the cause is by use of the OpenSSL 3.0.

And in the "ubuntu-22.04 (ruby) 3.1" case, the rake compile with -Werror passed without the compiler warnings and the rake test also passed.

[rhenium]

Sorry for the misleading quotation. I missed the fact the "ubuntu-22.04 (ruby) 3.0" uses OpenSSL 1.1.1.

The "openssl-3.0.8" configuration uses Ruby 3.0 + OpenSSL 3.0 + -Werror and it passed without failures or errors.

This is an issue when all of these are met: Ruby >= 3.1, OpenSSL >= 3.0, and -Werror.

[junaruga]

All right. That makes sense. 👍🏼

[junaruga]

I don't see this issue on the CI environments on the #631. O assume that the #633 fixed it. Anyway, I think we can close this ticket.