test_pkey_ec.rb test failures in OpenSSL FIPS

[junaruga]

test_pkey_ec.rb test failures in OpenSSL FIPS

[rhenium]

Yes, test-unit-ruby-core 1.0.2 fixed assert_nothing_raised for me. PR #673 lets this repository use the gem instead of embedding core_assertions.rb.

[rhenium]

I submitted the above patch as #675. This should work without and with the FIPS provider without reducing test coverage.

The other failure is probably the same issue as #643, the usage of MD5.

[rhenium]

The "-----BEGIN EC PRIVATE KEY-----" PEM can't be decoded in FIPS 140-compliant systems because it uses MD5 to derive encryption keys from passwords (#643, #645). I don't think AES-128-CBC is the issue here.

The error message is unfortunate. As you analyzed it, unlike other PKey types, OpenSSL::PKey::EC.new(string) attempts to treat the given string as a curve name if it turns out not to be a PEM encoding, but the string starting with "-----BEGIN EC PRIVATE KEY-----" is not a curve name either.

The test case (and also corresponding ones in other test_pkey_*.rb files) should be skipped.

[junaruga]

Error: test_ec_key(OpenSSL::TestEC): NoMethodError: undefined method `filter_backtrace' for module Test
/home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:188:in `block in assert_nothing_raised'

For the error above, where does the Test.filter_backtrace come from?

openssl/test/lib/core_assertions.rb

Lines 185 to 190 in f4b8dac

	rescue *(args.empty? ? Exception : args) => e
	msg = message(msg) {
	"Exception raised:\n<#{mu_pp(e)}>\n""Backtrace:\n" <<
	Test.filter_backtrace(e.backtrace).map{|frame| " #{frame}"}.join("\n")
	}
	raise Test::Unit::AssertionFailedError, msg.call, e.backtrace

[junaruga]

For the error above, where does the Test.filter_backtrace come from?

The test/lib/core_assertions.rb including the Test.filter_backtrace was added by the 520601e by @hsbt. @hsbt, do you know why the error by the Test.filter_backtrace happens?

And I am seeing the same name method filter_backtrace in the test-unit. Is it just co-incidence?
https://github.com/test-unit/test-unit/blob/8cd57617a3cb470985cf5768691e10ca72928a1c/lib/test/unit/util/backtracefilter.rb#L20-L55

[rhenium]

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.generate("secp112r1").check_key'
-e:1:in `check_key': EVP_PKEY_check: initialization error (OpenSSL::PKey::ECError)
	from -e:1:in `<main>'

secp112r1 isn't allowed in FIPS 140.

[rhenium]

Error: test_ec_key(OpenSSL::TestEC): NoMethodError: undefined method `filter_backtrace' for module Test
/home/jaruga/var/git/ruby/openssl/test/lib/core_assertions.rb:188:in `block in assert_nothing_raised'

For the error above, where does the Test.filter_backtrace come from?

openssl/test/lib/core_assertions.rb

Lines 185 to 190 in f4b8dac

	rescue *(args.empty? ? Exception : args) => e
	msg = message(msg) {
	"Exception raised:\n<#{mu_pp(e)}>\n""Backtrace:\n" <<
	Test.filter_backtrace(e.backtrace).map{|frame| " #{frame}"}.join("\n")
	}
	raise Test::Unit::AssertionFailedError, msg.call, e.backtrace

First, we should switch to using https://github.com/ruby/test-unit-ruby-core instead of embedding core_assertions.rb. This is just not yet worked on.

The same issue is in ruby/test-unit-ruby-core. Test.filter_backtrace appears to be defined in ruby/ruby's tool/lib/test/unit.rb. It could be included in, too, but there doesn't seem a need for CoreAssertions to define assert_nothing_raised since test-unit already provides this assertion method.

[hsbt]

ruby/test-unit-ruby-core#2 (comment) is helpful for openssl?

[junaruga]

@hsbt @rhenium thanks! On the latest ruby/opessl including the #673, the assert_nothing_raised fails properly.

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  bundle exec ruby -I./lib -ropenssl test/openssl/test_pkey_ec.rb
Loaded suite test/openssl/test_pkey_ec
Started
E
==========================================================================================
Error: test_ECPrivateKey_encrypted(OpenSSL::TestEC): OpenSSL::PKey::ECError: invalid curve name
test/openssl/test_pkey_ec.rb:247:in `initialize'
test/openssl/test_pkey_ec.rb:247:in `new'
test/openssl/test_pkey_ec.rb:247:in `test_ECPrivateKey_encrypted'
     244:     0/dGSU5SzFG+iT9iFXCwCvv+bxyegkBOyALFje1NAsM=
     245:     -----END EC PRIVATE KEY-----
     246:     EOF
  => 247:     key = OpenSSL::PKey::EC.new(pem, "abcdef")
     248:     assert_same_ec p256, key
     249:     key = OpenSSL::PKey::EC.new(pem) { "abcdef" }
     250:     assert_same_ec p256, key
==========================================================================================
F
==========================================================================================
Failure: test_ec_key(OpenSSL::TestEC):
  Exception raised:
  <#<OpenSSL::PKey::ECError: EVP_PKEY_check: initialization error>>
  Backtrace:
    test/openssl/test_pkey_ec.rb:19:in `check_key'
    test/openssl/test_pkey_ec.rb:19:in `block (2 levels) in test_ec_key'.
test/openssl/test_pkey_ec.rb:19:in `check_key'
test/openssl/test_pkey_ec.rb:19:in `block (2 levels) in test_ec_key'
     16:       key = OpenSSL::PKey::EC.generate(curve_name)
     17:       assert_predicate key, :private?
     18:       assert_predicate key, :public?
  => 19:       assert_nothing_raised { key.check_key }
     20:     end
     21: 
     22:     key1 = OpenSSL::PKey::EC.generate("prime256v1")
/home/jaruga/var/git/ruby/openssl/bundle/ruby/3.3.0+0/gems/test-unit-ruby-core-1.0.2/lib/core_assertions.rb:219:in `assert_nothing_raised'
test/openssl/test_pkey_ec.rb:19:in `block in test_ec_key'
test/openssl/test_pkey_ec.rb:11:in `each'
test/openssl/test_pkey_ec.rb:11:in `test_ec_key'
==========================================================================================
Finished in 0.050731255 seconds.
------------------------------------------------------------------------------------------
16 tests, 124 assertions, 1 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
87.5% passed
------------------------------------------------------------------------------------------
315.39 tests/s, 2444.25 assertions/s

[junaruga]

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.generate("secp112r1").check_key'
-e:1:in `check_key': EVP_PKEY_check: initialization error (OpenSSL::PKey::ECError)
	from -e:1:in `<main>'

secp112r1 isn't allowed in FIPS 140.

How did you know that? Could you share a document link that you checked for that?

[junaruga]

secp112r1 isn't allowed in FIPS 140.

How did you know that? Could you share a document link that you checked for that?

Is it https://www.openssl.org/source/ - The OpenSSL 3.0.0 or 3.0.8 security policy document?

[rhenium]

secp112r1, defined on a 112-bit finite field, would provide at most 56 bits of security. I don't know how to cite the specifications for FIPS 140, but it must definitely be prohibited for any use.

The test case in question is about OpenSSL::PKey::EC.builtin_curves. I don't think the current assertions make much sense anyway. I think it can use something like:

diff --git a/test/openssl/test_pkey_ec.rb b/test/openssl/test_pkey_ec.rb
index e5fef940a6c3..ab777a8b48a4 100644
--- a/test/openssl/test_pkey_ec.rb
+++ b/test/openssl/test_pkey_ec.rb
@@ -5,20 +5,6 @@
 
 class OpenSSL::TestEC < OpenSSL::PKeyTestCase
   def test_ec_key
-    builtin_curves = OpenSSL::PKey::EC.builtin_curves
-    assert_not_empty builtin_curves
-
-    builtin_curves.each do |curve_name, comment|
-      # Oakley curves and X25519 are not suitable for signing and causes
-      # FIPS-selftest failure on some environment, so skip for now.
-      next if ["Oakley", "X25519"].any? { |n| curve_name.start_with?(n) }
-
-      key = OpenSSL::PKey::EC.generate(curve_name)
-      assert_predicate key, :private?
-      assert_predicate key, :public?
-      assert_nothing_raised { key.check_key }
-    end
-
     key1 = OpenSSL::PKey::EC.generate("prime256v1")
 
     # PKey is immutable in OpenSSL >= 3.0; constructing an empty EC object is
@@ -49,6 +35,17 @@ def test_ec_key
     end
   end
 
+  def test_builtin_curves
+    builtin_curves = OpenSSL::PKey::EC.builtin_curves
+    assert_not_empty builtin_curves
+    assert_equal 2, builtin_curves[0].size
+    assert_kind_of String, builtin_curves[0][0]
+    assert_kind_of String, builtin_curves[0][1]
+
+    builtin_curve_names = builtin_curves.map { |name, comment| name }
+    assert_include builtin_curve_names, "prime256v1"
+  end
+
   def test_generate
     assert_raise(OpenSSL::PKey::ECError) { OpenSSL::PKey::EC.generate("non-existent") }
     g = OpenSSL::PKey::EC::Group.new("prime256v1")

[junaruga]

secp112r1, defined on a 112-bit finite field, would provide at most 56 bits of security. I don't know how to cite the specifications for FIPS 140, but it must definitely be prohibited for any use.

The test case in question is about OpenSSL::PKey::EC.builtin_curves. I don't think the current assertions make much sense anyway. I think it can use something like:

diff --git a/test/openssl/test_pkey_ec.rb b/test/openssl/test_pkey_ec.rb
index e5fef940a6c3..ab777a8b48a4 100644
--- a/test/openssl/test_pkey_ec.rb
+++ b/test/openssl/test_pkey_ec.rb
@@ -5,20 +5,6 @@
 
 class OpenSSL::TestEC < OpenSSL::PKeyTestCase
   def test_ec_key
-    builtin_curves = OpenSSL::PKey::EC.builtin_curves
-    assert_not_empty builtin_curves
-
-    builtin_curves.each do |curve_name, comment|
-      # Oakley curves and X25519 are not suitable for signing and causes
-      # FIPS-selftest failure on some environment, so skip for now.
-      next if ["Oakley", "X25519"].any? { |n| curve_name.start_with?(n) }
-
-      key = OpenSSL::PKey::EC.generate(curve_name)
-      assert_predicate key, :private?
-      assert_predicate key, :public?
-      assert_nothing_raised { key.check_key }
-    end
-
     key1 = OpenSSL::PKey::EC.generate("prime256v1")
 
     # PKey is immutable in OpenSSL >= 3.0; constructing an empty EC object is
@@ -49,6 +35,17 @@ def test_ec_key
     end
   end
 
+  def test_builtin_curves
+    builtin_curves = OpenSSL::PKey::EC.builtin_curves
+    assert_not_empty builtin_curves
+    assert_equal 2, builtin_curves[0].size
+    assert_kind_of String, builtin_curves[0][0]
+    assert_kind_of String, builtin_curves[0][1]
+
+    builtin_curve_names = builtin_curves.map { |name, comment| name }
+    assert_include builtin_curve_names, "prime256v1"
+  end
+
   def test_generate
     assert_raise(OpenSSL::PKey::ECError) { OpenSSL::PKey::EC.generate("non-existent") }
     g = OpenSSL::PKey::EC::Group.new("prime256v1")

I see. The solution looks okay to me. I may notice that one issue related to the OpenSSL::PKey::EC.builtin_curves (the used EC_get_builtin_curves). The issue is it seems to me that the result of the EC_get_builtin_curves(curves, crv_len) in the code below looks same in both FIPS and non-FIPS cases.

https://github.com/ruby/openssl/blob/7c34a439bae71f110dd40be35454b68bd2b82e37/ext/openssl/ossl_pkey_ec.c#L885C9-L885C9

FIPS

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  ruby -I./lib -ropenssl -e 'p OpenSSL::PKey::EC.builtin_curves.size'
82

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  ruby -I./lib -ropenssl -e 'p OpenSSL::PKey::EC.builtin_curves[0]'
["secp112r1", "SECG/WTLS curve over a 112 bit prime field"]

Non-FIPS

$ ruby -I./lib -ropenssl -e 'p OpenSSL::PKey::EC.builtin_curves.size'
82

$ ruby -I./lib -ropenssl -e 'p OpenSSL::PKey::EC.builtin_curves[0]'
["secp112r1", "SECG/WTLS curve over a 112 bit prime field"]

This is weird to me. Because the EC_get_builtin_curves returns the curve_list.

https://github.com/openssl/openssl/blob/a2608e4bc430d6216bbf36f50a29278e8759103a/crypto/ec/ec_curve.c#L3316-L3331

And it seems that the curve_list is declared conditionally in both FIPS (FIPS_MODULE is defined) at crypto/ec/ec_curve.c#L2827 and non-FIPS (FIPS_MODULE not defined) crypto/ec/ec_curve.c#L2901. So, I expect the result of the EC_get_builtin_curves is different between FIPS and non-FIPS cases. I might see that FIPS_MODULE macro is not defined in the FIPS case. I plan to ask this to OpenSSL project.

[junaruga]

I submitted the above patch as #675. This should work without and with the FIPS provider without reducing test coverage.

The other failure is probably the same issue as #643, the usage of MD5.

Okay. Thanks!

[rhenium]

This is weird to me. Because the EC_get_builtin_curves returns the curve_list.

https://github.com/openssl/openssl/blob/a2608e4bc430d6216bbf36f50a29278e8759103a/crypto/ec/ec_curve.c#L3316-L3331

And it seems that the curve_list is declared conditionally in both FIPS (FIPS_MODULE is defined) at crypto/ec/ec_curve.c#L2827 and non-FIPS (FIPS_MODULE not defined) crypto/ec/ec_curve.c#L2901. So, I expect the result of the EC_get_builtin_curves is different between FIPS and non-FIPS cases. I might see that FIPS_MODULE macro is not defined in the FIPS case. I plan to ask this to OpenSSL project.

The EC_get_builtin_curves() called from ruby/openssl would be the one in libcrypto.so. Since this function isn't provider-aware (rather, crypto/ec is the backend used by the default/fips providers), and considering it's possible to use FIPS module and non-FIPS module at the same time with custom property query string (which is however not currently supported in ruby/openssl - https://www.openssl.org/docs/man3.0/man7/fips_module.html), I don't think this is fixable.

The commit that introduced the defined(FIPS_MODULE) version of the list (openssl/openssl@10c2564, 2019) says it's for tests.

[junaruga]

Thanks for the investigation. I found the issue ticket, openssl/openssl#18273 (comment) related to this topic, and commented there. It seems that the curve_list is not flexible.

[junaruga]

Error: test_ECPrivateKey_encrypted(OpenSSL::TestEC): OpenSSL::PKey::ECError: invalid curve name

A minimal reproducer

For the test failure above, below is a minimal reproducer. Does the "AES-128-CBC" mean MD5? And it is not allowed in FIPS 140? I found the AES's Wikipedia page including the FIPS 140 things.

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.new("-----BEGIN EC PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: AES-128-CBC,85743EB6FAC9EA76BF99D9328AFD1A66\n\nnhsP1NHxb53aeZdzUe9umKKyr+OIwQq67eP0ONM6E1vFTIcjkDcFLR6PhPFufF4m\ny7E2HF+9uT1KPQhlE+D63i1m1Mvez6PWfNM34iOQp2vEhaoHHKlR3c43lLyzaZDI\n0/dGSU5SzFG+iT9iFXCwCvv+bxyegkBOyALFje1NAsM=\n-----END EC PRIVATE KEY-----\n", "abcdef")'
-e:1:in `initialize': invalid curve name (OpenSSL::PKey::ECError)
	from -e:1:in `new'
	from -e:1:in `<main>'

I referred to the following code.

openssl/test/openssl/test_pkey_ec.rb

Lines 237 to 247 in 7c34a43

	pem = <<~EOF
	-----BEGIN EC PRIVATE KEY-----
	Proc-Type: 4,ENCRYPTED
	DEK-Info: AES-128-CBC,85743EB6FAC9EA76BF99D9328AFD1A66
	nhsP1NHxb53aeZdzUe9umKKyr+OIwQq67eP0ONM6E1vFTIcjkDcFLR6PhPFufF4m
	y7E2HF+9uT1KPQhlE+D63i1m1Mvez6PWfNM34iOQp2vEhaoHHKlR3c43lLyzaZDI
	0/dGSU5SzFG+iT9iFXCwCvv+bxyegkBOyALFje1NAsM=
	-----END EC PRIVATE KEY-----
	EOF
	key = OpenSSL::PKey::EC.new(pem, "abcdef")

Debug with GDB

FIPS case

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/ssl/openssl_fips.cnf \
  gdb --args ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.new("-----BEGIN EC PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: AES-128-CBC,85743EB6FAC9EA76BF99D9328AFD1A66\n\nnhsP1NHxb53aeZdzUe9umKKyr+OIwQq67eP0ONM6E1vFTIcjkDcFLR6PhPFufF4m\ny7E2HF+9uT1KPQhlE+D63i1m1Mvez6PWfNM34iOQp2vEhaoHHKlR3c43lLyzaZDI\n0/dGSU5SzFG+iT9iFXCwCvv+bxyegkBOyALFje1NAsM=\n-----END EC PRIVATE KEY-----\n", "abcdef")'
...
(gdb) set environment LD_LIBRARY_PATH /home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/lib
(gdb) b ossl_ec_key_initialize
(gdb) r
...
(gdb) f
#0  ec_key_new_from_group (arg=140737044503040)
    at ../../../../ext/openssl/ossl_pkey_ec.c:85
85		    ossl_raise(eECError, "invalid curve name");
(gdb) bt
#0  ec_key_new_from_group (arg=140737044503040)
    at ../../../../ext/openssl/ossl_pkey_ec.c:85
#1  0x00007fffe5802544 in ossl_ec_key_initialize (argc=2, argv=0x7fffe9eff048, 
    self=140737044094080) at ../../../../ext/openssl/ossl_pkey_ec.c:170
#2  0x00007ffff7cd34ff in ractor_safe_call_cfunc_m1 (recv=140737044094080, argc=2, 
    argv=0x7fffe9eff048, func=0x7fffe58023e4 <ossl_ec_key_initialize>)
    at vm_insnhelper.c:3242
#3  0x00007ffff7ced42a in vm_call0_cfunc_with_frame (ec=0x40a180, 
    calling=0x7fffffffc120, argv=0x7fffe9eff048) at vm_eval.c:170
#4  0x00007ffff7ced5a9 in vm_call0_cfunc (ec=0x40a180, calling=0x7fffffffc120, 
    argv=0x7fffe9eff048) at vm_eval.c:184
#5  0x00007ffff7ced815 in vm_call0_body (ec=0x40a180, calling=0x7fffffffc120, 
    argv=0x7fffe9eff048) at vm_eval.c:230
#6  0x00007ffff7cecfd9 in vm_call0_cc (ec=0x40a180, recv=140737044094080, id=3137, 
    argc=2, argv=0x7fffe9eff048, cc=0x7fffe5851c58, kw_splat=0) at vm_eval.c:107
#7  0x00007ffff7cee89c in rb_call0 (ec=0x40a180, recv=140737044094080, mid=3137, argc=2, 
    argv=0x7fffe9eff048, call_scope=CALL_FCALL, self=140737042249120) at vm_eval.c:566
#8  0x00007ffff7cef573 in rb_call (recv=140737044094080, mid=3137, argc=2, 
    argv=0x7fffe9eff048, scope=CALL_FCALL) at vm_eval.c:892
#9  0x00007ffff7cefbfb in rb_funcallv_kw (recv=140737044094080, mid=3137, argc=2, 
    argv=0x7fffe9eff048, kw_splat=0) at vm_eval.c:1085
#10 0x00007ffff7af3f7f in rb_obj_call_init_kw (obj=140737044094080, argc=2, 
    argv=0x7fffe9eff048, kw_splat=0) at eval.c:1687
#11 0x00007ffff7b9b136 in rb_class_new_instance_pass_kw (argc=2, argv=0x7fffe9eff048, 
    klass=140737042249120) at object.c:2107
#12 0x00007ffff7cd34ff in ractor_safe_call_cfunc_m1 (recv=140737042249120, argc=2, 
    argv=0x7fffe9eff048, func=0x7ffff7b9b0f9 <rb_class_new_instance_pass_kw>)
    at vm_insnhelper.c:3242
#13 0x00007ffff7cd414d in vm_call_cfunc_with_frame_ (ec=0x40a180, 
    reg_cfp=0x7fffe9ffef90, calling=0x7fffffffc730, argc=2, argv=0x7fffe9eff048, 
    stack_bottom=0x7fffe9eff040) at vm_insnhelper.c:3433
#14 0x00007ffff7cd4371 in vm_call_cfunc_with_frame (ec=0x40a180, reg_cfp=0x7fffe9ffef90, 
    calling=0x7fffffffc730) at vm_insnhelper.c:3461
#15 0x00007ffff7cd4482 in vm_call_cfunc_other (ec=0x40a180, reg_cfp=0x7fffe9ffef90, 
    calling=0x7fffffffc730) at vm_insnhelper.c:3487
#16 0x00007ffff7cd489a in vm_call_cfunc (ec=0x40a180, reg_cfp=0x7fffe9ffef90, 
    calling=0x7fffffffc730) at vm_insnhelper.c:3569
#17 0x00007ffff7cd6e2c in vm_call_method_each_type (ec=0x40a180, cfp=0x7fffe9ffef90, 
    calling=0x7fffffffc730) at vm_insnhelper.c:4327
#18 0x00007ffff7cd78d2 in vm_call_method (ec=0x40a180, cfp=0x7fffe9ffef90, 
    calling=0x7fffffffc730) at vm_insnhelper.c:4453
#19 0x00007ffff7cd7ad0 in vm_call_general (ec=0x40a180, reg_cfp=0x7fffe9ffef90, 
    calling=0x7fffffffc730) at vm_insnhelper.c:4497
#20 0x00007ffff7cda260 in vm_sendish (ec=0x40a180, reg_cfp=0x7fffe9ffef90, cd=0x7dd400, 
    block_handler=0, method_explorer=mexp_search_method) at vm_insnhelper.c:5487
#21 0x00007ffff7ce1c5b in vm_exec_core (ec=0x40a180, initial=0) at insns.def:835
#22 0x00007ffff7cf866c in rb_vm_exec (ec=0x40a180) at vm.c:2384
#23 0x00007ffff7cf94c6 in rb_iseq_eval_main (iseq=0x7fffe5851f28) at vm.c:2643
#24 0x00007ffff7af0c0b in rb_ec_exec_node (ec=0x40a180, n=0x7fffe5851f28) at eval.c:287
#25 0x00007ffff7af0d6c in ruby_run_node (n=0x7fffe5851f28) at eval.c:328
#26 0x00000000004011d9 in rb_main (argc=5, argv=0x7fffffffd898) at ./main.c:39
#27 0x0000000000401236 in main (argc=5, argv=0x7fffffffd898) at ./main.c:58

The error happens in the ec_key_new_from_group after failing to get the pkey from ossl_pkey_read_generic.

openssl/ext/openssl/ossl_pkey_ec.c

Lines 166 to 172 in 7c34a43

	pkey = ossl_pkey_read_generic(in, pass);
	BIO_free(in);
	if (!pkey) {
	ossl_clear_error();
	ec = ec_key_new_from_group(arg);
	goto legacy;
	}

The error happens in the ec_key_new_from_group.

openssl/ext/openssl/ossl_pkey_ec.c

Line 85 in 7c34a43

ossl_raise(eECError, "invalid curve name");

Non-FIPS case

In the non-FIPS case, the ossl_pkey_read_generic can get the pkey unlike FIPS case.

$ gdb --args ruby -I./lib -ropenssl -e 'OpenSSL::PKey::EC.new("-----BEGIN EC PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: AES-128-CBC,85743EB6FAC9EA76BF99D9328AFD1A66\n\nnhsP1NHxb53aeZdzUe9umKKyr+OIwQq67eP0ONM6E1vFTIcjkDcFLR6PhPFufF4m\ny7E2HF+9uT1KPQhlE+D63i1m1Mvez6PWfNM34iOQp2vEhaoHHKlR3c43lLyzaZDI\n0/dGSU5SzFG+iT9iFXCwCvv+bxyegkBOyALFje1NAsM=\n-----END EC PRIVATE KEY-----\n", "abcdef")'
(gdb) set environment LD_LIBRARY_PATH /home/jaruga/.local/openssl-3.2.0-dev-fips-debug-cf712830b7/lib
(gdb) b ossl_ec_key_initialize
...
(gdb) n
166	    pkey = ossl_pkey_read_generic(in, pass);
(gdb) n
167	    BIO_free(in);
(gdb) p pkey
$1 = (EVP_PKEY *) 0x7ea480
(gdb) f
#0  ossl_ec_key_initialize (argc=2, argv=0x7fffe9eff048, self=140737044094080)
    at ../../../../ext/openssl/ossl_pkey_ec.c:167
167	    BIO_free(in);

So, in the code below, after ossl_pkey_read_generic fails to get the pkey, then we want to call the ec_key_new_from_group in FIPS case? Because when ossl_pkey_read_generic fails to get the pkey, we can know the key is not FIPS-approved?

openssl/ext/openssl/ossl_pkey_ec.c

Lines 166 to 172 in 7c34a43

	pkey = ossl_pkey_read_generic(in, pass);
	BIO_free(in);
	if (!pkey) {
	ossl_clear_error();
	ec = ec_key_new_from_group(arg);
	goto legacy;
	}

[junaruga]

By the way, I was discussing how we can know a key or key name is allowed or not allowed in OpenSSL FIPS at openssl/openssl#21830. Unfortunately I don't find a clear way to know it. But I think if ossl_pkey_read_generic doesn't return the value for pkey, we can assume that the key is not allowed in OpenSSL FIPS.

[junaruga]

All right. I sent the PR to #681 fix the test_pkey_ec.rb.