Revise schema and formats facility

[secure-systems-lab]

Revise schema and formats facility

[lukpueh]

Just discovered another documented, albeit (to me) unexpected behavior:

• Unrecognized keys in an object schema are allowed

>>> from securesystemslib import schema
>>> FOO_SCHEMA = schema.Object("FOO_SCHEMA", foo=schema.AnyString())
>>> FOO_SCHEMA.matches({"foo": "ABC", "bar": "interesting..."}) # Would expect to return False due to "bar"
True

[lukpueh]

• The error messages from checking schemas with the check_match method are often not helpful, because they usually don't show the value of the checked variable or lack context.

Similar observations were made in theupdateframework/python-tuf#243

[lukpueh]

The many check_match-calls also have an impact on performance. E.g. when delegating 16K hashed bins for warehouse, >5% of the execution time was just check_match-ing.

[lukpueh]

While testing a format errors on different Python versions, I just discovered another issue with the Object(Schema):

Given that in Python <3.7 the dict key order is not guaranteed, iterating over the "required" fields ...

securesystemslib/securesystemslib/schema.py

Line 750 in 1c4161e

self._required = list(required.items())

... of a schema object in check_match may yield different error messages, if multiple required fields are missing.

securesystemslib/securesystemslib/schema.py

Lines 759 to 768 in 1c4161e

	for key, schema in self._required:
	# Check if 'object' has all the required dict keys. If not one of the
	# required keys, check if it is an Optional().
	try:
	item = object[key]
	except KeyError:
	# If not an Optional schema, raise an exception.
	if not isinstance(schema, Optional):
	message = 'Missing key ' + repr(key) + ' in ' + repr(self._object_name)