mokkabonna/json-schema-merge-allof

Simplify your schema by combining allOf

json-schema-merge-allof

Merge schemas combined using allOf into a more readable composed schema free from allOf.

npm install json-schema-merge-allof --save

Features

• Real and safe merging of schemas combined with allOf
• Takes away all allOf found in the whole schema
• Lossless in terms of validation rules, merged schema does not validate more or less than the original schema
• Results in a more readable root schema
• Removes almost all logical impossibilities
• Throws if no logical intersection is found (your schema would not validate anything from the start)
• Validates in a way not possible by regular simple meta validators
• Correctly considers additionalProperties, patternProperties and properties as a part of an whole when merging schemas containing those
• Correctly considers items and additionalItems as a whole when merging schemas containing those
• Supports merging schemas with items as array and direct schema
• Supports merging dependencies when mixed array and schema
• Supports all JSON schema core/validation keywords (v6, use custom resolvers to support other keywords)
• Option to override common impossibility like adding properties when using additionalProperties: false
• Pluggable keyword resolvers

How

Since allOf require ALL schemas provided (including the parent schema) to apply, we can iterate over all the schemas, extracting all the values for say, type, and find the intersection of valid values. Here is an example:

{
  type: ['object', 'null'],
  additionalProperties: {
    type: 'string',
    minLength: 5
  },
  allOf: [{
    type: ['array', 'object'],
    additionalProperties: {
      type: 'string',
      minLength: 10,
      maxLength: 20
    }
  }]
}

This result in the schema :

{
  type: 'object',
  additionalProperties: {
    type: 'string',
    minLength: 10,
    maxLength: 20
  }
}

Notice that type now excludes null and array since those are not logically possible. Also minLength is raised to 10. The other properties have no conflict and are merged into the root schema with no resolving needed.

For other keywords other methods are used, here are some simple examples:

• minLength, minimum, minItems etc chooses the highest value of the conflicting values.
• maxLength, maximum, maxItems etc chooses the lowest value of the conflicting values.
• uniqueItems is true if any of the conflicting values are true

As you can see above the strategy is to choose the most restrictive of the set of values that conflict. For some keywords that is done by intersection, for others like required it is done by a union of all the values, since that is the most restrictive.

What you are left with is a schema completely free of allOf. Except for in a couple of values that are impossible to properly intersect/combine:

not

When multiple conflicting not values are found, we also use the approach that pattern use, but instead of allOf we use anyOf. When extraction of common rules from anyOf is in place this can be further simplified.

Options

ignoreAdditionalProperties default false

Allows you to combine schema properties even though some schemas have additionalProperties: false This is the most common issue people face when trying to expand schemas using allOf and a limitation of the json schema spec. Be aware though that the schema produced will allow more than the original schema. But this is useful if just want to combine schemas using allOf as if additionalProperties wasn't false during the merge process. The resulting schema will still get additionalProperties set to false.

deep boolean, default true If false, resolves only the top-level allOf keyword in the schema.

If true, resolves all allOf keywords in the schema.

resolvers Object Override any default resolver like this:

mergeAllOf(schema, {
  resolvers: {
    title: function(values, path, mergeSchemas, options) {
      // choose what title you want to be used based on the conflicting values
      // resolvers MUST return a value other than undefined
    }
  }
})

The function is passed:

• values an array of the conflicting values that need to be resolved
• path an array of strings containing the path to the position in the schema that caused the resolver to be called (useful if you use the same resolver for multiple keywords, or want to implement specific logic for custom paths)
• mergeSchemas a function you can call that merges an array of schemas
• options the options mergeAllOf was called with

Combined resolvers

Some keyword are dependant on other keywords, like properties, patternProperties, additionalProperties. To create a resolver for these the resolver requires this structure:

mergeAllOf(schema, {
  resolvers: {
    properties:
      keywords: ['properties', 'patternProperties', 'additionalProperties'],
      resolver(values, parents, mergers, options) {

      }
    }
  }
})

This type of resolvers are expected to return an object containing the resolved values of all the associated keywords. The keys must be the name of the keywords. So the properties resolver need to return an object like this containing the resolved values for each keyword:

{
    properties: ...,
    patternProperties: ...,
    additionalProperties: ...,
}

Also the resolve function is not passed mergeSchemas, but an object mergers that contains mergers for each of the related keywords. So properties get passed an object like this:

const mergers = {
    properties: function mergeSchemas(schemas, childSchemaName){...},
    patternProperties: function mergeSchemas(schemas, childSchemaName){...},
    additionalProperties: function mergeSchemas(schemas){...},
}

Some of the mergers requires you to supply a string of the name or index of the subschema you are currently merging. This is to make sure the path passed to child resolvers are correct.

Default resolver

You can set a default resolver that catches any unknown keyword. Let's say you want to use the same strategy as the ones for the meta keywords, to use the first value found. You can accomplish that like this:

mergeJsonSchema({
  ...
}, {
  resolvers: {
    defaultResolver: mergeJsonSchema.options.resolvers.title
  }
})

Resolvers

Resolvers are called whenever multiple conflicting values are found on the same position in the schemas.

You can override a resolver by supplying it in the options.

Lossy vs lossless

All built in reducers for validation keywords are lossless, meaning that they don't remove or add anything in terms of validation.

For meta keywords like title, description, $id, $schema, default the strategy is to use the first possible value if there are conflicting ones. So the root schema is prioritized. This process possibly removes some meta information from your schema. So it's lossy. Override this by providing custom resolvers.

$ref

If one of your schemas contain a $ref property you should resolve them using a ref resolver like json-schema-ref-parser to dereference your schema for you first. Resolving $refs is not the task of this library. Currently it does not support circular references either. But if you use bundle in json-schema-ref-parser it should work as expected.

Other libraries

There exists some libraries that claim to merge schemas combined with allOf, but they just merge schemas using a very basic logic. Basically just the same as lodash merge. So you risk ending up with a schema that allows more or less than the original schema would allow.

Restrictions

We cannot merge schemas that are a logical impossibility, like:

{
  type: 'object',
  allOf: [{
    type: 'array'
  }]
}

The library will then throw an error reporting the values that had no valid intersection. But then again, your original schema wouldn't validate anything either.

Roadmap

• Treat the interdependent validations like properties and additionalProperties as one resolver (and items additionalItems)
• Extract repeating validators from anyOf/oneOf and merge them with parent schema
• After extraction of validators from anyOf/oneOf, compare them and remove duplicates.
• If left with only one in anyOf/oneOf then merge it to the parent schema.
• Expose seperate tools for validation, extraction
• Consider adding even more logical validation (like minLength <= maxLength)

Contributing

Create tests for new functionality and follow the eslint rules.

License

MIT © Martin Hansen

json-schema-merge-allof's Issues

Need a config option to allow not merging certain `allOf` schemas

allOf schemas using multiple conditional schemas using if, then, and else cannot be merged. E.g.

  allOf:
    - if:   { properties: { foo: { const: true } } }
      then: { required: [ bar ] }
    - if:   { properties: { xyz: { const: true } } }
      then: { required: [ abc ] } 

There is no way to write a resolver for if and then.

If would be useful to have an option that tell the library to skip trying to resolve certain schema keys and leave them in the allOf.

Required resolver not being executed

eIt seems as if the required value resolved is not firing when mixing required and non-required values:

export function mergeSchemas(schemas: any[]) {
  return mergeAllOf({
    allOf: schemas,
  }, {
    resolvers: {
      type: (values) => {
        const finalType = []
        for (const value of values) {
          if (Array.isArray(value)) {
            finalType.push(...value)
          } else {
            finalType.push(value)
          }
        }
        return finalType as any
      },
      required: (values) => {
        console.log("required resolver", values)
        return false
      }
    }
  })
}

console.log(mergeSchemas([ {
			"type": "object",
			"properties": {
				"hey": {
					"type": "string",
					"required": true
				},
				"this": {
					"type": "number",
					"required": true
				},
				"is": {
					"type": "array",
					"items": {
						"type": "string",
						"required": true
					},
					"required": true
				}
			},
			"required": true
		},  {
			"type": "object",
			"properties": {
				"hey": {
					"type": "string"
				},
				"this": {
					"type": "number"
				},
				"is": {
					"type": "array",
					"items": {
						"type": "string"
					}
				}
			}
		}]))

Outputs:

{
  type: 'object',
  required: true,
  properties: {
    hey: { type: 'string', required: true },
    this: { type: 'number', required: true },
    is: { type: 'array', required: true, items: [Object] }
  }
}

notice that:

1. I am returning false always for the resolver
2. There is no console logging of the required resolver line

Impossible to write resolver that ignores values coming from allOf items

I think that the best way to handle fields title, description and examples is to preserve what is declared in the schema itself and ignore everything coming from allOf items.

Consider the following schema:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "definitions": {
    "developer": {
      "title": "Developer",
      "description": "A nerd",
      "type": "object",
      "properties": {
        "knowsLanguages": {
          "type": "array",
          "items": { "type": "string" }
        }
      }
    },
    "manager": {
      "title": "Manager",
      "description": "People person",
      "type": "object",
      "properties": {
        "maxNumberOfSubordinates": { "type": "number" }
      }
    }
  },
  "title": "Team",
  "properties": {
    "teamLeader": {
      "description": "The Boss",
      "allOf": [
        {
          "$ref": "#/definitions/developer"
        },
        {
          "$ref": "#/definitions/manager"
        }
      ]
    }
  }
}

If we resolve $refs and run merge, we got title copied from the first type (Developer).
This just makes no sense.

So, I need a custom resolver that could detect whether value comes from parent schema (and keep it) or from items of allOf (and ignore it).

The problem is that it is impossible having current set of arguments resolver is called with.
Please, add a way.
Or, better, make this behavior as default.

Porting to typescript

Thanks for the lib it is really usefull.
I didn't managed to make it work in angular in a typescript context.
I think porting this lib to make it work on typescript can be really usefull.

Is this library still maintained?

It seems like the last commit was over a year ago. We are actively using this library and would hate to see it no longer being active. Please let us know whether we can expect more support.

Doesn't work with schemas that have cyclical references

Some schemas have cyclical references. For example, a definition in the schema tree references a parent or ancestor

For an example in the wild see the bitbucket API swagger definition. https://api.bitbucket.org/swagger.json. The schema definition for "repository" in the definitions section has a reference to a parent repository.

Trying to use json-schema-merge-allof on this schema will cause Exception: Maximum call stack size exceeded because of infinite recursion.

For now, I can set deep: false but it would be nice if the library could detect that it is in an infinite recursion or alternatively, I could set a depth that would set the maximum recursions on a schema.

how to keep the local references?

I'd like to eliminate allOf but to keep the references, in fact I have to keep them, because the schema is recursive.
If not, can you consider supporting local refs. json-schema-ref-parser has a mode to make all refs local.

Could you update the releases and package.json?

npm install without specifying a version doesn't pull latest.

Allow Different Resolver Functionality based on Parent Field

First of all, I would like to thank you for a very good allOf resolver. I really appreciate the work.

I do have a slight tricky use case that I would like to be able to solve. I am using this library to merge a lot of different schemas and at times it would be very beneficial to have a different resolver based on what the actual field is. Here would be an example:

Schema A:
{
    "country": {
        enum: [ 'US'] 
    },
    "identificationType": {
        enum: [ 'SOCIAL_SECURITY' ]
    }
}
Schema B:
{
    "country": {
        enum: [ 'CA'] 
    },
    "identificationType": {
        enum: [ 'PASSPORT' ]
    }
}

Say in this case, I would like the country field to overwrite by taking the last value, but would like the identificationType to merge all values.

Today, as far as I'm aware, I have only be able to define a function per enum. I thought I was going to be able to use the key argument but that for to support the same function across types.

I feel like this is a good case to use anyOf. Would it be possible to update to be able to resolve anyOf with merged schemas?

Thanks!

Unit tests with patterns

The unit tests create patterns like (?=bar)(?=foo),
e.g. in the test that merges contains.

Yet this resulting pattern seems to be unsatisfiable.

The following instances are valid w.r.t. the original schema but not the merged schema:

[
  {
    "name": "foo-and-bar"
  }
]

[
    {
        "name": "foo"
    },
    {
        "name": "bar"
    }
]

Package does not get transpiled when used with webpack/babel

I am using this as dependency/part of https://github.com/rjsf-team/react-jsonschema-form/ and arrow functions and ... rest operator where not transpiled, that is, the app was not running in IE11.

I guess when publishing to npm the package should be transpiled.

I fixed locally with having this workaround in my webpack.config.js, that is, adding the package to be resolved locally:

        {
          test: /\.(js|jsx)$/,
          include: [
            path.resolve("src"),
            path.resolve("./node_modules/json-schema-compare"),
            path.resolve("./node_modules/json-schema-merge-allof")
          ],
          loader: "babel-loader",
        },

Merge incompatible types

If there are schema with with types string and number being merged, is there an option to have the merged schema come out with type: ['string', 'number']?

Process schemas nested inside another document (like in OpenAPI schema)

First of all, thank you for this awesome library! It saved me a lot of effort for rewriting API specification due to allOf and additionalProperties: false (see http://json-schema.org/understanding-json-schema/reference/combining.html#id5 and https://stackoverflow.com/questions/22689900/json-schema-allof-with-additionalproperties for details) but still requires some additional work to use it with OpenAPI specs.

Problem
In OpenAPI specs, JSON schemas for requests and responses are located deep inside the API spec:

Fragment of an example OpenAPI spec for reference

openapi: 3.0.1
info:
  title: Example Application API
  version: v1
paths:
  /applications:
    get:
      summary: Gets a list of Applications.
      responses:
        '200':
          description: Success
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/ExtendedApplication'
components:
  schemas:
    Application:
      required:
        - name
        - id
      type: object
      properties:
        id:
          type: string
        name:
          type: string
      additionalProperties: false
    ExtendedApplication:
      allOf:
        - $ref: '#/components/schemas/Application'
        - type: object
          properties:
            additionalInfo:
              type: string

And if I feed whole spec into mergeAllOf then all the subschemas won't get merged automatically. Is there any option for this?

Workaround

Write recursive function to find subschemas in a swagger spec.

async function preprocessOpenapiSpec(apiSpec) {
  const schema = await $RefParser.dereference(apiSpec)

  function mergeAllOfRecursively(candidate, parent, k) {
    if (typeof candidate !== "object") return

    if (candidate.allOf) {
      parent[k] = mergeAllOf(candidate, { ignoreAdditionalProperties: true, deep: true })
    } else {
      for (const [key, value] of Object.entries(candidate)) {
        mergeAllOfRecursively(value, candidate, key)
      }
    }
  }

  for (const [key, value] of Object.entries(schema)) {
    mergeAllOfRecursively(value, apiSpec, key)
  }

  return schema
}

Confirmation of browser support

Can you confirm whether browser environments are supported, and if any special processes are required to get it running in the browser.
Regards.

In what way should the order of items in an allOf matter for descriptions?

Should the order of the items in an allOf matter for a description property override? Seems that if a description is a lone schema in the second part of an allOf that it's ignored.

const mergeAllOf = require('json-schema-merge-allof');

console.log(
  mergeAllOf([
    {
      type: 'string',
      format: 'date-time',
      description: 'A description on the schema.',
    },
    {
      description: 'A description override in the `allOf`.',
    },
  ])
);

/* {
  type: 'string',
  format: 'date-time',
  description: 'A description on the schema.'
} */

const mergeAllOf = require('json-schema-merge-allof');
console.log(
  mergeAllOf([
    {
      description: 'A description override in the `allOf`.',
    },
    {
      type: 'string',
      format: 'date-time',
      description: 'A description on the schema.',
    },
  ])
);

/* {
  description: 'A description override in the `allOf`.',
  type: 'string',
  format: 'date-time'
} */

Merging compatible Integers and Numbers

Integer and Number JSON Schemas are immediately marked as incompatible without inspecting the other keywords in the schema. If a JSON Schema is of "type": "number" and has a "multipleOf" keyword value that is an integer, then it can be merged with a JSON Schema of "type": "integer" and should maintain the "multipleOf" keyword. For example:

{
  "type": "object",
  "properties": {
    "num": {
      "allOf": [
        {
          "type": "integer",
          "minimum": 1,
          "maximum": 12
        },
        {
          "type": "number",
          "minimum": 2,
          "maximum": 42,
          "multipleOf": 2
        }
      ]
    }
  }
};

can be merged into:

{
  "type": "object",
  "properties": {
    "num": {
      "type": "integer",
      "minimum": 2,
      "maximum": 12,
      "multipleOf": 2
    }
  }
};

What is version 0.7.0?

@mokkabonna I noticed that version 0.7.0 was published to npm 3 months ago.

However, I don't see any 0.7.0 tag in the github source repo -- so which version of the code does 0.7.0 refer to?

Merging formats

format should be merged if possible.

For example, uri is a subset of uri-reference. So, both are merged to uri-reference.

Merging examples

Hiya, I'm not sure if this should be in json-schema-merge-allof or just left up to me to write a custom resolver.

When merging JSONSchemas, I think we can assume that any examples of each of the to-be-merged schemas should also be merged. Since JSONSchema examples is an array, we need to merge each corresponding examples array element in each of the to-be-merged schemas. When encountering arrays, the default behavior seems to be to concatenate them, but examples should have each element merged.

I wrote a custom resolver for examples here: https://github.com/wikimedia/jsonschema-tools/pull/14/files
but I could alternatively submit a PR to make this the default behavior for examples if that would make sense to you.

Feel free to decline this issue if you'd rather not change the behavior.