Relative $ref's against absolute id URL don't (always) work

[everit-org]

Relative $ref's against absolute id URL don't (always) work

[alessiostalla]

Hmm. Perhaps it would also work with a file:/// URL without an embedded web server?

Anyway, in this case the id is absolute, so the approach would not work. But, even if the solution is not ideal, the only case left uncovered is that of a third-party schema with an absolute id that does not point to the actual JSON schema document. With XML schemas sadly that happens often, perhaps with JSON schemas that's less common because the specification is recent and people today care more about this stuff.

[erosb]

Hi! I'm not sure if such behavior would conform to the specs. id and $ref handling has been a sensitive topic during the development of this library. I should take extra care when changing this, to avoid breaking backward-compatibility.

[alessiostalla]

I didn't read the spec's fine print, but the example I linked is supposed to be conforming (it is on the JSON Schema spec website!). I'm not in desperate need of a fix, but I have to point out that the current implementation will always hit the network when handling schemas with an id set and relative inner $ref's, which implies that, if you're developing a schema and want to use it for validation, you have either to omit the id until you publish the schema, or you have to publish it at its definitive location even during development.

[erosb]

Hi!

Your usecase is clearly valid, so it should be possible to perform schema
loading without publishing everything at a final (probably prod env)
location.

Currently the best workaround I can recommend to achieve it is to

• use relative URIs in the schema
• for the time of validation, start an embedded jetty instance which serves
  the schema files
• initialize the schema loader instance to use the embedded jetty's address
  as an initial resolution scope
  https://github.com/everit-org/json-schema#resolution-scopes.

Not elegant, but it works. Still requires network communication, but it is
local. This is the same mechanism as how the integration tests of this
library work.

Regarding your specific example: regarding pointer resolution, the schema
spec defines 2 algorithms to perform it: canonical dereferencing and inline
dereferencing. Canonical must be supported mandatorily by implementations,
while inline dereferencing is optional. I suspect that this particular
example utilizes inline dereferencing. But I will double-check it.

I hope it helps.

2016-08-17 9:11 GMT+02:00 Alessio Stalla [email protected]:

I didn't read the spec's fine print, but the example I linked is supposed
to be conforming (it is on the JSON Schema spec website!). I'm not in
desperate need of a fix, but I have to point out that the current
implementation will always hit the network when handling schemas with
an id set and relative inner $ref's, which implies that, if you're
developing a schema and want to use it for validation, you have either to
omit the id until you publish the schema, or you have to publish it at its
definitive location even during development.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#47 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAQrBpp2ue8AuN4ZtshUSLioom3YqB3Hks5qgrQTgaJpZM4Ji2Kk
.

Bence Erősme@github https://github.com/erosb

[erosb]

Probably the solution will be installing a custom protocol handler. I found this, very detailed description about custom protocol handling in java: http://stackoverflow.com/questions/861500/url-to-load-resources-from-the-classpath-in-java

So probably the library should support a protocol handler to read resources from the classpath.

[elderjeb]

@alessiostalla, the file:/// URL is what I used to get it working in my test case. Based on the example:

`SchemaLoader schemaLoader = SchemaLoader.builder().schemaJson(rawSchema).resolutionScope("file:///Users/masterjeb/Documents/Workspace/JSONValidator/resources/schema/").build();

{"$ref":"Event.json"},

[dseltzer]

I tried to resolve this using the method @masterjeb mentioned, but I am running into a SchemaException.

Code

class JsonValidationTest {

    @Test
    void validatePerson() {
        def personJsonSchemaPath = "src/test/resources/person-schema.json"
        def personJsonPath = "src/test/resources/person.json"
        def personSchemaJson = new File( personJsonSchemaPath ).text
        def personJson = new File( personJsonPath ).text

        def loader = SchemaLoader.builder()
                .schemaJson( new JSONObject( personSchemaJson ) )
                .resolutionScope( "file:" + Paths.get("").toUri().path + "src/test/resources/" )
                .build()

        def schema = loader.load().build()

        schema.validate( new JSONObject( personJson ) )
    }
}

Where personJsonSchema is based on http://json-schema.org/examples.html with some inline definitions sprinkled in:

person-schema.json

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "title": "Person",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string"
    },
    "lastName": {
      "type": "string"
    },
    "age": {
      "description": "Age in years",
      "type": "integer",
      "minimum": 0
    },
    "interest": {
      "anyOf": [
        {
          "$ref": "#/definitions/baseball"
        },
        {
          "$ref": "#/definitions/music"
        }
      ]
    }
  },
  "required": [
    "firstName",
    "lastName",
    "interest"
  ],
  "definitions": {
    "baseball": {
      "properties": {
        "type": {
          "type": "string",
          "enum": [
            "Sport"
          ]
        }
      }
    },
    "music": {
      "properties": {
        "type": {
          "type": "string",
          "enum": [
            "Art"
          ]
        }
      }
    }
  }
}

And personJson is:

person.json

{
    "firstName": "John",
    "lastName": "Smith",  
    "interest": { 
    	"type": "Art"
    }      
}

The refs "$ref": "#/definitions/baseball" and "$ref": "#/definitions/music" inside of the interests schema will cause the validate method to fail with the following Exception:

Exception

org.everit.json.schema.SchemaException: failed to parse person.jsonperson-schema.json

	at org.everit.json.schema.loader.internal.JSONPointer.executeWith(JSONPointer.java:94)
	at org.everit.json.schema.loader.internal.JSONPointer.lambda$forURL$1(JSONPointer.java:133)
	at org.everit.json.schema.loader.internal.JSONPointer.query(JSONPointer.java:152)
	at org.everit.json.schema.loader.ReferenceLookup.lookup(ReferenceLookup.java:99)
	at org.everit.json.schema.loader.SchemaLoader.buildSchemaWithoutExplicitType(SchemaLoader.java:253)
	at org.everit.json.schema.loader.SchemaLoader.lambda$load$0(SchemaLoader.java:290)
	at java.util.Optional.orElseGet(Optional.java:267)
	at org.everit.json.schema.loader.SchemaLoader.load(SchemaLoader.java:288)
	at org.everit.json.schema.loader.SchemaLoader.loadChild(SchemaLoader.java:346)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.IntPipeline$4$1.accept(IntPipeline.java:250)
	at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
	at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
	at org.everit.json.schema.loader.CombinedSchemaLoader.load(CombinedSchemaLoader.java:61)
	at org.everit.json.schema.loader.SchemaLoader.load(SchemaLoader.java:287)
	at org.everit.json.schema.loader.SchemaLoader.loadChild(SchemaLoader.java:346)
	at org.everit.json.schema.loader.ObjectSchemaLoader.lambda$addPropertySchemaDefinition$4(ObjectSchemaLoader.java:79)
	at org.everit.json.schema.loader.internal.TypeBasedMultiplexer$IdModifyingTypeConsumerImpl.lambda$then$0(TypeBasedMultiplexer.java:78)
	at org.everit.json.schema.loader.internal.TypeBasedMultiplexer.orElse(TypeBasedMultiplexer.java:233)
	at org.everit.json.schema.loader.internal.TypeBasedMultiplexer.requireAny(TypeBasedMultiplexer.java:243)
	at org.everit.json.schema.loader.ObjectSchemaLoader.addPropertySchemaDefinition(ObjectSchemaLoader.java:81)
	at org.everit.json.schema.loader.ObjectSchemaLoader.lambda$populatePropertySchemas$3(ObjectSchemaLoader.java:70)
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
	at org.everit.json.schema.loader.ObjectSchemaLoader.populatePropertySchemas(ObjectSchemaLoader.java:69)
	at org.everit.json.schema.loader.ObjectSchemaLoader.lambda$load$0(ObjectSchemaLoader.java:34)
	at org.everit.json.schema.loader.internal.TypeBasedMultiplexer$IdModifyingTypeConsumerImpl.lambda$then$0(TypeBasedMultiplexer.java:78)
	at org.everit.json.schema.loader.internal.TypeBasedMultiplexer.orElse(TypeBasedMultiplexer.java:233)
	at org.everit.json.schema.loader.internal.TypeBasedMultiplexer.requireAny(TypeBasedMultiplexer.java:243)
	at org.everit.json.schema.loader.ObjectSchemaLoader.load(ObjectSchemaLoader.java:35)
	at org.everit.json.schema.loader.SchemaLoader.buildObjectSchema(SchemaLoader.java:324)
	at org.everit.json.schema.loader.SchemaLoader.loadForExplicitType(SchemaLoader.java:317)
	at org.everit.json.schema.loader.SchemaLoader.loadForType(SchemaLoader.java:335)
	at org.everit.json.schema.loader.SchemaLoader.lambda$load$0(SchemaLoader.java:292)
	at java.util.Optional.orElseGet(Optional.java:267)
	at org.everit.json.schema.loader.SchemaLoader.load(SchemaLoader.java:288)
	at org.everit.json.schema.loader.SchemaLoader$load$0.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
	at com.iai.json.JsonValidationTest.validatePerson(JsonValidationTest.groovy:26)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: org.json.JSONException: A JSONObject text must begin with '{' at 1 [character 2 line 1]
	at org.json.JSONTokener.syntaxError(JSONTokener.java:451)
	at org.json.JSONObject.<init>(JSONObject.java:195)
	at org.everit.json.schema.loader.internal.JSONPointer.executeWith(JSONPointer.java:90)
	... 72 more

The only way I found to correct this is to explicitly insert the name of the schema file in the ref values: e.g. "$"ref": "person-schema.json#/definitions/baseball" and "$ref": "person-schema.json#/definitions/music".

Is this expected behavior? If not, any suggestions for another way to resolve this?

[erosb]

@dseltzer I think your example would work if you wouldn't set any resolutionScope explicitly.