microsoft/jschema

Includes an implementation of JSON Schema Draft 4, an implementation of JSON pointer, and a JSON-schema-to-C# code generator

Microsoft/jschema

A set of .NET components for working with JSON Schema Draft 4

The JSchema repo consists of the following components, each of which is documented in its own README.md:

• Microsoft.Json.Pointer: an implementation of the JSON Pointer specification (RFC 6901). Documentation: src/Json.Pointer/README.md.

• Microsoft.Json.Schema: an almost but not quite complete implementation of JSON Schema Draft 4. Documentation: coming soon.

• Microsoft.Json.Schema.Validation: a library to validate a JSON instance document against a JSON schema document. Documentation: coming soon.

• Microsoft.Json.Schema.Validation.Cli: a command-line tool to validate a JSON instance document against a JSON schema document, built on the Microsoft.Json.Schema.Validation library.

• Microsoft.Json.Schema.ToDotNet: a library to generate .NET classes from a JSON schema. Documentation (incomplete): src/Json.Schema.ToDotNet/README.md.

• Microsoft.Json.Schema.ToDotNet.Cli: a command-line tool to generate .NET classes from a JSON schema, built on the Microsoft.Json.Schema.ToDotNet library. Documentation: coming soon.

All facilities built from the JSchema repo, including the command line tools, are available for both the net461 and netcoreapp2.1 platforms.

This project has adopted the Microsoft Open Source Code of Conduct. For more information, see the Code of Conduct FAQ, or contact [email protected] with any additional questions or comments.

jschema's Issues

ToDotNet application can't find its resources

When you run the ToDotNet command line app, it fails because it can't find the resource stream. Somehow the .resx and the .Resources.designer files got separated in the project file. Fix the project file to connect the resources back up.

Code generation fails for an array of uri-formatted strings

With this property definition:

"workItemUris": {
  "description": "The URIs of the work items associated with this result",
  "type": "array",
  "minItems": 1,
  "uniqueItems": true,
  "items": {
    "type": "string",
    "format": "uri"
  }
},

JsonSchemaToDotNet.exe fails with the error:

EXEC : error : The schema does not contain information describing the property
or element WorkItemUris[][].

Removing "format": "uri" fixes the error, but that's not an option.

@michaelcfanning @EasyRhin0

Need code gen hint to avoid emitting default

@michaelcfanning wrote, in a comment on a PR in the SARIF SDK:

... we need to instantiate the Run type with DefaultColumnKind set to Utf16CodeUnits (which is not the literal default given by the schema, instead it's one that imposed by our Windows SDK to dig a 'pit of success' for this platform).

By doing this, however, we also need to make sure to strip the other default handling. Otherwise we would never round-trip the alternate value (and actual default UnicodeCodePoints). That bad scenario would look like this:

1. user creates a log file, we init to utf16codeunits. user explicitly resets to unicodecodepoints.
2. user persists log file, and unicodecodepoints is elided (because it is the default)
3. user reopens log file, ctor inits to utf16codeunits and the unicodecodepoints setting is lost.

Instead, we basically overwrite everything such that run.columnKind is always persisted (by our SDK). If we happen to open a file without this value set, our SDK will treat it as Utf16CodeUnits.

As a result, Michael had to discard the autogenerated Run.cs, and use a hand-coded version where he ripped out the default values that the code generator produced.

We need a code gen hint to tell the code generator not to emit a default on a specified property.

Merge develop to master and remove the develop branch

JSchema still uses the develop/master structure, which we have abandoned in favor of always working off master. Once we’ve merged PR #103, we should merge develop to master and then remove the develop branch.

@michaelcfanning @harleenkohli FYI

Property with uri-format should be Uri, not string

When the code generator generates a .NET property from a schema property whose type is string, the .NET property also has type string -- unless the schema property has format uri (in which case the .NET property has type System.Uri) or date-time (in which case the .NET property has type System.DateTime).

But the code generator didn't take account of the uri-reference format, and it generated a string property when it should have generated a System.Uri.

Provide a mechanism to apply arbitrary attributes to emitted C# files

It would be helpful, when converting to a specific deserialization technology such as JSON.NET to be able to apply things like the JsonCoverterAttribute:

[JsonConverter(typeof(SarifVersionConverter)]

Change "JS" rule prefix to "JSON"

Code emit overwrites any property named 'properties' with a property bag

SARIF contains a property named 'properties' that is not a property bag. JSchema is disregarding the type reference for this, it appears and emitting it as a property bag instead.

Nuget package id is arguably wrong for Validation.CLI

https://github.com/Microsoft/jschema/blob/master/src/Json.Schema.ToDotNet.Cli/Json.Schema.ToDotNet.Cli.nuspec#L4

    <id>Microsoft.Json.Schema.Validation</id>
    <title>Microsoft JSON Schema Validation</title>
<description>A command line tool that validates files against a JSON schema.</description>`

Current value is: Microsoft.Json.Schema.Validation
Value should be: Microsoft.Json.Schema.Validation.CLI

Right now, this is not causing anyone an issue internally because you're only really releasing the CLI program with the library compiled in under the same project. However, when the time comes to extract the library out into it's own Nuget package and depend on it, you're going to have a problem because that is the package that should be named Microsoft.Json.Schema.Validation. I'm raising this now so you can fix it before any more users start depending on the CLI under this package id.

Suggestion: declare and reference const default values where appropriate

e..g, "en-US" is a default locale in our jschema use. Jschema embeds this literal in ctor and attribute references. our own code needs to know about this literal and declares its own uses.

better:jschema creates a public const literal for all default values and itself utilizes them in code emit. OM consumers can also leverage these constants.

Two JSON Schemas from Microsoft that don't work

I thought it might be helpful for you to have more examples of real-world schemas that expose corner cases for unit tests. I don't know if these expose issues with the generator or issues with the schema. In any case, here they are.

Unfortunately, from the CLI outputs, I was not able to deduce what the issues were.

tasks.vs.json

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "definitions": {
    "default": {
      "type": "object",
      "properties": {
        "taskName": {
          "type": "string"
        },
        "taskLabel": {
          "type": "string"
        },
        "appliesTo": {
          "type": "string"
        },
        "contextType": {
          "type": "string",
          "enum": [
            "custom",
            "build",
            "clean",
            "rebuild"
          ]
        },
        "output": {
          "type": "string"
        },
        "inheritEnvironments": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      }
    },
    "defaultTask": {
      "allOf": [
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "enum": [
                "default"
              ]
            }
          }
        },
        {
          "$ref": "#/definitions/default"
        }
      ]
    },
    "remote": {
      "type": "object",
      "properties": {
        "type": {
          "type": "string",
          "enum": [
            "remote"
          ]
        },
        "remoteMachineName": {
          "type": "string"
        },
        "command": {
          "type": "string"
        },
        "remoteWorkingDirectory": {
          "type": "string"
        },
        "localCopyDirectory": {
          "type": "string"
        },
        "remoteCopyDirectory": {
          "type": "string"
        },
        "remoteCopyMethod": {
          "type": "string",
          "enum": [
            "none",
            "sftp",
            "rsync"
          ]
        },
        "remoteCopySourcesOutputVerbosity": {
          "type": "string",
          "enum": [
            "Normal",
            "Verbose",
            "Diagnostic"
          ]
        },
        "rsyncCommandArgs": {
          "type": "string",
          "default": "-t --delete"
        },
        "remoteCopyExclusionList": {
          "type": "array"
        }
      }
    },
    "remoteTask": {
      "allOf": [
        {
          "$ref": "#/definitions/default"
        },
        {
          "$ref": "#/definitions/remote"
        }
      ]
    },
    "launch": {
      "type": "object",
      "properties": {
        "type": {
          "type": "string",
          "enum": [
            "launch"
          ]
        },
        "command": {
          "type": "string"
        },
        "args": {
          "type": "array"
        },
        "launchOption": {
          "type": "string",
          "enum": [
            "None",
            "ContinueOnError",
            "IgnoreError"
          ]
        },
        "workingDirectory": {
          "type": "string"
        },
        "customLaunchCommand": {
          "type": "string"
        },
        "customLaunchCommandArgs": {
          "type": "string"
        },
        "envVars": {
          "type": "object"
        },
        "commands": {
          "type": "array",
          "items": [
            {
              "type": "object",
              "properties": {
                "command": {
                  "type": "string"
                },
                "args": {
                  "type": "array"
                },
                "launchOption": {
                  "type": "string",
                  "enum": [
                    "None",
                    "ContinueOnError",
                    "IgnoreError"
                  ]
                },
                "workingDirectory": {
                  "type": "string"
                },
                "customLaunchCommand": {
                  "type": "string"
                },
                "customLaunchCommandArgs": {
                  "type": "string"
                },
                "envVars": {
                  "type": "object"
                }
              }
            }
          ]
        }
      }
    },
    "launchTask": {
      "allOf": [
        {
          "$ref": "#/definitions/default"
        },
        {
          "$ref": "#/definitions/launch"
        }
      ]
    },
    "msbuild": {
      "type": "object",
      "properties": {
        "type": {
          "type": "string",
          "enum": [
            "msbuild"
          ]
        },
        "verbosity": {
          "type": "string",
          "enum": [
            "Quiet",
            "Minimal",
            "Normal",
            "Detailed",
            "Diagnostic"
          ]
        },
        "toolsVersion": {
          "type": "string"
        },
        "globalProperties": {
          "type": "object"
        },
        "properties": {
          "type": "object"
        },
        "targets": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      }
    },
    "msbuildTask": {
      "allOf": [
        {
          "$ref": "#/definitions/default"
        },
        {
          "$ref": "#/definitions/msbuild"
        }
      ]
    }
  },
  "type": "object",
  "properties": {
    "version": {
      "type": "string"
    },
    "variables": {
      "type": "object"
    },
    "tasks": {
      "type": "array",
      "items": {
        "oneOf": [
          {
            "$ref": "#/definitions/defaultTask"
          },
          {
            "$ref": "#/definitions/remoteTask"
          },
          {
            "$ref": "#/definitions/launchTask"
          },
          {
            "$ref": "#/definitions/msbuildTask"
          }
        ]
      }
    }
  }
}

launch.vs.json

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "definitions": {
    "default": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string"
        },
        "project": {
          "type": "string"
        },
        "projectTarget": {
          "type": "string"
        },
        "debugType": {
          "type": "string",
          "enum": [
            "native",
            "managed",
            "mixed"
          ]
        },
        "inheritEnvironments": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "args": {
          "type": "array"
        },
        "currentDir": {
          "type": "string"
        },
        "noDebug": {
          "type": "boolean"
        },
        "stopOnEntry": {
          "type": "boolean"
        },
        "remoteMachine": {
          "type": "string"
        },
        "buildConfigurations": {
          "type": "array",
          "items": {
            "oneOf": [
              {
                "$ref": "#/definitions/default"
              }
            ]
          }
        }
      }
    },
    "defaultFile": {
      "allOf": [
        {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "enum": [
                "default"
              ]
            }
          }
        },
        {
          "$ref": "#/definitions/default"
        }
      ]
    },
    "dll": {
      "type": "object",
      "properties": {
        "type": {
          "type": "string",
          "enum": [
            "dll"
          ]
        },
        "exe": {
          "type": "string"
        }
      }
    },
    "dllFile": {
      "allOf": [
        {
          "$ref": "#/definitions/default"
        },
        {
          "$ref": "#/definitions/dll"
        }
      ]
    },
    "pyproj": {
      "type": "object",
      "properties": {
        "type": {
          "type": "string",
          "enum": [
            "pyproj"
          ]
        }
      }
    },
    "pyprojFile": {
      "allOf": [
        {
          "$ref": "#/definitions/default"
        },
        {
          "$ref": "#/definitions/pyproj"
        }
      ]
    },
    "python": {
      "type": "object",
      "properties": {
        "type": {
          "type": "string",
          "enum": [
            "python"
          ]
        },
        "interpreter": {
          "type": "string"
        },
        "interpreterArguments": {
          "type": "string"
        },
        "scriptArguments": {
          "type": "string"
        },
        "workingDirectory": {
          "type": "string"
        },
        "nativeDebug": {
          "type": "boolean"
        },
        "webBrowserUrl": {
          "type": "string"
        }
      }
    },
    "pythonFile": {
      "allOf": [
        {
          "$ref": "#/definitions/default"
        },
        {
          "$ref": "#/definitions/python"
        }
      ]
    },
    "cpp_schema": {
      "type": "object",
      "properties": {
        "program": {
          "type": "string",
          "description": "Full path to program executable."
        },
        "processId": {
          "type": "integer",
          "description": "Optional process id to attach the debugger to."
        },
        "sourceFileMap": {
          "type": "object",
          "description": "Optional source file mappings passed to the debug engine. Format: '{ \"<Compiler source location>\": \"<Editor source location>\" }' OR '{ \"<Compiler source location>\": { \"editorPath\": \"<Editor source location>\", \"useForBreakpoints\": true } }'. \nExample: '{ \"/home/user/foo\": \"C:\\foo\" }' OR '{ \"/home/user/foo\": { \"editorPath\": \"c:\\foo\", \"useForBreakpoints\": true } }'.",
          "additionalProperties": {
            "anyOf": {
              "type": "string",
              "$ref": "#/definitions/cpp_schema/definitions/sourceFileMapOptions"
            }
          }
        },
        "MIMode": {
          "type": "string",
          "description": "Indicates the type of MI-enabled console debugger that the MIDebugEngine will connect to. \nAllowed values are \"gdb\", \"lldb\"."
        },
        "args": {
          "type": "array",
          "description": "Command line arguments passed to the program.",
          "items": {
            "type": "string"
          }
        },
        "environment": {
          "type": "array",
          "description": "Environment variables to add to the environment for the program. \nExample: [ { \"name\": \"squid\", \"value\": \"clam\" } ].",
          "items": {
            "type": "object",
            "properties": {
              "name": {
                "type": "string"
              },
              "value": {
                "type": "string"
              }
            }
          }
        },
        "targetArchitecture": {
          "type": "string",
          "description": "The architecture of the debuggee. This will automatically be detected unless this parameter is set. \nAllowed values are x86, arm, arm64, mips, x64, amd64, x86_64."
        },
        "cwd": {
          "type": "string",
          "description": "The working directory of the target."
        },
        "miDebuggerPath": {
          "type": "string",
          "description": "The path to the MI-enabled debugger (such as gdb). When unspecified, it will search PATH first for the debugger."
        },
        "miDebuggerServerAddress": {
          "type": "string",
          "description": "Network address of the MI-enabled debugger server to connect to. \nExample: localhost:1234."
        },
        "setupCommands": {
          "type": "array",
          "description": "One or more GDB/LLDB commands to execute in order to setup the underlying debugger. \nExample: \"setupCommands\": [ { \"text\": \"-enable-pretty-printing\", \"description\": \"Enable GDB pretty printing\", \"ignoreFailures\": true }].",
          "items": {
            "$ref": "#/definitions/cpp_schema/definitions/launchSetupCommands"
          }
        },
        "customLaunchSetupCommands": {
          "type": "array",
          "description": "If provided, this replaces the default commands used to launch a target with some other commands. \nFor example, this can be \"-target-attach\" in order to attach to a target process. \nAn empty command list replaces the launch commands with nothing, which can be useful if the debugger is being provided launch options as command line options. \nExample: \"customLaunchSetupCommands\": [ { \"text\": \"target-run\", \"description\": \"run target\", \"ignoreFailures\": false }].",
          "items": {
            "$ref": "#/definitions/cpp_schema/definitions/launchSetupCommands"
          }
        },
        "launchCompleteCommand": {
          "type": "string",
          "enum": [
            "exec-run",
            "exec-continue",
            "None"
          ],
          "description": "The command to execute after the debugger is fully setup in order to cause the target process to run. \nAllowed values are \"exec-run\", \"exec-continue\", \"None\". \nThe default value is \"exec-run\"."
        },
        "debugServerPath": {
          "type": "string",
          "description": "Optional full path to debug server to launch. \nDefaults to null."
        },
        "debugServerArgs": {
          "type": "string",
          "description": "Optional debug server args. \nDefaults to null."
        },
        "serverStarted": {
          "type": "string",
          "description": "Optional server-started pattern to look for in the debug server output. \nDefaults to null."
        },
        "filterStdout": {
          "type": "boolean",
          "description": "Search stdout stream for server-started pattern and log stdout to debug output. \nDefaults to true."
        },
        "filterStderr": {
          "type": "boolean",
          "description": "Search stderr stream for server-started pattern and log stderr to debug output. \nDefaults to false."
        },
        "serverLaunchTimeout": {
          "type": "integer",
          "description": "Optional time, in milliseconds, for the debugger to wait for the debugServer to start up. Default is 10000."
        },
        "coreDumpPath": {
          "type": "string",
          "description": "Optional full path to a core dump file for the specified program. \nDefaults to null."
        },
        "externalConsole": {
          "type": "boolean",
          "description": "If true, a console is launched for the debuggee. If false, no console is launched. \nDefaults to false. NOTE: This option is ignored in some cases for technical reasons."
        },
        "pipeTransport": {
          "$ref": "#/definitions/cpp_schema/definitions/pipeTransportOptions"
        }
      },
      "definitions": {
        "launchSetupCommands": {
          "type": "object",
          "properties": {
            "text": {
              "type": "string",
              "description": "The debugger command to execute."
            },
            "description": {
              "type": "string",
              "description": "Optional description for the command."
            },
            "ignoreFailures": {
              "type": "boolean",
              "description": "If true, failures from the command should be ignored. \nDefaults to false."
            }
          }
        },
        "pipeTransportOptions": {
          "type": "object",
          "description": "When present, this tells the debugger to connect to a remote computer using another executable as a pipe that will relay standard input/output between Visual Studio \nand the MI-enabled debugger (such as gdb).",
          "properties": {
            "pipeCwd": {
              "type": "string",
              "description": "The fully qualified path to the working directory for the pipe program."
            },
            "pipeProgram": {
              "type": "string",
              "description": "The fully qualified pipe command to execute."
            },
            "pipeArgs": {
              "type": "array",
              "description": "Command line arguments passed to the pipe program to configure the connection.",
              "items": {
                "type": "string"
              }
            },
            "debuggerPath": {
              "type": "string",
              "description": "The full path to the debugger on the target machine, for example /usr/bin/gdb."
            },
            "pipeEnv": {
              "type": "object",
              "additionalProperties": {
                "type": "string"
              },
              "description": "Environment variables passed to the pipe program."
            },
            "quoteArgs": {
              "type": "boolean",
              "description": "If individual arguments contain characters (such as spaces or tabs), should it be quoted? If 'false', the debugger command will no longer be automatically quoted. \nDefault is 'true'."
            }
          }
        },
        "sourceFileMapOptions": {
          "type": "object",
          "properties": {
            "editorPath": {
              "type": "string",
              "description": "The location of the source code for the editor to locate."
            },
            "useForBreakpoints": {
              "type": "boolean",
              "description": "When setting breakpoints, this source mapping should be used. \nIf 'false', only the filename and line number is used for setting breakpoints. \nIf 'true', breakpoints will be set with the full path to the file and line number only when this source mapping is used. Otherwise just filename and line number will be used when setting breakpoints. \nDefault is 'true'."
            }
          }
        }
      }
    },
    "cppTemplateLayout": {
      "allOf": [
        {
          "$ref": "#/definitions/default"
        },
        {
          "$ref": "#/definitions/cpp_schema"
        }
      ]
    },
    "nativeLayout": {
      "allOf": [
        {
          "$ref": "#/definitions/default"
        }
      ]
    },
    "mixedLayout": {
      "allOf": [
        {
          "$ref": "#/definitions/default"
        }
      ]
    },
    "managedLayout": {
      "allOf": [
        {
          "$ref": "#/definitions/default"
        }
      ]
    }
  },
  "type": "object",
  "properties": {
    "version": {
      "type": "string"
    },
    "defaults": {
      "properties": {
        ".dll": {
          "$ref": "#/definitions/dll"
        },
        ".pyproj": {
          "$ref": "#/definitions/pyproj"
        },
        ".py": {
          "$ref": "#/definitions/python"
        },
        ".pyw": {
          "$ref": "#/definitions/python"
        }
      }
    },
    "configurations": {
      "type": "array",
      "items": {
        "oneOf": [
          {
            "$ref": "#/definitions/defaultFile"
          },
          {
            "$ref": "#/definitions/dllFile"
          },
          {
            "$ref": "#/definitions/pyprojFile"
          },
          {
            "$ref": "#/definitions/pythonFile"
          },
          {
            "$ref": "#/definitions/cppTemplateLayout"
          },
          {
            "$ref": "#/definitions/nativeLayout"
          },
          {
            "$ref": "#/definitions/mixedLayout"
          },
          {
            "$ref": "#/definitions/managedLayout"
          }
        ]
      }
    }
  }
}

Downgrade to .NET v4.5

To allow it to be incorporated into more projects. There's nothing here that requires 4.5.1.

Design-Time codegen using ToDotNet library

This fairly recent article describes the use of codegen in a build process:

https://docs.microsoft.com/en-us/visualstudio/modeling/code-generation-in-a-build-process?view=vs-2017

It is talking specifically about T4 templates which are separate from this project. However, from a Visual Studio/MSBuild perspective, this would seem the appropriate mechanism to generate C# classes just prior to the compilation of regular project sources, so that the generated sources can be compiled in. With that said, I don't see anywhere in the documentation discussing support for the use of JSchema ToDotNet functionality with MSBuild as part of Design-Time codegen.

Is this supported or planned?

Equality comparer doesn't compare base types

If a generated class has a BaseTypeHint, the generate equality comparer compares only the derived type's properties. It doesn't delegate to the base type's equality comparer to compare the base type's properties.

The nature of the base class comparison must depend on the ComparisonKind of the base class. Unfortunately, as the code stands, ComparisonKind is a property of the PropertyInfo class; that is, the code generator understands how to compare property values, but only because it understands the schema that describes the property. There is nothing that represents the ComparisonKind of a class. I'll need to add that feature.

Add Stack<ISarifNode> for in-progress visit

SARIF SDK mechanism uses this in order to do things like exclude originalUriBaseId file location visit (in cases where only files, not directories, need processing).

Document code generator

When a property has a default value, emit System.ComponentModel.DefaultValueAttribute

Do this automatically, without requiring an AttributeHint​​​​​​​ in CodeGenHints.json.

Suggest using fully qualified namespace for System.ComponentModel.DefaultValue

As it happens, System.ComponentModel pulls in names that collide with terms such as 'EnumConverter' that are likely to be utilized by projects that consume JSON.NET.

An easy fix is to use the fully qualified name in the code emit. This will prevent others from having to use fully qualified names in things like attribute hint arguments.

Document JSchema

JSchema doesn't initialize default values in ctor.

JSchema doesn't emit default values. in the SARIF SDK we have to maintain these manually, currently

        /// </summary>
        [DataMember(Name = "columnKind", IsRequired = false, EmitDefaultValue = false)]
        [JsonConverter(typeof(EnumConverter))]
        [System.ComponentModel.DefaultValue(ColumnKind.UnicodeCodePoints)]
        public ColumnKind ColumnKind { get; set; }

Document validator

Validator does not report error locations in JSON pointer format

In its error messages, the validator reports error locations like this: runs[0].rules.TST0001.helpUri instead of using the JSON pointer syntax /runs/0/rules/TST0001/helpUri.

Some package attributes wrong or missing in packages built from project files

The JsonPointer and JsonSchema packages, which are built automatically from their respective project files, don't get the right values for Authors and Owners, and they don't have a Title at all.

Emit a JsonProperty attribute on all non-required properties

Regardless of the presence of the System.ComponentModel.DefaultValue attribute (which we now emit for any property for which the schema specifies a default value; see #58), that default is ignored on deserialization unless the property is generated with JsonProperty attribute that specifies "populate default on deserialize".

We need to emit that attribute.

An enhancement if you're ok with the workaround of adding a CodeGenHint; otherwise a bug.

Please update Jschema to latest SARIF-SDK

Code gen fails for dictionaries in nested properties

@michaelcfanning @boAndron

@boAndron reported this in microsoft/sarif-sdk#1046. I will leave that issue open, because it won't be resolved until we fix the underlying problem in JSchema, publish a new JSchema NuGet package, consume the new package from sarif-sdk, and publish a new Sarif.Sdk NuGet package.

The problem is as follows: The code generator correctly deals with this case:

run.files: string => file

That is (in my bespoke notation), it correctly deals with a run object that defines a files property whose value is a dictionary from string to file object. By "correctly", I mean that it generates a call to FileEqualityComparer.Equals to compare the objects in the run.files dictionary.

But the code generator fails on this case:

run.resources.rules: string => rule

That is, it fails for a run object that defines a resources property that defines a rules property whose value is a dictionary from string to rule. By "fails", I mean that it generates a call to object.Equals to compare the objects in the run.resources.rules dictionary.

And the difference is that the rules dictionary is nested one level deeper than the files dictionary. I have a vague memory of worrying, years ago (literally) that I didn't handle that case. And now it's bitten me.

Wrong rule description for JS0001

"The schema is not a valid JSON document"

This isn't the right description for a syntax error that occurs in an instance document.

Pass dictionary keys to dictionary element visitors

How do we allow for special ctor initializations in the emitted code?

Relates to #92, where we need to alter default ctor initializations.

In the SARIF SDK, see SarifLog, which needs to set default Version and SchemaUri details.

Code generation fails a property of a non-primitive type specifies a default

If your schema specifies a property of type (for example) "array of integer", and it specifies the default value as (for example) [], the code generation will fail, because it will not be able to create a compile-time constant literal for the empty array. If the default value isn't a compile-time constant, you can't set it as an argument to the DefaultValue attribute:

[DefaultValue(42)]       // ok

[DefaultValue(new int[]) // not so much

Fix this by only emitting the DefaultValue attribute for properties of primitive type.

Don't emit default-related attributes if the JSON schema default is the same as the .NET default

RewritingVisitor does not visit some dictionaries

For example, it visits neither the MessageStrings nor the Rules dictionary-valued property of Resources:

        public virtual Resources VisitResources(Resources node)
        {
            if (node != null)
            {
            }

            return node;
        }

and it does not visit Run.Graphs or Result.Graphs.

It does visit Run.Files, so that will be a good clue.

@michaelcfanning FYI

Unencoded URIs validate

Look at the attached SARIF file and search for uses such as the following. This file validates, why is that?

invaliduris.sarif.txt
(url) "workItemUris": [ "c:\path with a space\file.txt" ],

Handle a special case involving oneOf

Description

There are certain properties in the SARIF specification whose values, according to the spec, can either be an array or null. For these properties, a null value has different semantics than an empty array: an empty array means "I know there weren't any of these things", and null means "I don't know whether or not there were any of these things."

For these properties, the SARIF schema needs to allow the property's type to be either array or null, because array and null are different JSON types. For example:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "properties": {
    "myProp": {
      "description": "Hello",
      "oneOf": [
        {
          "type": "array",
        },
        {
          "type": "null"
        }
      ],
      "minItems": 1,
      "items": {
        "type": "integer"
      }
    }
  }
}

But the code generator doesn't know how to handle oneOf. Since it's generating a property of a fixed .NET type, it doesn't think it needs to deal with multiple types.

But this is a special case. "null" isn't a valid JSON array, but it's a fine value for a .NET IList<int>.

So we special-case this. If the code generator encounters a oneOf with exactly two branches, and one of them is "type": null, it ignores that branch, and pretends that the type was specified by the other branch.

Notes

NOTE 1: In discussions, we've referred to this as the "any of" feature. "oneOf" is actually more appropriate. It means that the instance document must conform to exactly one of the alternate schemas. "anyOf" means that instance document must conform to one or more of the alternate schemas.

Of course, if one of the schemas says "type is null" and the other says "type is array", then it's impossible for an instance document to conform to both, so "anyOf" will do just as well. Still the intent of "oneOf" is a better match for the feature we're supporting: a property can be either an array or null.

NOTE 2: The code generator will generate exactly the same code with the "oneOf" shown in the example as it does for a simple "type": "array". The reason for allowing the "oneOf" is to allow an instance document that uses null to validate. For example:

{
  "myProp": null
}

This document validates against the schema in the example, but not if you change the schema to specify a simple "type": "array".

Limitations

This is a very limited hack. In particular:

• You might expect that if the two branches had types integer and null, that the code generator would generated Nullable<int>, but it doesn't.
• The code generator completely ignores the contents of the "oneOf" element except for the "type".
• We actually only support the case where one of the types is array and the other is null. We could broaden this if needed.

We could do much more work in this area if we needed to.

PackageLicenseExpression not working for Android bidning projects

Tried to use the new attributes on an Android binding project but it does not output the node in the nuspec. It does work however for a multi-target NetStandard2.0 project.

Codegen emits wrong values for Flags enums

Should be 1, 2, 4, 8, ..., but is 1, 2, 3, 4,...

Attribute hint doesn't work on a wildcard

See @michaelcfanning's comment in microsoft/sarif-sdk#1103.

Document JPointer

DefaultValueAttribute needs to provide type for numeric defaults

Repro: declare a default value of 0.0 (a double).

Emitted code specifiers a default value of 0 (int32). As a result, things like ignore and populate aren't honored (presumably due to some distinction between JSON number and integer types).

Need to mark specific class methods as virtual

SARIF SDK no longer utilizes interfaces. Instead, we want to mark specific class' methods as virtual.

BaseTypeHint isn't honored if a class is renamed via ClassNameHint

  "exception": [
    {
      "kind": "ClassNameHint",
      "arguments": {
        "className": "ExceptionData"
      }
    },
    {
      "kind": "BaseTypeHint",
      "arguments": {
        "baseTypeNames": [
          "PropertyBagHolder"
        ]
      }
    }
  ],

Consider using xunit.MSBuild

A set of MSBuild tasks that run xunit tests.

anyOf support for required properties would be very helpful

Default values for enumerated types don't work properly

Defaults for enumerated types don't work -- you get a string value. For example, on Run.ColumnKind, it generates the attribute

[DefaultValue("unicodeCodePoints")]

(instead of ColumnKind.UnicodeCodePoints) on the ColumnKind property, and it generates the initializer

ColumnKind = "unicodeCodePoints"

in the default ctor. Fortunately the latter doesn't compile, or we wouldn't have noticed the problem (because the DefaultValue attribute, taking as it does an object, does compile).

Downgrade JSON.NET package to v9

Generate an "analyzing visitor"

In the SARIF SDK, the SARIF validation utility (Sarif.Multitool -validate) visits each node in the SARIF object model, running each validation rule over each node. At the moment, the visiting code is hand-written (see SarifValidationSkimmerBase.cs). The code generator should generate the analyzing visitor.

It might turn out that the RewritingVisitor that the code generator already generates can do the job, but I think I already looked at this question once and decided that it couldn't.

PackageLicenseExpression not respected in packages built from custom nuspec files

@michaelcfanning This is the bug you reported to me. See also #68, which I noticed while working on this bug, and which I addressed as far as I think I can.

Build warning when creating NuGet packages: duplicate file VersionConstants.cs

Run BuildAndTest.cmd. During the "pack" phase, we get this message, once per package:

C:\Program Files\dotnet\sdk\2.1.2\Sdks\NuGet.Build.Tasks.Pack\buildCrossTargeting\
NuGet.Build.Tasks.Pack.targets(204,5): warning : File 'C:\Code\jschema\
bld\obj\Json.Schema.Validation\AnyCPU_Release\netstandard2.0\VersionConstants.cs'
is not added because the package already contains file 'src\Json.Schema.Validation\VersionConstants.cs'
[C:\Code\jschema\src\Json.Schema.Validation\Json.Schema.Validation.csproj]

Code gen "suffix" feature ignores AttributeHint arguments

The code gen feature accepts an option that adds a suffix to all type names, and qualifies all namespace names. For example, if you specify --suffix VersionOne, then the type name Location becomes LocationVersionOne and the namespace Microsoft.CodeAnalysis.Sarif becomes Microsoft.CodeAnalysis.Sarif.VersionOne.

The problem occurs if you specify an AttributeHint that mentions type names or namespace names to which you want to apply the suffix, for example:

"AnnotatedCodeLocation.Id": [
  {
    "kind": "AttributeHint",
    "arguments": {
      "typeName": "JsonConverter",
      "namespaceName": "Newtonsoft.Json",
      "arguments": [
        "typeof(Microsoft.CodeAnalysis.Sarif.Readers.AnnotatedCodeLocationIdConverter)"
      ]
    }
  },
  ...
]

This produces the line:

[JsonConverter(typeof(Microsoft.CodeAnalysis.Sarif.Readers.AnnotatedCodeLocationIdConverter))]

but we actually want want this:

[JsonConverter(typeof(Microsoft.CodeAnalysis.Sarif.Readers.VersionOne.AnnotatedCodeLocationIdConverterVersionOne))]

One possibility is to recognize escape sequences within arguments.arguments, for example:

"AnnotatedCodeLocation.Id": [
  {
    "kind": "AttributeHint",
    "arguments": {
      "typeName": "JsonConverter",
      "namespaceName": "Newtonsoft.Json",
      "arguments": [
        "typeof(Microsoft.CodeAnalysis.Sarif.Readers{ns}.AnnotatedCodeLocationIdConverter{ts})"
      ]
    }
  },
  ...
]

The code gen would recognize {ns} as "namespace suffix", and, if --suffix was specified, it would replace it with .TheSuffix (note the leading dot). Similarly, it would recognize {ts} as "type suffix", and replace it with TheSuffix (note the lack of leading dot).

Strictly speaking this is a breaking change since the string {ns} and {ts} could legitimately appear in an argument, but heck JSchema is still in Version 0.

Unique arrays s/be emitted as hash sets with value comparers

Currently, we emit arrays that s/be unique by schema as Lists. These constructs don't guarantee the uniqueness of the array. We should prefer emitting HashSets for these, that pass the emitted value comparers in the ctor.

e..g, using a SARIF example

run.files = new HashSet(FileData.ValueComparer);