glideapps/quicktype

Generate types and converters from JSON, Schema, and GraphQL

quicktype's Issues

Make Elm output nicer

• Remove comments
• Omit unnecessary parentheses
• Use toplevel name for Root
• Add explanatory comments
• All type definitions before encoders and decoders
• Move toplevel definition up
• Expose all types
• No more underscores in names
• Array and Dict instead of Array.Array and Dict.Dict

Sort classes

I think the classes should be sorted by a breadth-first traversal of the graph, starting from the top-level type. At the same depth we could sort alphanumerically, for example.

Simple feedback feature

Enforce required fields in Go

This code via Gustavo Poscidonio from Slack. We could use this to do arbitrary validation.

package main

import (
	"encoding/json"
	"fmt"
)

type Pokemon struct {
	Name     string   // required
	Type     []string // required with length >= 1
	Nickname string   // not required
}

func (p *Pokemon) UnmarshalJSON(data []byte) error {
	var aux struct {
		Name     string
		Type     []string
		Nickname string
	}

	err := json.Unmarshal(data, &aux)
	if err != nil {
		return err
	}

	if aux.Name == "" {
		return fmt.Errorf("Name field not set for Pokemon")
	}

	if len(aux.Type) < 1 {
		return fmt.Errorf("Type field not set for Pokemon")
	}
	
	p.Name = aux.Name
	p.Type = aux.Type
	p.Nickname = aux.Nickname

	return nil
}

var validPokemon = []byte(`
{
	"name" : "Bulbasaur",
	"type" : ["grass", "poison"],
	"nickname" : "little foot"
}
`)

var invalidPokemon = []byte(`
{
	"name" : "Bulbasaur",
	"nickname" : "little foot"
}
`)

func main() {
	var pokemon Pokemon
	fmt.Println(json.Unmarshal(invalidPokemon, &pokemon))
	fmt.Println(json.Unmarshal(validPokemon, &pokemon))
	fmt.Printf("My pokemon: %+v\n", pokemon)
}

Subclasses in the IR?

Extending the similarity metric to include subclasses as similar led to some mayhem, but if we had proper subclasses in the IR that could be avoided. For example, in github-events.json, there's a small repo and a large repo class, the large one containing a strict superset of the small one. The large one could be a proper subclass of the smaller one.

Organize large golang literal

• Ideally in a separate file (?)
• Preserving indentation

No manual build dependencies

npm test doesn't rebuild the CLI.

Support Swift

struct TopLevel {
  let `class`: String
}

Launch blog post

Once we add options, we should blog about them and also mention:

• Name combination
• Class de-duplication
• Map detection
• Optional types rather than default values (e.g. null : int? not 0 : int)
• Numeric types

More sound similar class detector

Right now we pick one class and then find all classes that are similar to it. But depending on which class you start out with, you can end up with different sets. So we might want to find sets where all classes are pairwise similar. The complexity of that is most likely larger, definitely when implemented naively.

Don't generate invalid type and property names

Like for us-average-temperatures.json:

		[JsonProperty("192612")]
		public 189512 192612 { get; set; }

Escape C# keywords

public bool @private { get; set; }

Allow pasting more than one document

Just read them all and unify them.

Handle toplevel properly

• TopLevel class should come first
• don't test for class name TopLevel
• Handle non-class JSON (we don't even generate a TopLevel class for that now)

Support enums

For string properties, can we form a relatively small set of equivalence classes from a large number of samples? We might represent that as an enum (if the target language has enums).

There are two parts to this:

• Supporting enums in our IR, accepting them from JSON Schema, and producing code for them
• Automatically detecting them from given JSON samples

We can and should do the former, whether we ever do the latter or not.

Better map heuristics

Heuristics:

• Same property types
• Large number of properties
• Property names that don't look like names
• Detect specific patterns in names, like hashes or numbers

Fix excluded tests on all backends

Newlines in property names, empty property names, and unicode stuff, doesn't seem to work in Go. See identifiers.json.

Figure out why large files are slow

The 270MB Boris validation corpus, for example.

Use incremental parsing

We can't keep the whole JSON document in memory. I get heap overflows with the default V8 settings for JSON files as small as 250ish MB.

Preserve order of properties

C# member names cannot be the same as their enclosing type

This doesn't compile:

class A
{
    public object A { get; set; }
}

Handle toplevel better

fromJson should ideally be in the top-level class. Figure out how to handle these three cases:

1. There is one class that can be interpreted as top level.
2. There is no class at all.
3. There is more than one class that can be interpreted as top level.

Exposing the Converter class is not ideal because it inherits from JsonConverter, which serves a different purpose than what we want to expose.

Allow non-inferred class names

Right now all class names are inferred from property names via regatherClassNames. That's fine for JSON input, where classes are not named. In JSON Schema it is possible to name classes, so wherever we have a class name we should preserve it, and not try re-infer it. This should be easy to do by making the names property of IRClassData an Either String (Set String) instead of just a Set String.

Build a test harness

Run through samples, generate code, compile, run.

Implement serialization in C#

Implement `IRUnion` in C# somehow

We'll probably end up generating individual classes for each IRUnion that need custom converters.

HTTPS

Fix dropdown overflow bug

Add more tests

In particular, all the smaller ones from the Awesome JSON dataset.

Show "Go" in the dropdown when going to go.quicktype.io

Right now it shows C#.

Command line client

If only for the use case of running with large JSON files.

Generate go to parse latest Bitcoin block

$ quicktype -o latest-block.go \
  "https://blockchain.info/rawblock/0000000000000bae09a7a393a8acded75aa67e46cb81f7acaa5ad94f9eacd103"

Test that Json->X and Json->Schema->X are equivalent

Probably depends on #27

Appear in organic search

Make Go output fmt-compliant

• JSON attributes are not horizontally aligned:

 type TopLevel struct {
-	Y []*float64 `json:"y"` // array of optional
-	X []DoubleOrNullOrString `json:"x"`
+	Y []*float64               `json:"y"`
+	X []DoubleOrNullOrString   `json:"x"`
 	Z []DoubleOrDoubleArrayOrZ `json:"z"`
 }

And weirdly it deletes comments. That must be a bug.

• Align comments (on consecutively commented lines)
• Use tab for indentation
• Don't pad last table column

Support dates

{
    "DepartureDateTime": "2016-08-13T18:15:00",
    "ArrivalDateTime": "2016-08-13T19:27:00"
}

Disable language selection and only do C#

Detect name parts like "url" and "id" for special capitalization

C# naming convention uses Url, but Go (and others) use URL.

File read from command line option must also use `stream-json`

Otherwise you get

LambdaBokPro430:quicktype schani$ time node cli/quicktype.js --src /tmp/jsons/corpus-validation-adhoc.json 
"toString()" failed

Users should be able to determine maps vs objects

dependencies and devDependencies are Map String String but are treated as custom objects.

We'll have to expose some way for people to determine these.

Option to parse missing arrays as empty rather than null

In this example, one person has no friends property, so this may result in a null array property in C#. We should consider an option to parse missing arrays as empty rather than null.

{
    "people": [
        {
            "name": "Nice Bob",
            "friends": ["David"]
        },
        {
            "name": "Evil Bob"
        }
    ]
}

public class Person
{
	[JsonProperty("friends")]
	public string[] Friends { get; set; }

	[JsonProperty("name")]
	public string Name { get; set; }

}

Support tuples

In the simplest case, an array of arrays, or dictionary of arrays, where each inner array has the same structure (same length, same element types), can be interpreted as an array of tuples, or dictionary of tuples. Eg.

[[1, true, "abc"],
 [2, false, "foo"],
 [3, true, "bar"]]

would be an Array (Tuple [Int, Bool, String])

Unique names for classes

There are two Repos in github-events.json, for example.

Nice marshal/unmarshal functions for Go

And a short comment on how to use it

Print TopLevel type first

Test Schema->Schema

The schema shouldn't change, obviously.

Fix IRAnything

IRAnything currently serves two purposes in the IR:

1. In inference, to signify a type we know nothing about. The only way for it to come about is as the element type of an empty array.

2. In generation, for any type, which we translate to object in C#.

We make no explicit distinction between the two in the code, i.e. they are one and the same.

That mostly works for now, but we will run into trouble if our inference algorithm ever wants to distinguish between the two. In addition to that, JSON Schema does have a representation for "any type", but our unification algorithm does not treat IRAnything as "any type", so we can't use and expect to get correct results. For example, if we treat "any type" as IRAnything, we would unify an array that contains one "any type" and one integer as an array containing integers. The correct unification is an array containing "any type".

If we had IRNoInformation and IRAny, unification would look like this:

unify IRNoInformation x = x
unify IRAny x = IRAny

Of course, to preserve more type information we could still do

unify IRAny x = IRUnion [IRAny, x]

Command line package

$ # Generate go to parse latest Bitcoin block
$ quicktype -o latest-block.go \
  "https://blockchain.info/rawblock/0000000000000bae09a7a393a8acded75aa67e46cb81f7acaa5ad94f9eacd103"

Check equivalence of input to output for tests

Customization

For the backends we can start with boolean options, for all languages we'd like the ability to at least give a top level class name.

We can later expand this to typed options. It seems that generics can be used for this.

Detect dates