scrapinghub/skinfer

Skinfer is a tool for inferring and merging JSON schemas

Skinfer - tool for working with JSON schemas

Simple tool to infer and/or merge JSON schemas

• Free software: BSD license
• Documentation: https://skinfer.readthedocs.org.

Features

• Generating schema in JSON Schema draft 4 format
• Inferring schema from multiple samples
• Merging schemas - nice for generating schema in Map-Reduce fashion or updating an old schema with new data

Example of using skinfer to generate a schema from a list of samples:

$ cat samples.jsonl
{"name": "Claudio", "age": 29}
{"name": "Roberto", "surname": "Gomez", "age": 72}
$ skinfer --jsonlines samples.jsonl
{
    "$schema": "http://json-schema.org/draft-04/schema",
    "required": [
        "age",
        "name"
    ],
    "type": "object",
    "properties": {
        "age": {
            "type": "number"
        },
        "surname": {
            "type": "string"
        },
        "name": {
            "type": "string"
        }
    }
}

Install with:

$ pip install skinfer

Or, if you don't have pip, you can still install it with:

$ easy_install skinfer

skinfer's Issues

Old json_schema_generator on PyPI doesn't work with Python 3

When I try to run under Python3 after installing with Pip, I get

Traceback (most recent call last):
  File "/bin/skinfer", line 9, in <module>
    from skinfer import schema_inferer
  File "/lib/python3.5/site-packages/skinfer/__init__.py", line 36, in <module>
    from .schema_inferer import generate_schema_for_sample as generate_schema  # NOQA
  File "/lib/python3.5/site-packages/skinfer/schema_inferer.py", line 5, in <module>
    from skinfer.draft4_generator import JsonSchemaGenerator
  File "/lib/python3.5/site-packages/skinfer/draft4_generator.py", line 5, in <module>
    import json_schema_generator
  File "/lib/python3.5/site-packages/json_schema_generator/__init__.py", line 1, in <module>
    from .generator import *
  File "/lib/python3.5/site-packages/json_schema_generator/generator.py", line 4, in <module>
    from .schema_types import Type, ObjectType, ArrayType
  File "/lib/python3.5/site-packages/json_schema_generator/schema_types.py", line 60, in <module>
    types.NoneType: NullType,
AttributeError: module 'types' has no attribute 'NoneType'

It seems that is because the json_schema_generator on PyPI doesn't have this fix for Python 3.

Support standard input

It would be nice to support reading data from standard input.

skinfer needs work for recursive JSON. See mentioned repository and data zip in the repository

https://github.com/coderextreme/skinfer.needs.work

json-schema-generator requires jsonschema==0.8.0

running develop
running egg_info
writing requirements to skinfer.egg-info/requires.txt
writing skinfer.egg-info/PKG-INFO
writing top-level names to skinfer.egg-info/top_level.txt
writing dependency_links to skinfer.egg-info/dependency_links.txt
reading manifest file 'skinfer.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '__pycache__' found under directory '*'
warning: no previously-included files matching '*.py[co]' found under directory '*'
writing manifest file 'skinfer.egg-info/SOURCES.txt'
running build_ext
Creating /home/.../lib/python2.7/site-packages/skinfer.egg-link (link to .)
skinfer 0.2.0 is already the active version in easy-install.pth
Installing skinfer script to /home/.../bin
Installing schema_merger script to /home/.../bin
Installing json_validator script to /home/.../bin

Installed /home/.../skinfer
Processing dependencies for skinfer==0.2.0
Searching for jsonschema>=0.8.0
Reading https://pypi.python.org/simple/jsonschema/
Best match: jsonschema 2.5.1
Downloading https://pypi.python.org/packages/source/j/jsonschema/jsonschema-2.5.1.zip#md5=e491298485bdefeecb0f1ca3d65e5292
Processing jsonschema-2.5.1.zip
Writing /tmp/easy_install-l21iNy/jsonschema-2.5.1/setup.cfg
Running jsonschema-2.5.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-l21iNy/jsonschema-2.5.1/egg-dist-tmp-7mk7u3
warning: build_py: byte-compiling is disabled, skipping.

warning: install_lib: byte-compiling is disabled, skipping.

zip_safe flag not set; analyzing archive contents...

Installed /tmp/easy_install-l21iNy/jsonschema-2.5.1/.eggs/vcversioner-2.14.0.0-py2.7.egg
warning: build_py: byte-compiling is disabled, skipping.

warning: install_lib: byte-compiling is disabled, skipping.

zip_safe flag not set; analyzing archive contents...
Moving jsonschema-2.5.1-py2.7.egg to /home/.../lib/python2.7/site-packages
Adding jsonschema 2.5.1 to easy-install.pth file
Installing jsonschema script to /home/.../bin

Installed /home/.../lib/python2.7/site-packages/jsonschema-2.5.1-py2.7.egg
error: jsonschema 2.5.1 is installed but jsonschema==0.8.0 is required by set(['json-schema-generator'])

pip install not placing skinfer script correctly

When installing skinfer with pip tool, skinfer script file is not available for users.

Steps to reproduce:

• create your virtual env
• pip install skinfer
• try to execute skinfer

What happens is:

user@host:~/$ skinfer
skinfer: command not found

When you manually clone the repository, and execute:

python setup.py install

It works as expected.

For objects inside a list, all seen fields are seen as required

If we take the following sample as input to schema_inferer:

[{"age": 22, "optional": {"phone": "2222-3333", "address": "some street"}, "name": "albert einstein"},
{"age": 44, "optional": {"phone": "2222-3333", "address": "some street"}, "name": "galileo"},
{"age": 488, "name": "stephen hawking"}]

A schema with the optional field as a required field is generated:

 {
    "items": {
        "required": [
            "age", 
            "optional", 
            "name"
        ],   
        "type": "object", 
        "properties": {
            "age": {
                "type": "number"
            },   
            "optional": {
                "required": [
                    "phone", 
                    "address"
                ],   
                "type": "object", 
                "properties": {
                    "phone": {
                        "type": "string"
                    },   
                    "address": {
                        "type": "string"
                    }    
                }    
            },   
            "name": {
                "type": "string"
            }    
        }    
    },   
    "$schema": "http://json-schema.org/draft-04/schema", 
    "type": "array"
}

Is this inference correct/expected?