Giter Site home page Giter Site logo

sheriff's Introduction

sheriff

GoDoc Build Status Coverage Status

go get github.com/liip/sheriff/v2

Package sheriff marshals structs conditionally based on tags on the fields.

A typical use is an API which marshals structs into JSON and maintains different API versions. Using sheriff, struct fields can be annotated with API version and group tags.

By invoking sheriff with specific options, those tags determine whether a field will be added to the output map or not. It can then be marshalled using "encoding/json".

NOTE: This package is tested only on Go 1.7+, it might work on Go 1.6 too, but no support is given.

Implemented tags

Groups

Groups can be used for limiting the output based on freely defined parameters. For example: restrict marshalling the email address of a user to the user itself by just adding the group personal if the user fetches his profile. Multiple groups can be separated by comma.

Example:

type GroupsExample struct {
    Username      string `json:"username" groups:"api"`
    Email         string `json:"email" groups:"personal"`
    SomethingElse string `json:"something_else" groups:"api,personal"`
}

Anonymous fields

Tags added to a struct’s anonymous field propagates to the inner-fields if no other tags are specified.

Example:

type UserInfo struct {
    UserPrivateInfo `groups:"private"`
    UserPublicInfo  `groups:"public"`
}
type UserPrivateInfo struct {
    Age string
}
type UserPublicInfo struct {
    ID    string
    Email string
}

Since

Since specifies the version since that field is available. It's inclusive and SemVer compatible using github.com/hashicorp/go-version. If you specify version 2 in a tag, this version will be output in case you specify version >=2.0.0 as the API version.

Example:

type SinceExample struct {
    Username string `json:"username" since:"2.1.0"`
    Email    string `json:"email" since:"2"`
}

Until

Until specifies the version until that field is available. It's the opposite of since, inclusive and SemVer compatible using github.com/hashicorp/go-version. If you specify version 2 in a tag, this version will be output in case you specify version <=2.0.0 as the API version.

Example:

type UntilExample struct {
    Username string `json:"username" until:"2.1.0"`
    Email    string `json:"email" until:"2"`
}

Example

package main

import (
	"encoding/json"
	"fmt"
	"log"

	"github.com/hashicorp/go-version"
	"github.com/liip/sheriff/v2"
)

type User struct {
	Username string   `json:"username" groups:"api"`
	Email    string   `json:"email" groups:"personal"`
	Name     string   `json:"name" groups:"api"`
	Roles    []string `json:"roles" groups:"api" since:"2"`
}

type UserList []User

func MarshalUsers(version *version.Version, groups []string, users UserList) ([]byte, error) {
	o := &sheriff.Options{
		Groups:     groups,
		ApiVersion: version,
	}

	data, err := sheriff.Marshal(o, users)
	if err != nil {
		return nil, err
	}

	return json.MarshalIndent(data, "", "  ")
}

func main() {
	users := UserList{
		User{
			Username: "alice",
			Email:    "[email protected]",
			Name:     "Alice",
			Roles:    []string{"user", "admin"},
		},
		User{
			Username: "bob",
			Email:    "[email protected]",
			Name:     "Bob",
			Roles:    []string{"user"},
		},
	}

	v1, err := version.NewVersion("1.0.0")
	if err != nil {
		log.Panic(err)
	}
	v2, err := version.NewVersion("2.0.0")

	output, err := MarshalUsers(v1, []string{"api"}, users)
	if err != nil {
		log.Panic(err)
	}
	fmt.Println("Version 1 output:")
	fmt.Printf("%s\n\n", output)

	output, err = MarshalUsers(v2, []string{"api"}, users)
	if err != nil {
		log.Panic(err)
	}
	fmt.Println("Version 2 output:")
	fmt.Printf("%s\n\n", output)

	output, err = MarshalUsers(v2, []string{"api", "personal"}, users)
	if err != nil {
		log.Panic(err)
	}
	fmt.Println("Version 2 output with personal group too:")
	fmt.Printf("%s\n\n", output)

}

// Output:
// Version 1 output:
// [
//   {
//     "name": "Alice",
//     "username": "alice"
//   },
//   {
//     "name": "Bob",
//     "username": "bob"
//   }
// ]
//
// Version 2 output:
// [
//   {
//     "name": "Alice",
//     "roles": [
//       "user",
//       "admin"
//     ],
//     "username": "alice"
//   },
//   {
//     "name": "Bob",
//     "roles": [
//       "user"
//     ],
//     "username": "bob"
//   }
// ]
//
// Version 2 output with personal group too:
// [
//   {
//     "email": "[email protected]",
//     "name": "Alice",
//     "roles": [
//       "user",
//       "admin"
//     ],
//     "username": "alice"
//   },
//   {
//     "email": "[email protected]",
//     "name": "Bob",
//     "roles": [
//       "user"
//     ],
//     "username": "bob"
//   }
// ]

Output ordering

Sheriff converts the input struct into a basic structure using map[string]interface{}. This means that the generated JSON will not have the same ordering as the input struct. If you need to have a specific ordering then a custom implementation of the KVStoreFactory can be passed as an option.

Providing a custom KV Store is likely to have a negative impact on performance, as such it should be used only when necessary.

For example:

package main

import (
	"github.com/liip/sheriff/v2"
	orderedmap "github.com/wk8/go-ordered-map/v2"
)

type OrderedMap struct {
	*orderedmap.OrderedMap[string, interface{}]
}

func NewOrderedMap() *OrderedMap {
	return &OrderedMap{orderedmap.New[string, interface{}]()}
}

func (om *OrderedMap) Set(k string, v interface{}) {
	om.OrderedMap.Set(k, v)
}

func (om *OrderedMap) Each(f func(k string, v interface{})) {
	for pair := om.Newest(); pair != nil; pair = pair.Prev() {
		f(pair.Key, pair.Value)
	}
}

func main() {
	opt := &sheriff.Options{
		KVStoreFactory: func() sheriff.KVStore {
			return NewOrderedMap()
		},
	}

	// ...
}

Benchmarks

There's a simple benchmark in bench_test.go which compares running sheriff -> JSON versus just marshalling into JSON and runs on every build. Just marshalling JSON itself takes usually between 3 and 5 times less nanoseconds per operation compared to running sheriff and JSON.

Want to make sheriff faster? Please send us your pull request or open an issue discussing a possible improvement 🚀!

Acknowledgements

  • This idea and code has been created partially during a Liip hackday.
  • Thanks to @basgys for reviews & improvements.

Related projects

mweibel/php-to-go is a code generator translating PHP models (using JMS serializer) to Go structs with sheriff tags. The two projects were initially developed together and just recently php-to-go has been split out and published too.

sheriff's People

Contributors

alecsammon avatar byashimov avatar fale avatar faryon93 avatar hwshadow avatar kevinlynx avatar mandeepji avatar masseelch avatar mweibel avatar nazdroth avatar simaotwx avatar torbenck avatar xuancanh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sheriff's Issues

Errors: "github.com/hashicorp/go-version" imported as fsnotify and not used ; undefined: version

Build of https://github.com/zmap/zdns fails in liip/sheriff:

github.com/liip/sheriff
# github.com/liip/sheriff
vendor/github.com/liip/sheriff/sheriff.go:10:2: "github.com/hashicorp/go-version" imported as fsnotify and not used
vendor/github.com/liip/sheriff/sheriff.go:25:14: undefined: version
vendor/github.com/liip/sheriff/sheriff.go:158:26: undefined: version
vendor/github.com/liip/sheriff/sheriff.go:168:26: undefined: version
*** Error code 1

FreeBSD 13.2

Support Unmarshal too

It is great to see people actually realize we need something work like sheriff in Golang's world.

As a Symfony developer too I enjoy JMSSerializationBundle very much on serializing and deserializing data easily.

It would be great if it can be supported too.

Support nil interfaces

Marshal currently triggers a panic if it runs into any nil pointers.

I think such values should be stored in the resulting map as nils. This would be consistent with the json package, allowing the resulting map to be passed to json.Marshal and get the same result as if the struct itself were passed directly to json.Marshal.

Example code:


import (
	"encoding/json"
	"fmt"
	"github.com/liip/sheriff"
)

func main() {
	expected := map[string]interface{}{"thing": nil}
	fmt.Println(expected)

	type Example struct {
		Thing *int `json:"thing" groups:"include"`
	}
	x := Example{}
	_, _ = json.Marshal(x) // {"thing":null}

	opt := sheriff.Options{
		Groups: []string{"include"},
	}
	result2, err := sheriff.Marshal(&opt, x)
	if err != nil {panic(err)}
	fmt.Println(result2)
}```

Implement unmarchaling

I think it would be very useful to also implement unmarshaling functions to be able to secure also writes for post/put/patch.

map[uint]Struct support

Please add support uint

marshaller: Unable to marshal type int. Struct required.

marshaller: Unable to marshal type uint. Struct required.

json.Marshal() and sheriff.Marshal() gives different results when empty map present in struct

When empty map is present in Structure sheriff.Marshal() gives nil but json.Marshal() gives empty json {}

I used existing test TestMarshal_EmptyMap and Marshal same object using json & sheriff and got different results.

Modified Test func:

func TestMarshal_EmptyMap(t *testing.T) {                   
        emp := EmptyMapTest{                                
                AMap: make(map[string]string),              
        }                                                   
        o := &Options{                                      
                Groups: []string{"test"},                   
        }                                                   
                                                            
        actualMap, err := Marshal(o, emp)                   
        assert.NoError(t, err)                              
                                                            
        actual, err := json.Marshal(actualMap)              
        assert.NoError(t, err)                              
                                                            
        expected, err := json.Marshal(emp)                  
        assert.NoError(t, err)                              
                                                            
        assert.Equal(t, string(expected), string(actual))   
}

Test result:

--- FAIL: TestMarshal_EmptyMap (0.00s)
    sheriff_test.go:510:
                Error Trace:    sheriff_test.go:510
                Error:          Not equal:
                                expected: "{\"a_map\":{}}"
                                actual  : "{\"a_map\":null}"

                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -{"a_map":{}}
                                +{"a_map":null}
                Test:           TestMarshal_EmptyMap
FAIL

XML Marshalling Support

Hey there, great library. I'm hoping to use it, however, when trying to use xml.Marshal, I get no output. Is there a way to add in support for this? I'm happy to do it if you give me a pointer or two.

Groups are not applied correclty in slices.

I have the following structs:

type Job struct {
	ID int `json:"id,omitempty" groups:"job:list"`
	Edges JobEdges `json:"edges" groups:"job:list"`
}

type JobEdges struct {
	Users []*User `json:"users" groups:"job:list"`
}

type User struct {
	ID int `json:"id,omitempty"`
	Edges UserEdges `json:"edges" groups:"user:read"`
}

type UserEdges struct {
	Sessions []*Session `json:"-"`
	Jobs []*Job `json:"jobs" groups:"user:read"`
}

type Session struct {
	ID go_token.Token `json:"token"`
	Edges         SessionEdges `json:"edges"`
}

type SessionEdges struct {
	User *User
}

Calling sheriff.Marshal(&sheriff.Options{Groups: []string{"job:list", "user:list"}}, job) on a Job-struct results in the Edges property of the attached users to be included in the resulting map. I'd expect it not be there since the groups given to sheriff.Marshal() are job:list and user:list but the Edges-property has none of the given groups attached.

Example output:

{
  "id": 1,
  "edges": {
    "users": [
      {
        "id": 1,
        "edges": {
          "jobs": null
        }
      }
    ]
  }
}

Cause bug when have struct method

Hi, I found a bug when a structure having method.
For the example,

Find the User struct in sample code

type User struct {
	Username string   `json:"username" groups:"api"`
	Email    string   `json:"email"`
	Name     string   `json:"name" groups:"api"`
	Roles    []string `json:"roles" groups:"api"`
}

Add the following code below,

func (u User) String() string {
	return fmt.Sprintf("User<%s %s>", u.Username, u.Name)
}

It will brick the scoping, marshal all the stuff into json.

Serialization discrepency between slice vs array of slice with custom MarshalJSON

Hi,
I have a TestModel with a Users field ([]UserTestModel). I have a MarshalJSON method that overrides this Users field with different values. If I serialize a slice of TestModel, the Users field is overridden correctly. However, if I serialize just one TestModel, the Users field is not serialized at all, regardless of the JSON tag I use.

package controller

import (
	"encoding/json"
	"github.com/liip/sheriff/v2"
	"log"
	"net/http"
)

type TestController struct{}

type UserTestModel struct {
	Id        int    `json:"id" groups:"test"`
	Firstname string `json:"firstname" groups:"test"`
	Lastname  string `json:"lastname" groups:"test"`
}

type TestModel struct {
	Name        string          `json:"name" groups:"test"`
	Description string          `json:"description" groups:"test"`
	Users       []UserTestModel `json:"-" groups:"test"`
}

func (testModel TestModel) MarshalJSON() ([]byte, error) {
	type TestModelAlias TestModel

	return json.Marshal(&struct {
		TestModelAlias
		Users []UserTestModel `json:"users" groups:"test"`
	}{
		TestModelAlias: TestModelAlias(testModel),
		Users: []UserTestModel{
			{
				Id:        999,
				Firstname: "superman",
				Lastname:  "superman",
			},
		},
	})
}

func (testController *TestController) TestHandler(w http.ResponseWriter, r *http.Request) {
	//testModel := TestModel{
	//	Name:        "test",
	//	Description: "this is a test",
	//	Users: []UserTestModel{
	//		{
	//			Id:        1,
	//			Firstname: "steve",
	//			Lastname:  "jobs",
	//		},
	//		{
	//			Id:        2,
	//			Firstname: "anthony",
	//			Lastname:  "hopkins",
	//		},
	//		{
	//			Id:        3,
	//			Firstname: "tom",
	//			Lastname:  "cruise",
	//		},
	//	},
	//}

	testModels := []TestModel{
		{
			Name:        "test",
			Description: "this is a test",
			Users: []UserTestModel{
				{
					Id:        1,
					Firstname: "steve",
					Lastname:  "jobs",
				},
				{
					Id:        2,
					Firstname: "anthony",
					Lastname:  "hopkins",
				},
				{
					Id:        3,
					Firstname: "tom",
					Lastname:  "cruise",
				},
			},
		},
		{
			Name:        "test",
			Description: "this is a test",
			Users: []UserTestModel{
				{
					Id:        1,
					Firstname: "steve",
					Lastname:  "jobs",
				},
				{
					Id:        2,
					Firstname: "anthony",
					Lastname:  "hopkins",
				},
				{
					Id:        3,
					Firstname: "tom",
					Lastname:  "cruise",
				},
			},
		},
	}

	options := &sheriff.Options{
		Groups: []string{
			"test",
		},
	}
	testModelSerialized, _ := sheriff.Marshal(options, testModels)

	if err := Encode(w, http.StatusOK, testModelSerialized); err != nil {
		log.Println(err)
		w.WriteHeader(http.StatusInternalServerError)
		return
	}
}

Result for a slice of TestModel

[
    {
        "name": "test",
        "description": "this is a test",
        "users": [
            {
                "id": 999,
                "firstname": "superman",
                "lastname": "superman"
            }
        ]
    },
    {
        "name": "test",
        "description": "this is a test",
        "users": [
            {
                "id": 999,
                "firstname": "superman",
                "lastname": "superman"
            }
        ]
    }
]

Result for one TestModel

{
    "description": "this is a test",
    "name": "test"
}

nil handling

pinging @lombare @simaotwx @TorbenCK @mandeepji

We have several conflicting opinions about how to handle nil and/or empty maps/slices:
see the following PRs:

I accidentally merged again something which was previously called a regression just before (sorry about that); will revert that change.

What is your opinion on how this should be handled? Would we need additional flags to capture this?
Just a new version (github.com/liip/sheriff/v1) to make sure there aren't regressions in existing codebases?

Please let me know your thoughts. Thanks.

Sheriff reorders the fields alphabetically

The usage of sheriff.Marshal(options, data) reorganizes the fields alphabetically instead of keeping the original order. (v2.0.0-beta.1)

Expected

{
    "id": "a5b002bb-b63d-4192-b665-ec4dadcb036b",
    "name": "test",
    "createdAt": "2024-03-17T06:56:04Z",
    "updatedAt": "2024-03-17T06:56:04Z"
}

Actual behavior

{
    "createdAt": "2024-03-17T06:56:04Z",
    "id": "a5b002bb-b63d-4192-b665-ec4dadcb036b",
    "name": "test",
    "updatedAt": "2024-03-17T06:56:04Z"
}

Panic when serializing null pointers

Hello I think there is an issue in the serialization process because when we serialize this kind of struct :

type Struct struct {
    Test map[string]*bool  `json:"test" groups:"test"`
}

Via :

data := Struct{
    Test: map[string]*bool{
        "a": &true // replace this by a function that makes a ptr out of a boolean
        "b": nil
     }
}

opts := sheriff.Options{Groups: string[]{"test"}}
jsonStr, err := sheriff.Marshal(&o, data); // <-- panics here
// error management here

Panic occurs here at sheriff.go line 222 :

220:	if k == reflect.Ptr {
221: 		v = v.Elem()
222:		val = v.Interface()
223:		k = v.Kind()
224:	}

It looks like when the value is a Ptr, sheriff tries to dereference it and then get it's value, but if the Ptr is nil then it dereferences a nil.

Tell me if I'm wrong but a safe fix for this could be :

if k == reflect.Ptr {
	if v.IsNil() {
		return v.Interface()
	}
    	v = v.Elem()
	val = v.Interface()
	k = v.Kind()
}

Use GORM Library with sheriff package

I'm trying to use the Golang Sheriff package with GORM Library somehow I'm getting an empty response. Tried a couple of ways no luck. Is it possible? Any help that would be great. TIA

//STRUCT
    type Book struct {
    ID        uint `json:"id" gorm:"primary_key" groups:"not_detail,detail"`
    Realname      string `json:"real_name" groups:"not_detail,detail"`
    LanguageId int `json:"language_id" groups:"not_detail,detail"`
    Language   Language `json:"Language" groups:"detail"`
    CreatedAt time.Time `json:"created_at" groups:"not_detail,detail"`
    UpdatedAt time.Time `json:"updated_at" groups:"not_detail,detail"`
}
//CONTROLLER LOGIC
func FindBooks(c *gin.Context)  {
    db := c.MustGet("db").(*gorm.DB)
    var books []models.Book
    result := db.Find(&books)

    o := sheriff.Options{
        Groups: []string{"not_detail"},
    }

    d, err := sheriff.Marshal(&o, result)
    if err != nil {
        panic(err)
    }

    c.JSON(http.StatusOK, gin.H{"data":d})
}

[Feature request] Add option to add fields without 'groups' tag by default

Hi,

First, thank for this library, very helpful for me.

I use sheriff in a project with big struct, with many fields. So it is not very practical to add the groups tag on fields on by one.

Can you provide a way to automatically add field without groups tag ? For exemple, an attribute addFieldsWithoutGroups in shriff.Options

Thanks !

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.