perimeterx/marshmallow

Marshmallow provides a flexible and performant JSON unmarshalling in Go. It specializes in dealing with unstructured struct - when some fields are known and some aren't, with zero performance overhead nor extra coding needed.

marshmallow's Issues

Support any types that has UnmarshalJSON/MarshalJSON method

I wonder if you could support non-primitive types such as time.Time just like the standard json.Unmarshal json.Marshal does. I only need it when such types implemented UnmarshalJSON/MarshalJSON methods.

Example for nested structures and rendering only a nested struct

My use case is the Square API, where I want to retrieve a list of items, and iterate over some internal data, updating only the internal objects, not the main one.

The data is structured like this:

type ApiWrapper struct {
	Cursor       string                 `json:"cursor,omitempty"`
	Objects      []*Object              `json:"objects,omitempty"`
}

type Object struct {
	Type              string                 `json:"type,omitempty"`
	Id                string                 `json:"id,omitempty"`
	UpdatedAt         time.Time              `json:"updated_at,omitempty"`
	CreatedAt         time.Time              `json:"created_at,omitempty"`
	Version           int64                  `json:"version,omitempty"`
	IsDeleted         bool                   `json:"is_deleted,omitempty"`
	ItemData          *Item                  `json:"item_data,omitempty"`
	ItemVariationData *ItemVariation         `json:"item_variation_data,omitempty"`
}

type Item struct {
	Name         string                 `json:"name,omitempty"`
	Description  string                 `json:"description,omitempty"`
	Visibility   string                 `json:"visibility,omitempty"`
	Variations   []*Object              `json:"variations,omitempty"`
}

type ItemVariation struct {
	ItemId       string                 `json:"item_id,omitempty"`
	SKU          string                 `json:"sku,omitempty"`
}

In this case, I want to fetch a list of ITEMs from the API, each of which will be of Type ITEM, and have ItemData set. In the ItemData, it will have a list of variations, each of which is an Object of type ITEM_VARIATION, with ItemVariationData set.

What I want to do here is modify each of the ItemVariationData items, and then render the Object holding that variation's data.

I've tried implementing HandleJSONData() such as:

func (e *Item) HandleJSONData(data map[string]interface{}) {
	e.InternalData = data
}

and then using something like this to render the Object out:

func spewJSON(m map[string]any, obj any) {
	structOut, err := json.Marshal(obj)
	if err != nil {
		panic(err)
	}

	err = json.Unmarshal(structOut, &m)
	if err != nil {
		panic(err)
	}

	out, err := json.MarshalIndent(m, "", "\t")
	if err != nil {
		panic(err)
	}

	fmt.Println(string(out))
}

func main() {
	for _, itemObject := range result.Objects {
		for _, variationObject := range itemObject.ItemData.Variations {
			if variationObject.ItemVariationData.SKU == "" {
				sku := makeSku()
				log.Printf("item %s, variation %s, sku was empty, is now %s", variationObject.ItemVariationData.ItemId, variationObject.Id, sku)
				variationObject.ItemVariationData.SKU = sku
			} else {
				log.Printf("item %s, variation %s, sku set to %s", variationObject.ItemVariationData.ItemId, variationObject.Id, variationObject.ItemVariationData.SKU)
			}
			spewJSON(variationObject.InternalData, variationObject)
		}
	}
}

However, while all the "unknown" fields in the Object render, the sub-object does not.

Any hints on where to go from here?

handling of Unmarshall of multiple files processed in parallel

as mentioned in Issue 26

proper solution to report errors from unknown elements is to implement methods for each part of the structure.

question remains, if marshmallow.Unmarshall() is executed in closure, in parallel processing, how can I provide the information to the error handlers, which file is being processed, especiallyt in parallel processing?

thx a lot

Support capturing nested unknown fields

Following the discussion on this issue, we want to be able to capture unknown fields in nested objects.

Reporting missing fields deeper in structure

Having a json:

{"field1": "val", "field2" : {"field3" : "value", "unknown" : { "inner" : "val1", "inner2" : "val2" }}}
and structures:

type Dummy struct {
	Field1 string json:"field1"
	Field2 Inner  json:"field2"
}

type Inner struct {
	Field3 string json:"field3"
}

unmarshall.go does not report the "unknown" key at all.

for !d.lexer.IsDelim('}') {
		key := d.lexer.UnsafeFieldName(false)
		d.lexer.WantColon()
>>		refInfo, exists := fields[key].  // => exists is false
		if exists {
			value, isValidType := d.valueByReflectType(refInfo.t)
			if isValidType {
				if value != nil && doPopulate {
					field := refInfo.field(structValue)
					assignValue(field, value)
				}
				if !d.options.excludeKnownFieldsFromMap {
					if result != nil {
						result[key] = value
					}
					if clone != nil {
						clone[key] = value
					}
				}
			} else {
				switch d.options.mode {
				case ModeFailOnFirstError:
					return nil, false
				case ModeFailOverToOriginalValue:
					if !forcePopulate {
						result[key] = value
					} else {
						clone[key] = value
						d.lexer.WantComma()
						d.drainLexerMap(clone)
						return clone, false
					}
				}
			}
>>		} else {. // => both result and clone is nil, so nothing is reported
			value := d.lexer.Interface()
			if result != nil {
				result[key] = value
			}
			if clone != nil {
				clone[key] = value
			}
		}

see comments inside code from unmarshall.go

would there be an option to modify the code?
best would be to have an information, that "field2.unknown" field is not correctly mapped in data.

thank you

Marek

Problem with nested object implementing JSONDataHandler with skipPopulateStruct in ModeFailOverToOriginalValue

Due to recent changes, the flow for nested object which implements JSONDataHandler when WithSkipPopulateStruct(true) and ModeFailOverToOriginalValue is broken.

Example in playground.

UnmarshalFromJSONMap doesn't support int/int64

Hi, thanks for sharing this great library.

I was surprised that UnmarshalFromJSONMap only supports float64. Are you open to a patch to add support for int/int64?

"Why should I use Marshmallow" section in README.md is not clear

The examples compare map[string]any unmarshalling to struct. The stdlib can also do selective unmashalling so it really is NOT clear how marshmallow is different (and is better)?

Eg. the code below is a valid stdlib code (I only changed marshmallow->json package), so putting the stdlib map example and marshmallow struct example is just not fair.

func isAllowedToDrive(data []byte) (bool, error) {
	v := struct {
		Age               int  `json:"age"`
		HasDriversLicense bool `json:"has_drivers_license"`
	}{}
	if err := json.Unmarshal(data, &v); err != nil {
		return false, err
	}

	if v.Age < 17 || !v.HasDriversLicense {
		return false, nil
	}

        /*
	for key := range result {
		if strings.Contains(key, "prior_conviction") {
			return false, nil
		}
	}
        */

	return true, nil
}

Can you please change the map implementation to struct similar to marshmallow and list how marshmallow is different(better?)? Eg. dealing with zero values (eg. bool false vs not present in JSON), which can be done with pointers in stdlib or perhaps dealing with arbitrary types? eg. 1234 vs "1234", etc.

Add YAML support

This project looks cool!
It would be really helpful if I could natively use it for YAML files too 😁

Unmarshal giving parse error on valid JSON

I am trying to use this package for a server which will take JSON input and fire off an alert to our monitoring system. I have a fairly simple JSON package:

{
    "alert_summary": "uh oh",
    "detailed_description": "i dont know what to do",
    "deviceName": "test",
    "eventInfo": {
        "eventType": "disconnect",
        "more": "stuff",
        "there": "may",
        "be": "more",
        "values": "here"
    },
    "options": {
        "staging": true
    }
}

Within the eventInfo object, I will have an indeterminate number of fields depending on how much info the alert will be providing (hence why I found this package, I am considering everything within "eventInfo" to be "extra metadata" to add to the alert but I won't necessarily know what that info will be when the event is submitted.

I have the following code:

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"net/http"

	"github.com/go-playground/validator/v10"
	"github.com/perimeterx/marshmallow"
	"myorg/utils"
)

// Expected formatting of an incoming JSON event
type IncomingMessage struct {
	AlertSummary        string        `json:"alert_summary" validate:"required"`
	DetailedDescription string        `json:"detailed_description" validate:"required"`
	DeviceName          string        `json:"deviceName" validate:"required"`
	EventMetaData       struct {
		EventType string `json:"eventType" validate:"required"`
	} `json:"eventInfo" validate:"required"`
	Options             struct {
		AffectedCi   string `json:"affected_ci"`
		AffectedArea string `json:"affected_area"`
		HelpURL      string `json:"help_url"`
	} `json:"options"`
}

func StartServer(port string) (*http.Server, error) {
	srv := &http.Server{Addr: ":" + port}
	marshmallow.EnableCache()
	http.HandleFunc("/api/v1/submitRequest", handleIncomingEvent)
	http.ListenAndServe(":"+port, nil)
	return srv, nil
}

// Validates the incoming JSON for the appropriate formatting.
// If all ok, passes it on to be processed.
func handleIncomingEvent(w http.ResponseWriter, req *http.Request) {
	var incoming IncomingMessage
	var validate = validator.New()

	body, readingErr := io.ReadAll(req.Body)
	if readingErr != nil {
		fmt.Println("error reading body")
                return
	}

	result, unmarshallingErr := marshmallow.Unmarshal(body, &incoming)
	if unmarshallingErr != nil {
		utils.Log.Warnf("Could not unmarshal incoming data from %s: %s", req.Host, unmarshallingErr.Error())
		message := fmt.Sprintf("error decoding request body: %s", unmarshallingErr)
		returnMessage("error", http.StatusBadRequest, message, w)
		return
	}

	validationErr := validate.Struct(incoming)
	if validationErr != nil {
		utils.Log.Warnf("Bad Request Recieved from %s: %s", req.Host, validationErr.Error())
		message := fmt.Sprintf("input not in expected format: %s: %s", validationErr, validationErr.Error())
		returnMessage("error", http.StatusBadRequest, message, w)
		return
	}

	// If we get here do some stuff!
	// ...
	returnMessage("ok", http.StatusOK, "well done", w)

}

func returnMessage(status string, responseCode int, message string, w http.ResponseWriter) {
	var response OutgoingMessage
	response.Status = status
	response.ResponseCode = responseCode
	response.Message = message
	jsonResponse, _ := json.Marshal(response)
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(responseCode)
	w.Write(jsonResponse)
}

I am submitting a payload to my server with Insomnia however Marshmallow is giving me an error: Could not unmarshal incoming data from localhost: parse error: syntax error near offset 156 of ': "stuff"...'

I'm not entirely sure why this is happening. I realise that "more": "stuff" is not part of the struct but I was under the impression that would just be ignored when writing it in to the struct, and these values would then be available in the resultant map that also gets produced.

Is this a bug, or am I formatting my JSON incorrectly and/or handling it incorrectly?

Thanks.

Excluding known typed fields from returned map

It would be great if there was an option that excludes from resulting map[string]interface{} the fields that were unmarshalled to the struct object, see the example below:

v := struct {
	Foo string `json:"foo"`
	Boo []int  `json:"boo"`
}{}
result, err := marshmallow.Unmarshal([]byte(`{"foo":"bar","boo":[1,2,3],"goo":12.6}`), &v)
fmt.Printf("v=%+v, result=%+v, err=%v", v, result, err)
// Output: v={Foo:bar Boo:[1 2 3]}, result=map[goo:12.6], err=<nil>
//                                  ^^^^^^     ^^^
// only "goo" field is present in result since "foo" and "boo" are explicitly typed

Is there a way to marshal de JSON

Hi!
I am looking for a way to read, modify and save a JSON of which I am only interested in part of its content.
This package looks like it could be useful for the first two steps, but I'm not sure about the last one.

Is there any way to do it? Either through the API of the package or in some recommended way.

Thanks!

(P.S. I tried to open a discussion instead of an issue but I can't do it)

Unmarshaling doesn't work for composed types

Hey,
I was trying to use the lib to unmarshal a composed struct, and expected it to work like the json package and to fill the nested struct as well.
It looks like the issue is in mapStructFields that doesn't handle composed structs so it skips the nested values.

package main

import (
	"fmt"
	"github.com/perimeterx/marshmallow"
)

type A struct {
	Foo string `json:"foo"`
	Boo []int  `json:"boo"`
}

type B struct {
	A
}

func main() {
	marshmallow.EnableCache() // this is used to boost performance, read more below
	a := A{}
	result, err := marshmallow.Unmarshal([]byte(`{"foo":"bar","boo":[1,2,3],"goo":12.6}`), &a)
	fmt.Printf("a=%+v, result=%+v, err=%v", a, result, err)
	// Output: a={Foo:bar Boo:[1 2 3]}, result=map[boo:[1 2 3] foo:bar goo:12.6], err=<nil>

	b := B{}
	result, err = marshmallow.Unmarshal([]byte(`{"foo":"bar","boo":[1,2,3],"goo":12.6}`), &b)
	fmt.Printf("b=%+v, result=%+v, err=%v", b, result, err)
	// Output: b={A:{Foo: Boo:[]}}, result=map[boo:[1 2 3] foo:bar goo:12.6], err=<nil>
}

Problem with nested object implementing JSONDataHandler with skipPopulateStruct

Due to recent changes, the flow for nested object which implements JSONDataHandler when WithSkipPopulateStruct(true) is broken.

Example in playground.