jacobdufault/fullserializer

A robust JSON serialization framework that just works with support for all major Unity export platforms.

Looking for maintainer

Please see #154.

Full Serializer

Full Serializer is an easy to use and robust JSON serializer that just works. It'll serialize pretty much anything you can throw at it and work on every major Unity platform, including consoles. Full Serializer doesn't use exceptions, so you can activate more stripping options on iOS.

Best of all, Full Serializer is completely free to use and available under the MIT license!

Why?

There were no serializers that just work in Unity that are free and target all export platforms. Full Inspector needed one, so here it is.

Installation

Import the Source folder into your Unity project! You're good to go! (Also see the DLL based import at the bottom of this document).

Usage

Usage is identical to Unity's default serialization, except that you don't have to mark types as [Serializable]. Here's an example:

struct SerializedStruct {
    public int Field;

    public Dictionary<string, string> DictionaryAutoProperty { get; set; }

    [SerializeField]
    private int PrivateField;
}

Here are the precise (default) rules:

• Public fields are serialized by default
• Auto-properties that are at least partially public are serialized by default
• All fields or properties annotated with [SerializeField] or [fsProperty] are serialized
• Public fields/public auto-properties are not serialized if annotated with [NonSerialized] or [fsIgnore]. [fsIgnore] can be used on properties (unlike [NonSerialized]).

Inheritance and cyclic object graphs are automatically handled by Full Serializer. You don't need to do anything.

Here's a simple example of how to use the Full Serializer API to serialize objects to and from strings.

using System;
using FullSerializer;

public static class StringSerializationAPI {
    private static readonly fsSerializer _serializer = new fsSerializer();

    public static string Serialize(Type type, object value) {
        // serialize the data
        fsData data;
        _serializer.TrySerialize(type, value, out data).AssertSuccessWithoutWarnings();

        // emit the data via JSON
        return fsJsonPrinter.CompressedJson(data);
    }

    public static object Deserialize(Type type, string serializedState) {
        // step 1: parse the JSON data
        fsData data = fsJsonParser.Parse(serializedState);

        // step 2: deserialize the data
        object deserialized = null;
        _serializer.TryDeserialize(data, type, ref deserialized).AssertSuccessWithoutWarnings();

        return deserialized;
    }
}

Note that this API example will throw exceptions if any errors occur. Additionally, error recovery is disabled in this example - if you wish to enable it, simply replace the AssertSuccessWithoutWarnings calls with AssertSuccess.

Custom Serialization

Full Serializer allows you to easily customize how serialization happens, via [fsObject], fsConverter, and fsObjectProcessor.

Simple Customization with [fsObject]

You can easily customize the serialization of a specific object by utilizing [fsObject]. There are a number of options:

You can specify what the default member serialization is by changing the MemberSerialization parameter. The options are OptIn, OptOut, and Default. OptIn requires that every serialize member be annotated with fsProperty, OptOut will serialize every member except those annotated with fsIgnore, and Default uses the default intelligent behavior where visibility level and property type are examined.

You can also specify a custom converter or processor to use directly on the model. This is more efficient than registering a custom converter / processor on the fsSerializer instance and additionally provides portability w.r.t. the actual fsSerializer instance; the fsSerializer creator does not need to know about this specific converter / processor.

Advanced Customization with Converters

Converters (to/from JSON) enable complete customization over the serialization of an object. Each converter expresses interest in what types it wants to take serialization over; there are two methods to do this. The more powerful (but slower) method is present in fsConverter, which determines if it interested via a function callback, and fsDirectConverter, which directly specifies which type of object it will take over conversion for. The primary limitation for fsDirectConverter is that it does not handle inheritance.

Example

Suppose we have this model:

public struct Id {
    public int Identifier;
}

We want to serialize this Id instance directly to an integer. Normally this is difficult, but with converters it is doable.

For example, we want to serialize new Id { Identifier = 3 } to 3.

Let's take a look at the converter which will handle this:

public class IdConverter : fsDirectConverter {
    public override Type ModelType { get { return typeof(Id); } }

    public override object CreateInstance(fsData data, Type storageType) {
        return new Id();
    }

    public override fsResult TrySerialize(object instance, out fsData serialized, Type storageType) {
        serialized = new fsData(((Id)instance).Identifier);
        return fsResult.Success;
    }

    public override fsResult TryDeserialize(fsData data, ref object instance, Type storageType) {
        if (data.IsInt64 == false) return fsResult.Fail("Expected int in " + data);

        instance = new Id { Identifier = (int)data.AsInt64 };
        return fsResult.Success;
    }
}

The converter is fairly straight-forward. ModelType maps to the type of object this converter applies to. It is used when you register a converter using either the fsConverterRegistrar or AddConverter.

CreateInstance allocates the actual object instance. If you're curious why this method is separate from TryDeserialize, then rest assured knowing that it is because cyclic object graphs require deserializing and object instance allocation to be separated (otherwise deserialization would enter into an infinite loop).

TrySerialize serializes the object instance. instance is guaranteed to be an instance of Id (or rather whatever CreateInstance returned), and storageType is guaranteed to be equal to typeof(Id).

TryDeserialize deserializes the json data into the object instance.

What's up with all of this fsResult stuff? Quite simply, fsResult is used so that Full Serializer doesn't have to use exceptions. Errors and problems happen when (typically when deserializing) - the fsResult instance can contain warning information or error information. If there is an error, then that almost certainly means that you want to stop deserialization, but for a warning you should keep going. You can ignore errors or treat them as warnings if your converter supports partial deserialization.

You may be curious what happens if we try to serialize, say, this object graph:

class Hello {
    public object field;
}

Serialize(new Hello { field = new Id() });

Rest assured knowning that it serializes correctly and as expected, even though we have a custom converter. Full Serializer takes care of these type of details so you don't have to worry about it.

Converter Registration

There are three ways to register a converter.

• If you have access to the model type itself, then you can simply add an [fsObject] annotation. This registration method is static and cannot be modified at runtime. Here's an example:

[fsObject(Converter = typeof(IdConverter))]
public struct Id {
    public int Identifier;
}

• If you don't have access to the model type or the serializer, then you can register the converter using fsConverterRegistrar. This registration method is static and cannot be modified at runtime. Here's an example:

namespace FullSerializer {
    partial class fsConverterRegistrar {
        // Method 1: Via a field
        // Important: The name *must* begin with Register
        public static IdConverter Register_IdConverter;

        // Method 2: Via a method
        // Important: The name *must* begin with Register
        public static void Register_IdConverter() {
            // do something here, ie:
            Converters.Add(typeof(IdConverter));
        }
    }
}

• If you have access to the fsSerializer instance, then you can dynamically determine which converters to register. For example:

void CreateSerializer() {
    var serializer = new fsSerializer();
    serializer.AddConverter(new IdConverter());
}

Full Converter Example

Here's the full example:

using System;
using FullSerializer;

[fsObject(Converter = typeof(IdConverter))]
public struct Id {
    public int Identifier;
}

public class IdConverter : fsDirectConverter {
    public override Type ModelType { get { return typeof(Id); } }

    public override object CreateInstance(fsData data, Type storageType) {
        return new Id();
    }

    public override fsResult TrySerialize(object instance, out fsData serialized, Type storageType) {
        serialized = new fsData(((Id)instance).Identifier);
        return fsResult.Success;
    }

    public override fsResult TryDeserialize(fsData data, ref object instance, Type storageType) {
        if (data.IsInt64 == false) return fsResult.Fail("Expected int in " + data);

        instance = new Id { Identifier = (int)data.AsInt64 };
        return fsResult.Success;
    }
}

Another Converter Example

Here's an example converter, with lots of comments to explain things as you read:

using System;
using FullSerializer;

// We're going to serialize MyType directly to/from a string.
[fsObject(Converter = typeof(MyTypeConverter))]
public class MyType {
    public string Value;
}

public class MyTypeConverter : fsConverter {
    public override bool CanProcess(Type type) {
        // CanProcess will be called over every type that Full Serializer
        // attempts to serialize. If this converter should be used, return true
        // in this function.
        return type == typeof(MyType);
    }

    public override fsResult TrySerialize(object instance,
        out fsData serialized, Type storageType) {

        // Serialize the data into the serialized parameter. fsData is a
        // strongly typed object store that maps directly to a JSON object model.
        // It's really easy to use.

        var myType = (MyType)instance;
        serialized = new fsData(myType.Value);
        return fsResult.Success;
    }

    public override fsResult TryDeserialize(fsData storage,
        ref object instance, Type storageType) {

        // Always make to sure to verify that the deserialized data is the of
        // the expected type. Otherwise, on platforms where exceptions are
        // disabled bad things can happen (if the data was actually an object
        // and you try to access a string, an exception will be thrown).
        if (storage.Type != fsDataType.String) {
            return fsResult.Fail("Expected string fsData type but got " + storage.Type);
        }

        // We just want to deserialize into the existing object instance. If
        // instance is a value type, then we can assign directly into instance
        // to update the value.

        var myType = (MyType)instance;
        myType.Value = storage.AsString;
        return fsResult.Success;
    }

    // Object instance construction is separated from deserialization so that
    // cycles can be correctly handled. If it's not possible to construct an
    // instance of the expected type here, then just return any non-null value
    // and construct the proper instance in TryDeserialize (though cycles will
    // *not* be handled properly).
    //
    // You do not need to override this method if your converted type is a
    // struct.
    public override object CreateInstance(fsData data, Type storageType) {
        return new MyType();
    }
}

A Third Converter Example

Let's take a look at this model:

public class Person {
    public string FirstName;
    public string LastName;
    public int Age;
}

We want to serialize it, but we want the serialized data to look like this:

{
    "Name": "John Doe",
    "Age": 25
}

which translates to this model instance:

var person = new Person {
    FirstName = "John",
    LastName = "Doe",
    Age = 25
};

Essentially, when we process an instance of Person, we want to serialize it to a single string field, ie, concat the first and last name.

Here's the converter:

using System;
using System.Collections.Generic;
using FullSerializer;

public class PersonConverter : fsDirectConverter<Person> {
    public override object CreateInstance(fsData data, Type storageType) {
        return new Person();
    }

    protected override fsResult DoSerialize(Person model, Dictionary<string, fsData> serialized) {
        // Serialize name manually
        serialized["Name"] = new fsData(model.FirstName + " " + model.LastName);

        // Serialize age using helper methods
        SerializeMember(serialized, null, "Age", model.Age);

        return fsResult.Success;
    }

    protected override fsResult DoDeserialize(Dictionary<string, fsData> data, ref Person model) {
        var result = fsResult.Success;

        // Deserialize name mainly manually (helper methods CheckKey and CheckType)
        fsData nameData;
        if ((result += CheckKey(data, "Name", out nameData)).Failed) return result;
        if ((result += CheckType(nameData, fsDataType.String)).Failed) return result;
        var names = nameData.AsString.Split(new[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
        if (names.Length != 2) return result += fsResult.Fail("Too many names");
        model.FirstName = names[0];
        model.LastName = names[1];

        // Deserialize age using basically only helper methods
        if ((result += DeserializeMember(data, null, "Age", out model.Age)).Failed) return result;

        return result;
    }
}

We're using fsDirectConverter again, but this time we derived from the generic variant. This generic variant makes writing the converter a bit easier, but it assumes that we will be serializing to a json object (and not, say, a string or number).

As you can see, the converter itself is pretty similar in structure to the last one. Our serialization logic is pretty simple - we are utilizing a few helper methods like SerializeMember though. Deserialization is a bit more hairy, since we have to validate a lot of information (since we don't want to throw any exceptions).

Here's the full example:

using System;
using System.Collections.Generic;
using FullSerializer;

[fsObject(Converter = typeof(PersonConverter))]
public class Person {
    public string FirstName;
    public string LastName;
    public int Age;
}

public class PersonConverter : fsDirectConverter<Person> {
    public override object CreateInstance(fsData data, Type storageType) {
        return new Person();
    }

    protected override fsResult DoSerialize(Person model, Dictionary<string, fsData> serialized) {
        // Serialize name manually
        serialized["Name"] = new fsData(model.FirstName + " " + model.LastName);

        // Serialize age using helper methods
        SerializeMember(serialized, null, "Age", model.Age);

        return fsResult.Success;
    }

    protected override fsResult DoDeserialize(Dictionary<string, fsData> data, ref Person model) {
        var result = fsResult.Success;

        // Deserialize name mainly manually (helper methods CheckKey and CheckType)
        fsData nameData;
        if ((result += CheckKey(data, "Name", out nameData)).Failed) return result;
        if ((result += CheckType(nameData, fsDataType.String)).Failed) return result;
        var names = nameData.AsString.Split(new[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
        if (names.Length != 2) return result += fsResult.Fail("Too many names");
        model.FirstName = names[0];
        model.LastName = names[1];

        // Deserialize age using basically only helper methods
        if ((result += DeserializeMember(data, null, "Age", out model.Age)).Failed) return result;

        return result;
    }
}

Advanced Customization with [fsObjectProcessor]

Object processors give you a bit of control of the serialization process before

Object processors are significantly more straightforward than converters. Here's the API:

Processor Example

using System;
using FullSerializer;

public class MyProcessor : fsObjectProcessor {
    public override bool CanProcess(Type type) {
        return true; // process everything that goes through the serializer
        return type == typeof(int); // process only ints
        return typeof(IInterface).IsAssignableFrom(type); // process only types that derive from IInterface
    }

    public override void OnBeforeDeserialize(Type storageType, ref fsData data) {
        // Invoked before deserialization begins. Feel free to modify the data that will be used to deserialize.
    }

    public override void OnAfterDeserialize(Type storageType, object instance) {
        // Invoked after deserialization has finished. Update any state in instance / etc.
    }

    public override void OnBeforeSerialize(Type storageType, object instance) {
        // Invoked before serialization begins. Update any state inside of instance / etc.
    }

    public override void OnAfterSerialize(Type storageType, object instance, ref fsData data) {
        // Invoked after serialization has finished. Update any state inside of instance, modify the output data, etc.
    }
}

Processor Registration

There are two ways to register a processor:

• You can specify it directly on the model. In this case, CanProcess will never get invoked (you should probably throw a NotSupportedException if it does).

[fsObject(Processor = typeof(MyProcessor))]
public class MyModel {}

• Add the processor to the serializer. CanProcess will be invoked on every type that the serializer tries to serialize/deserialize to determine if it is interested in the given type. For this reason, fsObject is the preferred method of registration since it is more performant.

void CreateSerializer() {
    var serializer = new fsSerializer();
    serializer.AddProcessor(new MyProcessor());
}

Versioning

Full Serializer supports versioning for serialization. You can specify the previous version of an object by utilizing these parameters for [fsObject].

PreviousModels
VersionString

PreviousModels is an array of types that specify the prior models that this object was migrated from. VersionString specifies a unique identifier for this model that separates it from all other prior model instances. Note that if a model should be versioned, it needs to initially have a VersionString parameter, otherwise no migration will be performed.

Here is a simple object migration:

[fsObject("1")]
public struct Model_v1 {
    public int A;
}

[fsObject("2", typeof(Model_v1))]
public struct Model {
    public int B;

    public Model(Model_v1 model) {
        B = model.A;
    }

    // note: We still should have a default constructor, but since we're a
    // struct one is automatically created for us      
}

Notice in particular that we have a constructor on Model that accepts an instance of Model_v1. If Full Serializer detects that we are deserializing old data, it will first deserialize it into an instance of Model_v1 and then return a newly constructed instance of Model via the Model_v1 constructor.

All version strings have to be unique (if not, an error will be issued) and there can be no cycles in the versioning import graph (there can be more than one previous model).

We can easily introduce a new Model type and then we just rename Model to Model_v2 and Full Serializer will automatically send a Model_v1 instance through to Model_v1(deserialized) -> Model_v2(Model_v1) -> Model(Model_v2). Running deserializing this way prevents an explosion of required constructor types.

AOT Compilation

Full Serializer has introduced some support for automatically creating converters when appropriate. These converters will provide a speedup because they can completely eliminate the usage of reflection. Further, these AOT compiled converters enable usage of Full Serializer on Unity platforms where reflection is broken (ie, consoles), or where reflection requires the full .NET framework target (ie, il2cpp).

The AOT compiled serializers are a bit interesting. As Full Serializer runs serialization and it notices a type can be AOT compiled, it will emit metadata to perform the AOT compilation. After having run some serialization code, you can check fsAotCompilationManager to see if there are any available compilations (also see below for a function which will automatically generate AOT compilations for types which contain [fsProperty] or [fsObject]). If there are AOT compilations available, the output will be in the form of a C# file (stored as a string). You should save this file to your project / Assets folder.

The following class makes using the AOT system easier. There are two methods, AddSeenAotCompilations, and AddDiscoverableAotCompilations. AddSenAotCompilations will emit AOT compilations for types that the serializer has actually seen and serialized or deserialized (so you're guaranteed to use them), whereas AddDiscoverableAotCompilations tries to compile all types which have either a [fsObject] or [fsProperty] annotation.

using System;
using System.IO;
using System.Reflection;
using FullSerializer;
using UnityEngine;

public static class AotHelpers {
    public const string OutputDirectory = "Assets/fsAotCompilations/";

    [UnityEditor.MenuItem("FullSerializer/Add Seen Aot Compilations (minimal output)")]
    public static void AddSeenAotCompilations() {
        if (Directory.Exists(OutputDirectory) == false) {
            Directory.CreateDirectory(OutputDirectory);
        }

        foreach (var aot in fsAotCompilationManager.AvailableAotCompilations) {
            Debug.Log("Performing AOT compilation for " + aot.Key.CSharpName(true));
            var path = Path.Combine(OutputDirectory, "AotConverter_" + aot.Key.CSharpName(true, true) + ".cs");
            var compilation = aot.Value;
            File.WriteAllText(path, compilation);
        }
    }

    [UnityEditor.MenuItem("FullSerializer/Add Discoverable Aot Compilations (more output)")]
    public static void AddAllDiscoverableAotCompilations() {
        if (Directory.Exists(OutputDirectory) == false) {
            Directory.CreateDirectory(OutputDirectory);
        }

        foreach (var assembly in AppDomain.CurrentDomain.GetAssemblies()) {
            foreach (Type t in assembly.GetTypes()) {
                bool performAot = false;

                // check for [fsObject]
                {
                    var props = t.GetCustomAttributes(typeof(fsObjectAttribute), true);
                    if (props != null && props.Length > 0) performAot = true;
                }

                // check for [fsProperty]
                if (!performAot) {
                    foreach (PropertyInfo p in t.GetProperties()) {
                        var props = p.GetCustomAttributes(typeof(fsPropertyAttribute), true);
                        if (props.Length > 0) {
                            performAot = true;
                            break;
                        }
                    }
                }

                if (performAot) {
                    string compilation = null;
                    if (fsAotCompilationManager.TryToPerformAotCompilation(t, out compilation)) {
                        Debug.Log("Performing AOT compilation for " + t);
                        string path = Path.Combine(OutputDirectory, "AotConverter_" + t.CSharpName(true, true) + ".cs");
                        File.WriteAllText(path, compilation);
                    } else {
                        Debug.Log("Failed AOT compilation for " + t.CSharpName(true));
                    }
                }
            }
        }
    }
}

Please note: If you change a model that has been AOT compiled, the model changes will not be reflected in serialization pipeline until you update the AOT compiled converter. Stale models are not currently detected.

Limitations

Full Serializer has minimal limitations, however, there are as follows:

• The WebPlayer build target requires all deserialized types to have a default constructor
• No multidimensional array support (this can be added with a custom converter, however)
• Delegates are not serialized (how? If you have any ideas, please let me know!)

Adding Full Serializer to my project

Source Based Import

Import the Source folder into your Unity project! You're good to go!

DLL Based Import

These instructions are easy to follow and will set you up with the DLLs for any version of Full Serializer.

• Download Full Serializer and unzip it to some directory.
• Navigate to the Build Files (DLL) folder.
• Open CommonData.csproj in your favorite text editor.
• Change the UnityInstallFolder value on line 6 to point to your Unity installation directory
  • On OSX this is likely /Applications/Unity/Editor
  • On Windows this is likely C:\Program Files (x86)\Unity\Editor
• Double click FullSerializer.sln to open up the solution
• Run a build-all (F6 in visual studio). Alternatively, you can right-click any of the three projects to build only one of them.
  • FullSerializer - NoUnity builds Full Serializer so that you can use it outside of Unity.
  • FullSerializer - Unity builds Full Serializer to a DLL
  • FullSerializer - Unity - WinRT builds Full Serializer with WinRT APIs (if you're targeting the Windows Store or the Windows Phone export platforms)
• You will find the DLLs inside of the Build folder. Please add them to your Unity project's Asset folder.

How do I run the tests?

To run automated tests, please also import Unity Test Tools into your project. Then you can run the NUnit tests via the standard unit test menu Unity Test Tools\Unit Tests\Run all unit tests.

Full Serializer also has a suite of runtime tests to ensure that various platform support actually works when deployed. You can run these tests by opening up the Testing/test_scene scene and hitting play.

Linker Options

If you're getting strange errors when exporting or running (like missing constructors), Unity might be stripping out Full Serializer methods. You can prevent this with a link.xml that looks something like

<linker>
    <assembly fullname="FullSerializer">
        <namespace fullname="FullSerializer" preserve="all"/>
        <namespace fullname="FullSerializer.Internal" preserve="all"/>
        <namespace fullname="FullSerializer.Internal.DirectConverters" preserve="all"/>
    </assembly>
</linker>

Using Full Serializer in an Asset Store Package

Feel free to use Full Serializer in your own asset store package. If you do so, please rename the Full Serializer namespace to something like MyPackage.FullSerializer so that there will be no conflict if there are multiple versions of Full Serializer installed.

NPM Support

This project can be installed directly into the Assets/packages/ folder from github using npm via:

npm init
npm install jacobdufault/fullserializer --save

License

Full Serializer is freely available under the MIT license. If you make any improvements, it would be greatly appreciated if you would submit a pull request with them (please match the existing code style).

fullserializer's Issues

Remove $type field

I noticed that the serializer adds a $type field to serialized objects. What would the quickest way to remove the $type field ? Is this something I can handle in my interface class ?

Populate Object / Clone Object

Populate Object / Clone Object.

This would be good additions. I did see ConstructInstance, cool.

[request] Support for serialization directly to streams

This will just involve changing how the JSON structure is emitted. Instead of directly using a StringBuilder, use a stream and then convert that stream to a string. Allow the user to specify the stream, and provide a default.

Parser throws an exception on empty input

Hello,

I've recently tested some stuff and found that fsJsonParser throws an exception if empty string was passed to it. I think this is not best behaviour, and we must have fsResult.Fail for that case instead of exception.

Better Converter Example

I do not understand how to use the json converters. Other converters are easy to understand. Take LITJson for instance. In LIT you manually you have access to a writer object with methods for writing each part of the object as json. Newtonsoft has a similar feature.

  public static void FilterExporter(MyObject value, JsonWriter writer)
        {
            writer.WriteObjectStart();

            writer.WritePropertyName("operator");
            writer.Write(value.op.ToString());

            if (!string.IsNullOrEmpty(value.item))
            {
                writer.WritePropertyName("item");
                writer.Write(value.item);
            }

            writer.WritePropertyName("value");
            WriteValue(value.value, writer);

            writer.WriteObjectEnd();
        }

Looking at the examples I was unable to discern how I could go about writing a custom filter for an arbitrary object.

Processors get invoked using the given storage/field type and not the actual object type

How to use AOT Converters

I'm not quite sure how to get this working on PS4 with in full-AOT mode.

I've marked all classes with [fsObject] and the aot-converter-classes are being generated successfuly, but I'm still getting errors when trying to serialize:

Attempting to JIT compile method '(wrapper delegate-invoke) :invoke_AppSettings__this___SaveData (SaveData)' while running with --aot-only.
http://gyazo.com/fbc2428cfe9b2b9d51c5efad9fbe7d14

Is there any other steps I should be doing? Do I need to manually register the converters with the serializer?

Question about references

I am a little confused about how deserialization works. Can you better explain what is happening here in the fsSerializer ?

// While object construction should technically be two-pass, we can do it in
// one pass because of how serialization happens. We traverse the serialization
// graph in the same order during serialization and deserialization, so the first
// time we encounter an object it'll always be the definition. Any times after that
// it will be a reference. Because of this, if we encounter a reference then we
 // will have *always* already encountered the definition for it.
if (IsObjectReference(data))
 {
               int refId = int.Parse(data.AsDictionary[ReferenceIdString].AsString);
               result = _references.GetReferenceObject(refId);
               return fsFailure.Success;
 }

Are we reusing objects for every deserialization of a type ?

fsConfig defaults to OptOut

The documentation in README.md states that FullSerializer's serialization rules will match what Unity's rules, and then it gives specific examples. I believe this behavior describes fsConfig.DefaultMemberSerialization = fsMemberSerialization.Default (see fsMetaType.CollectProperties).

However, fsConfig.DefaultMemberSerialization is automatically set to OptOut, not Default, which creates very different behavior. For example, undecorated private fields are serialized (see fsMetaType.CanSerializeField - it will pass optOut=true). This is very likely to lead to surprising serialization errors.

It's an easy change to make after integrating the source, but it seems odd that the default value of DefaultMemberSerialization would not be fsMemberSerialization.Default. The other option would be to document the behavior implied by fsMemberSerialization.OptOut.

Support all int values in serialization

fsResult.Fail in custom converters does not fail a de/serialization process

I use custom converter and return fsResult.Fail("message") if I can't serialize an object for some reason.
TrySerialize does not use this result and returns fsResult.Succeeded at the top level. The problem is in the fsResult.AddMessage() call instead of fsResult.Merge() in fsReflectedConverter. But maybe this is intented behaviour? If so, how can I handle failures in custom converters?

Classes need default constructor

class need default constructor in order to run on Android. MissingMethodException: Method not found: 'Default constructor not found...ctor() of FullSerializer.Internal.DirectConverters.AnimationCurve_DirectConverter'.

suppose this is not the only one

fsData.False incorrect definition

Hello,

fsData has incorrect definition for False:

    public readonly static fsData True = new fsData(true);
    public readonly static fsData False = new fsData(true); // must be fsData(false)

Problem deserializing JSON object as a Dictionary

It seems fullserializer does not serialize JSON dictionaries? Running the code below I get the following RawMessage: fsIEnumerableConverter expected Array but got Object in {"someKey":"someValue"}

public class SomeObject {
  public Dictionary<string, string> someField;
}

fsData data = fsJsonParser.Parse("{\"someField\": {\"someKey\": \"someValue\"}}");
object deserialized = null;
var serializer = new fsSerializer ();
serializer.TryDeserialize(data, typeof(someObject), ref deserialized).AssertSuccessWithoutWarnings();

Corrupted JSON on some locales. Double.ToString() returns ',' delimeter

When use Unity '-batchmode' from Termnal on Mac OS X with RU locale, Double.ToString() returns comma delimeter instead of dot.
As a result, we get JSON like this:

{ "value":0,5.5 }

Possible fix is to add InvariantCulture to ToString calls in fsJsonPrinter:

if (Double.IsInfinity(d) || Double.IsNaN(d)) return d.ToString(System.Globalization.CultureInfo.InvariantCulture);
String doubledString = d.ToString(System.Globalization.CultureInfo.InvariantCulture);

Add parsing support for /* */ style comments

JsonStrict Annotation

For complex objects I like to exclude all properties by default and then manually include properties that I wish to serialize. I do this by including a JsonStrict annotation.

/// <summary>
/// Explicitly marks a class to ignore all properties for serialization
/// except those that contain a JsonProperty attribute
/// </summary>
[AttributeUsage(AttributeTargets.Class)]
public sealed class JsonStrict : Attribute
{
}

I then modified fsMetaType

    private static List<fsMetaProperty> CollectProperties(Type reflectedType)
    {
        var properties = new List<fsMetaProperty>();

        var isStrict = Attribute.IsDefined(reflectedType, typeof(NonSerializedAttribute))
            || Attribute.IsDefined(reflectedType, typeof(JsonStrict));

        MemberInfo[] members = reflectedType.GetMembers(PropertyLookupFlags);
        foreach (MemberInfo member in members)
        {
            // We don't serialize members annotated with [fsIgnore] or [NonSerialized].
            if (Attribute.IsDefined(member, typeof(JsonIgnore)) ||
                Attribute.IsDefined(member, typeof(NonSerializedAttribute)))
            {
                continue;
            }

            if (isStrict && !Attribute.IsDefined(member, typeof(JsonProperty)))
            {
                continue;
            }


            PropertyInfo property = member as PropertyInfo;
            FieldInfo field = member as FieldInfo;

            if (property != null)
            {
                if (CanSerializeProperty(property, members))
                {
                    properties.Add(new fsMetaProperty(property));
                }
            }

            else if (field != null)
            {
                if (CanSerializeField(field))
                {
                    properties.Add(new fsMetaProperty(field));
                }
            }
        }

        return properties;
    }

Might want to include it as a contribution in the next push

fsObjectProcessor - OnBeforeDeserialize and "$type"

Hello Jacob,

After updating to the latest FS, I am encountering an issue with OnBeforeDeserialize of the fsObjectProcessor and a trick I used that modifies the "$type" json entry of the object proccessed.
As it seems the json deserialized is not being modified at all now. This used to work great a couble of versions back (about 3 weeks ago).
Here is the critical part of the processor code:

    public override void OnBeforeDeserialize(Type storageType, ref fsData data){

        if (data.IsNull)
            return;

        var json = data.AsDictionary;

        if (json.ContainsKey("$type")){

            var serializedType = ReflectionTools.GetType( json["$type"].AsString );
            if (serializedType == null){
                json["$type"] = new fsData( typeof(MissingNode).FullName );
            }
        }
    }

Thanks a lot!
Gavalakis Vaggelis

Do not fail to deserialize on minor errors

This requires a rework of fsFailure - it needs to handle a warning state.

Crash on WP8 in fsMetaType.Create().

Hi!
I am using NodeCanvas v2, a project that uses fullserializer as you are probably aware. I am experiencing a crash on WP8 in fsMetaType, beginning line 292.
try { return Activator.CreateInstance(ReflectedType, /*nonPublic:*/ true); }

Same crash was reported elsewhere on WP8 in an unrelated project with a google search http://opsive.com/forum/index.php/topic,635.15.html

If you use instead the type Constructor (https://msdn.microsoft.com/en-us/library/wccyzw83%28v=vs.110%29.aspx) then it doesn't crash.

I am not sure what effect this has on platforms other than WP8.

If you would like I can fork, open a PR and make the change.

Thanks,
PJ

Property support

I turned off the serializing of private fields and noticed the serializer does not support properties. uugh.

[fsIgnore]
private string _portraitArg;
public string PortraitArg
{
    get { return _portraitArg; }
    set
    {
        if (_portraitArg == value) return;
        _portraitArg = value;
        NotifyProperty("PortraitArg", value);
    }
}

Serialize flags enums using names

Serialize float without convertion to double

When serializing float values, fsPrimitiveConverter converts it to double. This adds some very small floating point jitter which leads to saving values using much more space in Json:
0.800000011920929
0.600000023841858
0.300000011920929
instead of normal 0.8, 0.6, etc.

When deserializing this value to float, all this noise won't fit to float and it's actually useless.

You can try it out yorself and see different result with simple code:

Debug.Log(0.1f); // prints 0.1
Debug.Log((double)0.1f); // prints 0.100000001490116
Debug.Log(0.1); // prints 0.1

If serializable type is float, convertion to double must not be preformed.

Serialize String as Key Dictionaries Cleanly

By default fullserializer will not deserialize something like this json:

{
  "controls": {
    "Action1": "Roll",
    "Action2": "Throw Knife",
    "Action3": "Smoke Bomb",
    "Action4": "4"
  }
}

into a dictionary for a class like this:

public class CharacterDesc {
    [fsProperty("controls")]
    public Dictionary<string, string> Controls;
}

I implemented myself, but I'm too lazy to rearrange my code and make a pull request:
https://gist.github.com/izackp/39e1e150b41fced0680a

Callback support for notifications when serializing/deserializing

Why? Because fsObjectProcessor does not have access to non-public members of the type.

Unable to parse 2015-04-27T15:54:02.763033Z into a DateTime

Hello,
I am using FullSerializer in Unity, and for certian DateTimes, I get fsResult from TryDeserialize with HasWarnings = true, and a formatted message like the one above.
Note that there are no such problems for DateTimes with 7 deciseconds symbols behind the last Dot.

For example, 2015-04-27T15:54:02.7630331Z works fine.

The result come from the default Json.NET serializer with Microsoft Web Api.
Any idea if this is a bug?

And btw, fsResult.HasErrors method should first check if _message != null?

Thanks

Windows Store App Support

I am getting errors when building for WindowsStoreApp. From the looks of it, the dll is inapropreatly trying to access System.Threading.Thread. Here are 2 of the exceptions, there are 231 in total.

Reference Rewriter: Error: type `System.Threading.Thread` doesn't exist in target framework. It is referenced from FullSerializer - Unity - WinRT.dll at System.Collections.Generic.IEnumerator`1<System.Object> FullSerializer.Internal.fsIDictionaryAdapter/<Iterate>d__1::System.Collections.Generic.IEnumerable<System.Object>.GetEnumerator().

UnityEngine.Debug:LogError(Object)
PostProcessMetroPlayer:RunReferenceRewriter(WSASDK, String, String, String, String) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:334)
PostProcessMetroPlayer:PostProcessSDKSpecific(WSASDK, BuildTarget, BuildOptions, String, String, ProjectImages, LibraryCollection) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:883)
PostProcessMetroPlayer:PostProcess(BuildTarget, BuildOptions, String, String, String, String) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:597)
UnityEditor.HostView:OnGUI()

Reference Rewriter: Error: type `System.AppDomain` doesn't exist in target framework. It is referenced from FullSerializer - Unity - WinRT.dll at System.Type FullSerializer.Internal.fsTypeLookup::GetType(System.String).

UnityEngine.Debug:LogError(Object)
PostProcessMetroPlayer:RunReferenceRewriter(WSASDK, String, String, String, String) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:334)
PostProcessMetroPlayer:PostProcessSDKSpecific(WSASDK, BuildTarget, BuildOptions, String, String, ProjectImages, LibraryCollection) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:883)
PostProcessMetroPlayer:PostProcess(BuildTarget, BuildOptions, String, String, String, String) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:597)
UnityEditor.HostView:OnGUI()

WinRT String Comparer

Found two WinRT errors

Line 80
https://github.com/NVentimiglia/fullserializer/blob/master/Source/fsData.cs

Line 362
https://github.com/NVentimiglia/fullserializer/blob/master/Source/fsJsonParser.cs

Anyway to override default serialization behavior?

First of all thanks for your great work! fullserializer is nicely crafted and tremendously useful.

I want to know if there's is a way to custom the default behavior. Say I want the serialization to include private fields. Currently I'm changing CanSerializeField() directly.

Is there a way I can override its behavior? Thanks!

WebGL support?

I've tried using FullSerializer (via source installation) with the Unity 5 release candidate (f1) WebGL exporter. When I try and serialize an object using StringSerializationAPI I get the following error:

TypeLoadExeption: A type load exception has occurred.

Looking at the stack trace, the specific line in the .NET framework is System.Type.MakeGenericType(System.Type[] typeArguments) and the lowest line in FullSerializer is FullSerializer.Internal.fsMetaProperty.Read(System.Object context)

Am I doing something wrong or is the WebGL exporter not yet supported?

better Unity AOT example

The section in the readme regarding compiling AOT converters for unity builds is unclear. I attempted making a menu option to generate my converters using the provided method (which is slightly out of date) inside an editor script. That method on its own does nothing without types being added to the AvailableAotCompilations list but the readme provides no information on how best to populate that list. Any advice would be appreciated.

IdConverter reference issue

Just a heads up: the IdConverter used as an example to explain converters only works if one and the same Id object is only serialized once. If it appears more than once in the graph, then it will be referenced via $ref but there won't be any corresponding $id.

I think the converters should get a chance to opt out of this referencing, especially if you use a converter to serialize an immutable object by its ID only, in which case referencing is a moot point anyway.

It's possible for now to always use dictionaries which will work.

Customizing serialization at the property/field level

I've looked over the source of FullSerializer but I'm struggling to find a way to customize serialization at the field/property level. My use case is properties that should only be serialized if they are NOT at the default value. For example, skipping serialization of Unity Vector3 if vector == Vector3.zero. Classes that take responsibility for default value initialization can benefit from not having to write out verbose JSON for default values.

FullSerializer converters are specified at the class level so I don't see a way to define a converter just for a specific Vector property using an attribute.

An attribute specified converter at the property level would be ideal but something similar to Json.NET's 'ShouldSerialize' would work too. http://www.newtonsoft.com/json/help/html/ConditionalProperties.htm

Any ideas would be helpful. I don't mind getting my hands dirty. Thanks.

Errors on compile in Unity 4.6.3

I am attempting to use fullserializer for my unity 4.6.3 project. However, I get the following errors when attempting to build.
http://puu.sh/hpHKW/b277078a90.png

fsEnumConverter.cs(39,52) Type 'StringBuilder' does not contain a definition for 'Append'
fsEnumConverter.cs(41,32) Type 'StringBuilder' does not contain a definition for 'Append'
fsJsonParser.cs(251,85): Type 'StringBuilder' does not contain a constructor that takes '1' argument.

This appears to be due to Unity using an older equivalent version of .Net, one without the extra methods and constructors for StringBuilder class. I am wondering how you managed to work around this problem?

fsDirectConverter support arrays

It would be a nice feature to have! fsDirectConverter offers perf improvements during serializer creation, so it would be nice to use an array with them.

Maybe fsDirectObjectConverter, fsDirectArrayConverter, and fsDirectConverter? The fsDirect*Converter classes would then contain helper methods.

Remember to update AOT generation.

Deserialization not invoking processors when actual type != storage type

Steps to reproduce:

1. Instantiate a ClassA object which implements IInterface, put it in an IInterface object
2. Serialize the IInterface object (which is actually ClassA), processors of ClassA gets called
3. Try to deserialize output from step 2 into an IInterface object, processors of ClassA does not get called
   Verified that serialized data does contain type information, which is ClassA.

Thanks for reading!

Partial deserialization for arrays

Lists work, but not arrays

Two objects of different types are serialized as references

If the equals method is overridden such that they are treated as equal even though their types are different, then the cyclic detector will treat them as the same reference (because it uses a dictionary internally). fsCyclicReferenceManager probably needs a unique dictionary per type.

More options for default properties serialization

I believe the Default serialization behaviour configuration for properties is sometimes lacking. There are two cases which I think should be configurable via fsConfig.cs:

1. Boolean to toggle serialization of properties with no public set method (current: true)
2. Boolean to toggle serialization of non-auto properties (current: false)

Lines 145-150 in fsMetaType.cs are currently:

// If it's an auto-property and it has either a public get or a public set method,
// then we serialize it
if (IsAutoProperty(property, members) &&
    (publicGetMethod != null || publicSetMethod != null)) {
    return true;
}

I would suggest they become:

// (Some updated comment)
if ((fsConfig.SerializeNonAutoProperties || IsAutoProperty(property, members)) &&
    (publicGetMethod != null && (fsConfig.SerializeNonPublicSetProperties || publicSetMethod != null))) {
    return true;
}

With the added configuration options in fsConfig.cs:

/// <summary>
/// Whether the default serialization behaviour should include non-auto properties.
/// </summary>
public static bool SerializeNonAutoProperties = false;

/// <summary>
/// Whether the default serialization behaviour should include properties with non-public setters.
/// </summary>
public static bool SerializeNonPublicSetProperties = true;

By default, the behaviour would stay unchanged for backwards compatibility, but allow for more customization by the user.

Support for Observableproperties

I am officially using FullSerializer for the Foundation Databinding toolkit

https://github.com/NVentimiglia/Unity3d-Databinding-Mvvm-Mvc

I noticed that observable properties are not automatically serialized. I had to use the [fsProperty] annotation to get them to work. Support for the following (without the fsProperty) would be awesome.

 private int _networkId =-1;
        /// <summary>
        /// Network Connection Id
        /// </summary>
        [fsProperty("NetworkId")]
        public int NetworkId
        {
            get { return _networkId; }
            set
            {
                if (_networkId == value)
                    return;
                _networkId = value;
                NotifyProperty("NetworkId", value);
            }
        }

Add `fsResult.HasWarnings`

See #35

using FullSerializer.Internal;

fsTypeExtensions.cs needs to reference FullSerializer.Internal;

please add :

     using FullSerializer.Internal;

https://github.com/jacobdufault/fullserializer/blob/master/Source/Internal/fsTypeExtensions.cs

Stack<T> serialization

Releases

Hi, could we have a simple release file to download (preferably a unitypackage)? Also a Unity Asset store page, why not?

It will help the lazy people and spread the word!

Compile errors on WSA.

Hi,
Im using FullSerializer in NodeCanvas and i'm getting compile errors in fsTypeExtensions.cs. They can probably be fixed easily enough with a NETFX_CORE defines.

Here are the errors. Thanks!
PJ

Assets\NodeCanvas\Framework_ParadoxNotion (shared)\Runtime\Serialization\FullSerializer\Internal\fsTypeExtensions.cs(41,60): error CS1061: 'System.Type' does not contain a definition for 'GetGenericArguments' and no extension method 'GetGenericArguments' accepting a first argument of type 'System.Type' could be found (are you missing a using directive or an assembly reference?)
Assets\NodeCanvas\Framework_ParadoxNotion (shared)\Runtime\Serialization\FullSerializer\Internal\fsTypeExtensions.cs(51,40): error CS1061: 'System.Type' does not contain a definition for 'GetGenericArguments' and no extension method 'GetGenericArguments' accepting a first argument of type 'System.Type' could be found (are you missing a using directive or an assembly reference?)
Assets\NodeCanvas\Framework_ParadoxNotion (shared)\Runtime\Serialization\FullSerializer\Internal\fsTypeExtensions.cs(52,81): error CS1061: 'System.Type' does not contain a definition for 'GetGenericArguments' and no extension method 'GetGenericArguments' accepting a first argument of type 'System.Type' could be found (are you missing a using directive or an assembly reference?)

Can't serialize on iOS

There is probably a way around this, but I can't see it in your documentation.

I get the following error when using with iOS:

ExecutionEngineException: Attempting to JIT compile method '(wrapper delegate-invoke) System.Reflection.MonoProperty/Getter`2<Job, Address>:invoke_Address__this___Job (Job)' while running with --aot-only.

This is the same error I got with the XML serialization (the reason I tried using this one).

$content and $type

Serializing of collections results in a funny schema. For instance I serialized an array of strings, here is my result.

"{
\"$content\":[\"0\",\"1\",\"2\",\"3\",\"4\",\"5\",\"6\",\"7\",\"8\",\"9\"],
\"$type\":\"System.String[]
\"}"

This output is not up to standards and does not work with other json serializers (specifically the ones running on my webserver). Is there any way I could turn off this functionality ? Specifically it should look like this.

[\"0\",\"1\",\"2\",\"3\",\"4\",\"5\",\"6\",\"7\",\"8\",\"9\"]

Config for case-insensitive deserialization

• Change fsJsonParser.cs around line 361 to var result = new Dictionary<string, fsData>(StringComparer.InvariantCultureIgnoreCase)
• Add a setting to enable this behavior

ability to convert json keys to different format without manual addition of fsProperty attribute on all properties

On our project we like have our json look like this:

{ "propertyKey" : "value" }

but the corresponding property in our code would be:

public string PropertyKey { get; set; }

or a more complicated example would be "property_key" => "PropertyKey"

It would be nice to be able to have a global rule for parsing that handles this rather than tons of fsProperty attributes. Newtonsoft's json.net handles this using something they call Contract Resolvers:
http://www.newtonsoft.com/json/help/html/ContractResolver.htm

I can't find an equivalent in FullSerializer. Am I missing something?

BSON support

I have a list of abstract data class i would like to serialize to a file.
public List level = new List ();

I tried doing it with the Unity serializer. I am able to write and read from a file but when i read back, the data is incorrect so I thought I would try full serializer

I was able to get to this point but not sure how to send to a file and then deserialize

// add stuff to level

    object classdata= level;
    fsData data;
    fsSerializer _serializer = new fsSerializer();
    _serializer.TrySerialize(classdata, out data).AssertSuccessWithoutWarnings();