jacobdufault/fullserializer

A robust JSON serialization framework that just works with support for all major Unity export platforms.

fullserializer's Issues

WebGL support?

I've tried using FullSerializer (via source installation) with the Unity 5 release candidate (f1) WebGL exporter. When I try and serialize an object using StringSerializationAPI I get the following error:

TypeLoadExeption: A type load exception has occurred.

Looking at the stack trace, the specific line in the .NET framework is System.Type.MakeGenericType(System.Type[] typeArguments) and the lowest line in FullSerializer is FullSerializer.Internal.fsMetaProperty.Read(System.Object context)

Am I doing something wrong or is the WebGL exporter not yet supported?

Deserialization not invoking processors when actual type != storage type

Steps to reproduce:

1. Instantiate a ClassA object which implements IInterface, put it in an IInterface object
2. Serialize the IInterface object (which is actually ClassA), processors of ClassA gets called
3. Try to deserialize output from step 2 into an IInterface object, processors of ClassA does not get called
   Verified that serialized data does contain type information, which is ClassA.

Thanks for reading!

Do not fail to deserialize on minor errors

This requires a rework of fsFailure - it needs to handle a warning state.

Property support

I turned off the serializing of private fields and noticed the serializer does not support properties. uugh.

[fsIgnore]
private string _portraitArg;
public string PortraitArg
{
    get { return _portraitArg; }
    set
    {
        if (_portraitArg == value) return;
        _portraitArg = value;
        NotifyProperty("PortraitArg", value);
    }
}

How to use AOT Converters

I'm not quite sure how to get this working on PS4 with in full-AOT mode.

I've marked all classes with [fsObject] and the aot-converter-classes are being generated successfuly, but I'm still getting errors when trying to serialize:

Attempting to JIT compile method '(wrapper delegate-invoke) :invoke_AppSettings__this___SaveData (SaveData)' while running with --aot-only.
http://gyazo.com/fbc2428cfe9b2b9d51c5efad9fbe7d14

Is there any other steps I should be doing? Do I need to manually register the converters with the serializer?

Remove $type field

I noticed that the serializer adds a $type field to serialized objects. What would the quickest way to remove the $type field ? Is this something I can handle in my interface class ?

$content and $type

Serializing of collections results in a funny schema. For instance I serialized an array of strings, here is my result.

"{
\"$content\":[\"0\",\"1\",\"2\",\"3\",\"4\",\"5\",\"6\",\"7\",\"8\",\"9\"],
\"$type\":\"System.String[]
\"}"

This output is not up to standards and does not work with other json serializers (specifically the ones running on my webserver). Is there any way I could turn off this functionality ? Specifically it should look like this.

[\"0\",\"1\",\"2\",\"3\",\"4\",\"5\",\"6\",\"7\",\"8\",\"9\"]

Two objects of different types are serialized as references

If the equals method is overridden such that they are treated as equal even though their types are different, then the cyclic detector will treat them as the same reference (because it uses a dictionary internally). fsCyclicReferenceManager probably needs a unique dictionary per type.

Stack<T> serialization

More options for default properties serialization

I believe the Default serialization behaviour configuration for properties is sometimes lacking. There are two cases which I think should be configurable via fsConfig.cs:

1. Boolean to toggle serialization of properties with no public set method (current: true)
2. Boolean to toggle serialization of non-auto properties (current: false)

Lines 145-150 in fsMetaType.cs are currently:

// If it's an auto-property and it has either a public get or a public set method,
// then we serialize it
if (IsAutoProperty(property, members) &&
    (publicGetMethod != null || publicSetMethod != null)) {
    return true;
}

I would suggest they become:

// (Some updated comment)
if ((fsConfig.SerializeNonAutoProperties || IsAutoProperty(property, members)) &&
    (publicGetMethod != null && (fsConfig.SerializeNonPublicSetProperties || publicSetMethod != null))) {
    return true;
}

With the added configuration options in fsConfig.cs:

/// <summary>
/// Whether the default serialization behaviour should include non-auto properties.
/// </summary>
public static bool SerializeNonAutoProperties = false;

/// <summary>
/// Whether the default serialization behaviour should include properties with non-public setters.
/// </summary>
public static bool SerializeNonPublicSetProperties = true;

By default, the behaviour would stay unchanged for backwards compatibility, but allow for more customization by the user.

[request] Support for serialization directly to streams

This will just involve changing how the JSON structure is emitted. Instead of directly using a StringBuilder, use a stream and then convert that stream to a string. Allow the user to specify the stream, and provide a default.

Support for Observableproperties

I am officially using FullSerializer for the Foundation Databinding toolkit

https://github.com/NVentimiglia/Unity3d-Databinding-Mvvm-Mvc

I noticed that observable properties are not automatically serialized. I had to use the [fsProperty] annotation to get them to work. Support for the following (without the fsProperty) would be awesome.

 private int _networkId =-1;
        /// <summary>
        /// Network Connection Id
        /// </summary>
        [fsProperty("NetworkId")]
        public int NetworkId
        {
            get { return _networkId; }
            set
            {
                if (_networkId == value)
                    return;
                _networkId = value;
                NotifyProperty("NetworkId", value);
            }
        }

fsConfig defaults to OptOut

The documentation in README.md states that FullSerializer's serialization rules will match what Unity's rules, and then it gives specific examples. I believe this behavior describes fsConfig.DefaultMemberSerialization = fsMemberSerialization.Default (see fsMetaType.CollectProperties).

However, fsConfig.DefaultMemberSerialization is automatically set to OptOut, not Default, which creates very different behavior. For example, undecorated private fields are serialized (see fsMetaType.CanSerializeField - it will pass optOut=true). This is very likely to lead to surprising serialization errors.

It's an easy change to make after integrating the source, but it seems odd that the default value of DefaultMemberSerialization would not be fsMemberSerialization.Default. The other option would be to document the behavior implied by fsMemberSerialization.OptOut.

Better Converter Example

I do not understand how to use the json converters. Other converters are easy to understand. Take LITJson for instance. In LIT you manually you have access to a writer object with methods for writing each part of the object as json. Newtonsoft has a similar feature.

  public static void FilterExporter(MyObject value, JsonWriter writer)
        {
            writer.WriteObjectStart();

            writer.WritePropertyName("operator");
            writer.Write(value.op.ToString());

            if (!string.IsNullOrEmpty(value.item))
            {
                writer.WritePropertyName("item");
                writer.Write(value.item);
            }

            writer.WritePropertyName("value");
            WriteValue(value.value, writer);

            writer.WriteObjectEnd();
        }

Looking at the examples I was unable to discern how I could go about writing a custom filter for an arbitrary object.

BSON support

I have a list of abstract data class i would like to serialize to a file.
public List level = new List ();

I tried doing it with the Unity serializer. I am able to write and read from a file but when i read back, the data is incorrect so I thought I would try full serializer

I was able to get to this point but not sure how to send to a file and then deserialize

// add stuff to level

    object classdata= level;
    fsData data;
    fsSerializer _serializer = new fsSerializer();
    _serializer.TrySerialize(classdata, out data).AssertSuccessWithoutWarnings();

Question about references

I am a little confused about how deserialization works. Can you better explain what is happening here in the fsSerializer ?

// While object construction should technically be two-pass, we can do it in
// one pass because of how serialization happens. We traverse the serialization
// graph in the same order during serialization and deserialization, so the first
// time we encounter an object it'll always be the definition. Any times after that
// it will be a reference. Because of this, if we encounter a reference then we
 // will have *always* already encountered the definition for it.
if (IsObjectReference(data))
 {
               int refId = int.Parse(data.AsDictionary[ReferenceIdString].AsString);
               result = _references.GetReferenceObject(refId);
               return fsFailure.Success;
 }

Are we reusing objects for every deserialization of a type ?

fsDirectConverter support arrays

It would be a nice feature to have! fsDirectConverter offers perf improvements during serializer creation, so it would be nice to use an array with them.

Maybe fsDirectObjectConverter, fsDirectArrayConverter, and fsDirectConverter? The fsDirect*Converter classes would then contain helper methods.

Remember to update AOT generation.

fsResult.Fail in custom converters does not fail a de/serialization process

I use custom converter and return fsResult.Fail("message") if I can't serialize an object for some reason.
TrySerialize does not use this result and returns fsResult.Succeeded at the top level. The problem is in the fsResult.AddMessage() call instead of fsResult.Merge() in fsReflectedConverter. But maybe this is intented behaviour? If so, how can I handle failures in custom converters?

Partial deserialization for arrays

Lists work, but not arrays

Serialize float without convertion to double

When serializing float values, fsPrimitiveConverter converts it to double. This adds some very small floating point jitter which leads to saving values using much more space in Json:
0.800000011920929
0.600000023841858
0.300000011920929
instead of normal 0.8, 0.6, etc.

When deserializing this value to float, all this noise won't fit to float and it's actually useless.

You can try it out yorself and see different result with simple code:

Debug.Log(0.1f); // prints 0.1
Debug.Log((double)0.1f); // prints 0.100000001490116
Debug.Log(0.1); // prints 0.1

If serializable type is float, convertion to double must not be preformed.

Add parsing support for /* */ style comments

Config for case-insensitive deserialization

• Change fsJsonParser.cs around line 361 to var result = new Dictionary<string, fsData>(StringComparer.InvariantCultureIgnoreCase)
• Add a setting to enable this behavior

IdConverter reference issue

Just a heads up: the IdConverter used as an example to explain converters only works if one and the same Id object is only serialized once. If it appears more than once in the graph, then it will be referenced via $ref but there won't be any corresponding $id.

I think the converters should get a chance to opt out of this referencing, especially if you use a converter to serialize an immutable object by its ID only, in which case referencing is a moot point anyway.

It's possible for now to always use dictionaries which will work.

Errors on compile in Unity 4.6.3

I am attempting to use fullserializer for my unity 4.6.3 project. However, I get the following errors when attempting to build.
http://puu.sh/hpHKW/b277078a90.png

fsEnumConverter.cs(39,52) Type 'StringBuilder' does not contain a definition for 'Append'
fsEnumConverter.cs(41,32) Type 'StringBuilder' does not contain a definition for 'Append'
fsJsonParser.cs(251,85): Type 'StringBuilder' does not contain a constructor that takes '1' argument.

This appears to be due to Unity using an older equivalent version of .Net, one without the extra methods and constructors for StringBuilder class. I am wondering how you managed to work around this problem?

Serialize flags enums using names

Problem deserializing JSON object as a Dictionary

It seems fullserializer does not serialize JSON dictionaries? Running the code below I get the following RawMessage: fsIEnumerableConverter expected Array but got Object in {"someKey":"someValue"}

public class SomeObject {
  public Dictionary<string, string> someField;
}

fsData data = fsJsonParser.Parse("{\"someField\": {\"someKey\": \"someValue\"}}");
object deserialized = null;
var serializer = new fsSerializer ();
serializer.TryDeserialize(data, typeof(someObject), ref deserialized).AssertSuccessWithoutWarnings();

fsObjectProcessor - OnBeforeDeserialize and "$type"

Hello Jacob,

After updating to the latest FS, I am encountering an issue with OnBeforeDeserialize of the fsObjectProcessor and a trick I used that modifies the "$type" json entry of the object proccessed.
As it seems the json deserialized is not being modified at all now. This used to work great a couble of versions back (about 3 weeks ago).
Here is the critical part of the processor code:

    public override void OnBeforeDeserialize(Type storageType, ref fsData data){

        if (data.IsNull)
            return;

        var json = data.AsDictionary;

        if (json.ContainsKey("$type")){

            var serializedType = ReflectionTools.GetType( json["$type"].AsString );
            if (serializedType == null){
                json["$type"] = new fsData( typeof(MissingNode).FullName );
            }
        }
    }

Thanks a lot!
Gavalakis Vaggelis

better Unity AOT example

The section in the readme regarding compiling AOT converters for unity builds is unclear. I attempted making a menu option to generate my converters using the provided method (which is slightly out of date) inside an editor script. That method on its own does nothing without types being added to the AvailableAotCompilations list but the readme provides no information on how best to populate that list. Any advice would be appreciated.

Populate Object / Clone Object

Populate Object / Clone Object.

This would be good additions. I did see ConstructInstance, cool.

Corrupted JSON on some locales. Double.ToString() returns ',' delimeter

When use Unity '-batchmode' from Termnal on Mac OS X with RU locale, Double.ToString() returns comma delimeter instead of dot.
As a result, we get JSON like this:

{ "value":0,5.5 }

Possible fix is to add InvariantCulture to ToString calls in fsJsonPrinter:

if (Double.IsInfinity(d) || Double.IsNaN(d)) return d.ToString(System.Globalization.CultureInfo.InvariantCulture);
String doubledString = d.ToString(System.Globalization.CultureInfo.InvariantCulture);

Unable to parse 2015-04-27T15:54:02.763033Z into a DateTime

Hello,
I am using FullSerializer in Unity, and for certian DateTimes, I get fsResult from TryDeserialize with HasWarnings = true, and a formatted message like the one above.
Note that there are no such problems for DateTimes with 7 deciseconds symbols behind the last Dot.

For example, 2015-04-27T15:54:02.7630331Z works fine.

The result come from the default Json.NET serializer with Microsoft Web Api.
Any idea if this is a bug?

And btw, fsResult.HasErrors method should first check if _message != null?

Thanks

Add `fsResult.HasWarnings`

See #35

fsData.False incorrect definition

Hello,

fsData has incorrect definition for False:

    public readonly static fsData True = new fsData(true);
    public readonly static fsData False = new fsData(true); // must be fsData(false)

Classes need default constructor

class need default constructor in order to run on Android. MissingMethodException: Method not found: 'Default constructor not found...ctor() of FullSerializer.Internal.DirectConverters.AnimationCurve_DirectConverter'.

suppose this is not the only one

using FullSerializer.Internal;

fsTypeExtensions.cs needs to reference FullSerializer.Internal;

please add :

     using FullSerializer.Internal;

https://github.com/jacobdufault/fullserializer/blob/master/Source/Internal/fsTypeExtensions.cs

Crash on WP8 in fsMetaType.Create().

Hi!
I am using NodeCanvas v2, a project that uses fullserializer as you are probably aware. I am experiencing a crash on WP8 in fsMetaType, beginning line 292.
try { return Activator.CreateInstance(ReflectedType, /*nonPublic:*/ true); }

Same crash was reported elsewhere on WP8 in an unrelated project with a google search http://opsive.com/forum/index.php/topic,635.15.html

If you use instead the type Constructor (https://msdn.microsoft.com/en-us/library/wccyzw83%28v=vs.110%29.aspx) then it doesn't crash.

I am not sure what effect this has on platforms other than WP8.

If you would like I can fork, open a PR and make the change.

Thanks,
PJ

Releases

Hi, could we have a simple release file to download (preferably a unitypackage)? Also a Unity Asset store page, why not?

It will help the lazy people and spread the word!

Can't serialize on iOS

There is probably a way around this, but I can't see it in your documentation.

I get the following error when using with iOS:

ExecutionEngineException: Attempting to JIT compile method '(wrapper delegate-invoke) System.Reflection.MonoProperty/Getter`2<Job, Address>:invoke_Address__this___Job (Job)' while running with --aot-only.

This is the same error I got with the XML serialization (the reason I tried using this one).

Windows Store App Support

I am getting errors when building for WindowsStoreApp. From the looks of it, the dll is inapropreatly trying to access System.Threading.Thread. Here are 2 of the exceptions, there are 231 in total.

Reference Rewriter: Error: type `System.Threading.Thread` doesn't exist in target framework. It is referenced from FullSerializer - Unity - WinRT.dll at System.Collections.Generic.IEnumerator`1<System.Object> FullSerializer.Internal.fsIDictionaryAdapter/<Iterate>d__1::System.Collections.Generic.IEnumerable<System.Object>.GetEnumerator().

UnityEngine.Debug:LogError(Object)
PostProcessMetroPlayer:RunReferenceRewriter(WSASDK, String, String, String, String) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:334)
PostProcessMetroPlayer:PostProcessSDKSpecific(WSASDK, BuildTarget, BuildOptions, String, String, ProjectImages, LibraryCollection) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:883)
PostProcessMetroPlayer:PostProcess(BuildTarget, BuildOptions, String, String, String, String) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:597)
UnityEditor.HostView:OnGUI()

Reference Rewriter: Error: type `System.AppDomain` doesn't exist in target framework. It is referenced from FullSerializer - Unity - WinRT.dll at System.Type FullSerializer.Internal.fsTypeLookup::GetType(System.String).

UnityEngine.Debug:LogError(Object)
PostProcessMetroPlayer:RunReferenceRewriter(WSASDK, String, String, String, String) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:334)
PostProcessMetroPlayer:PostProcessSDKSpecific(WSASDK, BuildTarget, BuildOptions, String, String, ProjectImages, LibraryCollection) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:883)
PostProcessMetroPlayer:PostProcess(BuildTarget, BuildOptions, String, String, String, String) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:597)
UnityEditor.HostView:OnGUI()

JsonStrict Annotation

For complex objects I like to exclude all properties by default and then manually include properties that I wish to serialize. I do this by including a JsonStrict annotation.

/// <summary>
/// Explicitly marks a class to ignore all properties for serialization
/// except those that contain a JsonProperty attribute
/// </summary>
[AttributeUsage(AttributeTargets.Class)]
public sealed class JsonStrict : Attribute
{
}

I then modified fsMetaType

    private static List<fsMetaProperty> CollectProperties(Type reflectedType)
    {
        var properties = new List<fsMetaProperty>();

        var isStrict = Attribute.IsDefined(reflectedType, typeof(NonSerializedAttribute))
            || Attribute.IsDefined(reflectedType, typeof(JsonStrict));

        MemberInfo[] members = reflectedType.GetMembers(PropertyLookupFlags);
        foreach (MemberInfo member in members)
        {
            // We don't serialize members annotated with [fsIgnore] or [NonSerialized].
            if (Attribute.IsDefined(member, typeof(JsonIgnore)) ||
                Attribute.IsDefined(member, typeof(NonSerializedAttribute)))
            {
                continue;
            }

            if (isStrict && !Attribute.IsDefined(member, typeof(JsonProperty)))
            {
                continue;
            }


            PropertyInfo property = member as PropertyInfo;
            FieldInfo field = member as FieldInfo;

            if (property != null)
            {
                if (CanSerializeProperty(property, members))
                {
                    properties.Add(new fsMetaProperty(property));
                }
            }

            else if (field != null)
            {
                if (CanSerializeField(field))
                {
                    properties.Add(new fsMetaProperty(field));
                }
            }
        }

        return properties;
    }

Might want to include it as a contribution in the next push

Customizing serialization at the property/field level

I've looked over the source of FullSerializer but I'm struggling to find a way to customize serialization at the field/property level. My use case is properties that should only be serialized if they are NOT at the default value. For example, skipping serialization of Unity Vector3 if vector == Vector3.zero. Classes that take responsibility for default value initialization can benefit from not having to write out verbose JSON for default values.

FullSerializer converters are specified at the class level so I don't see a way to define a converter just for a specific Vector property using an attribute.

An attribute specified converter at the property level would be ideal but something similar to Json.NET's 'ShouldSerialize' would work too. http://www.newtonsoft.com/json/help/html/ConditionalProperties.htm

Any ideas would be helpful. I don't mind getting my hands dirty. Thanks.

Serialize String as Key Dictionaries Cleanly

By default fullserializer will not deserialize something like this json:

{
  "controls": {
    "Action1": "Roll",
    "Action2": "Throw Knife",
    "Action3": "Smoke Bomb",
    "Action4": "4"
  }
}

into a dictionary for a class like this:

public class CharacterDesc {
    [fsProperty("controls")]
    public Dictionary<string, string> Controls;
}

I implemented myself, but I'm too lazy to rearrange my code and make a pull request:
https://gist.github.com/izackp/39e1e150b41fced0680a

Parser throws an exception on empty input

Hello,

I've recently tested some stuff and found that fsJsonParser throws an exception if empty string was passed to it. I think this is not best behaviour, and we must have fsResult.Fail for that case instead of exception.

Anyway to override default serialization behavior?

First of all thanks for your great work! fullserializer is nicely crafted and tremendously useful.

I want to know if there's is a way to custom the default behavior. Say I want the serialization to include private fields. Currently I'm changing CanSerializeField() directly.

Is there a way I can override its behavior? Thanks!

WinRT String Comparer

Found two WinRT errors

Line 80
https://github.com/NVentimiglia/fullserializer/blob/master/Source/fsData.cs

Line 362
https://github.com/NVentimiglia/fullserializer/blob/master/Source/fsJsonParser.cs

ability to convert json keys to different format without manual addition of fsProperty attribute on all properties

On our project we like have our json look like this:

{ "propertyKey" : "value" }

but the corresponding property in our code would be:

public string PropertyKey { get; set; }

or a more complicated example would be "property_key" => "PropertyKey"

It would be nice to be able to have a global rule for parsing that handles this rather than tons of fsProperty attributes. Newtonsoft's json.net handles this using something they call Contract Resolvers:
http://www.newtonsoft.com/json/help/html/ContractResolver.htm

I can't find an equivalent in FullSerializer. Am I missing something?

Compile errors on WSA.

Hi,
Im using FullSerializer in NodeCanvas and i'm getting compile errors in fsTypeExtensions.cs. They can probably be fixed easily enough with a NETFX_CORE defines.

Here are the errors. Thanks!
PJ

Assets\NodeCanvas\Framework_ParadoxNotion (shared)\Runtime\Serialization\FullSerializer\Internal\fsTypeExtensions.cs(41,60): error CS1061: 'System.Type' does not contain a definition for 'GetGenericArguments' and no extension method 'GetGenericArguments' accepting a first argument of type 'System.Type' could be found (are you missing a using directive or an assembly reference?)
Assets\NodeCanvas\Framework_ParadoxNotion (shared)\Runtime\Serialization\FullSerializer\Internal\fsTypeExtensions.cs(51,40): error CS1061: 'System.Type' does not contain a definition for 'GetGenericArguments' and no extension method 'GetGenericArguments' accepting a first argument of type 'System.Type' could be found (are you missing a using directive or an assembly reference?)
Assets\NodeCanvas\Framework_ParadoxNotion (shared)\Runtime\Serialization\FullSerializer\Internal\fsTypeExtensions.cs(52,81): error CS1061: 'System.Type' does not contain a definition for 'GetGenericArguments' and no extension method 'GetGenericArguments' accepting a first argument of type 'System.Type' could be found (are you missing a using directive or an assembly reference?)

Support all int values in serialization

Callback support for notifications when serializing/deserializing

Why? Because fsObjectProcessor does not have access to non-public members of the type.

Processors get invoked using the given storage/field type and not the actual object type