microsoft/frozenobjects

FrozenObjects serializer and deserializer

Frozen Objects

Serialize an object graph to frozen objects

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

frozenobjects's Issues

Add option to validate MVIDs when deserializing

Right now we blindly deserialize without doing any checks. Checking MVIDs so that we can report any problems is probably going to be the default option, with an escape hatch for the advanced user.

Static analyzer to restrict the kinds of types we'll allow

Banned types/things

• IntPtr
• Pointers
• Types that override GetHashCode and then call string.GetHashCode (e.g. Dictionary<string, T>)
• Types with finalizers
• Types with non-readonly fields
• Delegates
• COM types

Impossible TypeRefs being created by the serializer for "arrays"

As the code stands we're creating TypeRef's like:

TypeRef #10 (0100000a)
-------------------------------------------------------
Token:             0x0100000a
ResolutionScope:   0x23000002
TypeRefName:       System.Int32[]

AssemblyRef #2 (23000002)
-------------------------------------------------------
	Token: 0x23000002
	Public Key or Token: 7c ec 85 d7 be a7 79 8e 
	Name: System.Private.CoreLib
	Version: 4.0.0.0
	Major Version: 0x00000004
	Minor Version: 0x00000000
	Build Number: 0x00000000
	Revision Number: 0x00000000
	Locale: <null>
	HashValue Blob:
	Flags: [none] (00000000)

Such a TypeRef does not exist in the the Assembly whose AssemblyRef is pasted above. We need to add a check when we're adding the type references that if the Type is an array we do not just add a fake TypeReference.

This has not been harmful because those TypeRefs are never referenced and therefore not encountered by the Type Loader. You can imagine a type loader out there (let's say for CoreRT) that would in fact break if it couldn't resolve a TypeRef, although maybe even there things may have been fine.

Allow pure .NET 5 builds

I am a bit new to this so not sure if this is entirely accurate. Irrespective it appears that FrozenObjects should target multiple .net releases for if anyone trying to use it has just .NET 5 SDK installed.
I have tried to incorporate the changes within the branch [3] and first commit [4] but the build fails. Trying to duplicate the same information as in commit [4] here for completeness.

---- snip from comment within [4] ----
The build does fail on my system which has just .NET 5 SDK
(5.0.101). The brief errors is as follows. It is strange
that the build is not looking for System.Private.CoreLib.dll
under
"C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.1\System.Private.CoreLib.dll"
which exists.

dotnet build -f net5.0 -v detailed .\Microsoft.FrozenObjects.sln

Dependency "System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e".
	Could not resolve this reference. Could not locate the assembly "System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors.
	     For SearchPath "D:\other\FrozenObjects\src\obj\Debug\net5.0".
	     Considered "D:\other\FrozenObjects\src\obj\Debug\net5.0\System.Private.CoreLib.winmd", but it didn't exist.
	     Considered "D:\other\FrozenObjects\src\obj\Debug\net5.0\System.Private.CoreLib.dll", but it didn't exist.
	     Considered "D:\other\FrozenObjects\src\obj\Debug\net5.0\System.Private.CoreLib.exe", but it didn't exist.
	 Required by "Microsoft.FrozenObjects.InternalCalls".

1. dotPeek tells me that the PublicKeyToken used within
   System.Private.CoreLib is same within netcoreapp3.1 and
   net5.0.

Assembly System.Private.CoreLib, Version=5.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e

Assembly System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e

2. I have tried to follow [1] while targetting multiple frameworks,
   but could be missing something (considering build fails).

3. As per [2] the preprocessor for net5 is NET5_0.

[1]
https://docs.microsoft.com/en-us/nuget/create-packages/supporting-multiple-target-frameworks
[2]
https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/preprocessor-directives/preprocessor-if#remarks
[3] https://github.com/neeraj9/FrozenObjects/tree/dev-net5.0
[4] neeraj9@960dbae

Generate nuget packages for all valid .net core platforms

Linux

• x86_64 gblic
• arm glibc
• arm64 glibc
• x86_64 musl
• arm musl
• arm64 musl

Windows

• x86_64
• x86
• arm
• arm64 is supported?

macOS

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Checking for ArrayTypeDesc is no longer required in .NET 5.0

dotnet/runtime#1201

This eliminates one branch in the deserialization codepath.

Add documentation

• Describe Frozen Objects, RO segments, etc.
• Explain how such a segment is created
• Open Issues
  • Things we can't serialize (fields of void*[])
  • Things we shouldn't serialize (IntPtr, Delegate, etc.)
  • Other complicated types (types with Finalizers, non-readonly fields)
  • Should we allow prototypes for MT Tokens (i.e. the method table token/cookie for int is always 0, string is always 1, etc.)
  • Current limitations around the fact that the segment is writable (i.e. not page protected)
  • Diagnostics support (clrmd, profiling apis, etc.)
  • Is the current plan for security good enough?
  • Should we support Frozen Objects via Stream, or arbitrary memory
  • How do really support unloadability?
  • Misc: GC Heap Hard Limit
  • Testing on ARM32, ARM64

Dealing with Function Pointers on the Heap

.class public auto ansi beforefieldinit FunctionPointersClass extends [System.Runtime]System.Object
{
  .field public method instance void*(int32) [] FuncPtrField

  .method public hidebysig specialname rtspecialname instance void .ctor() cil managed
  {
    .maxstack 8
    ldarg.0
    call instance void [System.Runtime]System.Object::.ctor()
    ret
  }

  .method public hidebysig instance void Populate() cil managed
  {
    ldarg.0
    ldc.i4.s 1
    newarr method instance void*(int32)
    stfld method instance void*(int32)[] FunctionPointersClass::FuncPtrField
    ret
  }
}

Write more unit tests to build confidence in correctness

Cases:

• plain object
• object arrays
• mdarrays of object
• boxed primitives
• boxed enums
• mdarrays of generic value types
• jagged arrays
• arrays of generic reference types
• primitive arrays
• types with custom layouts specified in metadata
• Value Types that are nested
• Reference Types that are nested

Add System.Void to PrimitiveTypeCode

This will theoretically allow support for void*[].

Add test case that verifies that all the typerefs are reachable from at least one typesec

This issue to write a test case for #13.

Add support for deserializing the blob into memory allocated using Large Pages

An option on the deserializer that will allow the consumer to specify if we should call VirtualAlloc/mmap with the option to allocate in large pages.

Security: Add support for writing a secret (HMACSHA256) in the blob and deserializer

Add an option (probably it'll be the default) that allows writing an HMACSHA256 secret in the blob and support two options to provide the secret to the deserializer. Option 1 is that the assembly will have the secret embedded in it, and therefore the assembly itself must be guarded as a secret. And Option 2 will be that the deserializer will have an overload that takes in the secret as a `ReadOnlySpan'.

From a security perspective, it'll obviously be Option 2 that is desired but I feel like we should allow Option 1 and if we want to call it insecure that's fine and it can be used encode identity info in an untrusted manner.

CoreRT support

Part of the work is exposing the helpers from S.P.C so our serialized can call it.
dotnet/corert#7589

The other part is actually understanding how to go from an EEType* to GCDesc when deserializing. And for serializing hoping and praying that the layout algorithm is the same so that the serializer can run on .NET Core and deserialize on CoreRT when doing it on the same OS and architecture.