mutagen-modding/mutagen.bethesda.serialization

Library for generating serialization code between Bethesda plugin files to other formats

Overview

This is a library dedicated to providing the serialization code for converting Bethesda files to other formats. Right now Json and Yaml are supporeted, but it can be expanded to target other formats.

It's driven by using C# Source Generators to inspect the interfaces of Mutagen Objects and provide the necessary code to convert those objects to and from the desired formats.

Typical entry point API:

// Mutagen code to read in a SkyrimMod from disk.
using var modGetter = SkyrimMod.CreateFromBinaryOverlay(modPath, SkyrimRelease.SkyrimSE);

// Call into the Mutagen.Bethesda.Serialization system to Yaml
await MutagenYamlConverter.Instance.Serialize(mod, "SomePath/Output.yaml");

Customization

Source generators create code on-demand which lives in the end-user's project, rather than this library. Because of this, we can offer customization that are driven by your needs that actually changes the generated code to fulfill the customization, rather than checking/adjusting at runtime.

• Exporting a file per record, rather than a file per mod
• Name swaps
• Skipping fields
• Comments
• More to be added as desired

These customizations are driven by defining a Customization object, which contains all the instructions desired:

Here is an example of a customization file with some instructions that you could add:

public class CustomizeOverall : ICustomize
{
    public void Customize(ICustomizationBuilder builder)
    {
        builder
            .FilePerRecord()
            .EnforceRecordOrder();
    }
}
public class CustomizeNpc : ICustomize<INpcGetter>
{
    public void CustomizeFor(ICustomizationBuilder<INpcGetter> builder)
    {
        builder.Omit(n => n.Name);
    }
}

This would adjust your generated code to:

• Export a mod as a folder of record files
• Enforce those files to sort to be the same order as they appeared in the original mod (changes the file name to include index number)
• And for Npcs, specifically, omit the Name. (just for proof of concept)

This pattern can be used to provide many functionalities users might want, while generating the customized, optimized, and compiled code to fulfill it.

Spriggit

Spriggit is a downstream project that was developed alongside this library, with the goal of providing a more end-user experience, including a UI, for converting .esp files to .json or .yaml, mainly for the purpose of storing them in Git Repositories.

mutagen.bethesda.serialization's Issues

MajorRecord_Serialization too eager

MajorRecord_Serialization is generating switch cases for games that haven't been interacted with by the user. This leads to compilation errors if the user has overimported games that they aren't using

Write serialization versioning during serialization

Library versions, any options, etc

Starfield: New ESH / "Half" Plugins (header 0x400) overridden Forms not constructed correctly

The file SFBGS006.esm included with the latest Starfield patch is a new type of plugin that works similar to ESLs, but supports more objects. All plugins with the 0x400 are this new type, the xEdit team is calling "half" / ESH as opposed to Light / ESL.

These ESH plugins are loaded into the FDxxxxxx form prefix, and when overriding or referencing records in another ESH, rather than using the Master index as a prefix, 'FD' must be used.

I don't have much more detail than this - Using Spriggit I was able to create the .esm with the necessary modifications, but I then had to use a hex editor to change the leading '02's to 'FD's.

Here is the header from a the plugin I created that is able to override SFBGS006.esm:

SpriggitSource:
  PackageName: Spriggit.Yaml.Starfield
  Version: 0.20.0
ModKey: 1k_filter_empty_habs.esm
GameRelease: Starfield
ModHeader:
  Flags:
  - Master
  - 0x400
  Version: 2109579
  FormVersion: 559
  Stats:
    NumRecords: 46
    NextFormID: 4278194176
  Author: JMPZ11
  MasterReferences:
  - Master: Starfield.esm
  - Master: BlueprintShips-Starfield.esm
  - Master: SFBGS006.esm
  BNAM: 0x4465765F4D61696E00
  CHGL: 0x45F93C01

NextFormID is that same value on all three of the new plugins. Not sure why.

Example of (repaired) formlist override:

The highlighted bytes are the corrected values for formid when overriding an ESH record:

The record in yaml looks like :
"000341:SFBGS006.esm"

When translated to .esm, it is stored as '02000341', when it should be 'FD000341' (little-endianness always confuses me lol)

Also, not super important, but BNAM in the header appears to be a string - it almost looks like a branch name...

Hope this makes sense - I'm not familiar with how ESL's work, but I gather it is similar, but with the FE prefix instead, right?

Thanks!

Exclude `record` members in serialization

If meta classes are marked as record then some of their fields generage as Not Implemented

Duplicate Major Flag fields

Records export both:

• MajorFlags
• StarfieldMajorRecordFlags
• MajorRecordFlagsRaw

Would be good to consolidate into one

Record Customization

System to customize records via defining meta files, maybe like:

public class NpcCustomization : ICustomize<INpcGetter>
{
     public IRules Customize(IRules<INpcGetter> ruleBootstrap)
     {
           return ruleBootStrap
               .Exclude(n => n.Name)
               .AddComment(n => n.Health, "NPC's health")
               .AddComment(n => n.Magicka, n => $"NPC's mana: {n.Magicka}")
               .ReorderFirst(n => n.Stamina)
               .ReorderLast(n => n.Description);
     }
}

Generating for multiple games fails

Comment toggle

Have meta objects pass into serialize/deserialize call that control if comments are to be included. Can generate if checks on this meta object only if it's passed in, and optimize them out if not.