loot/libloot

A C++ library for accessing LOOT's metadata and sorting functionality.

libloot

Introduction

LOOT is a plugin load order optimisation tool for TES III: Morrowind, TES IV: Oblivion, TES V: Skyrim, TES V: Skyrim Special Edition, Fallout 3, Fallout: New Vegas, Fallout 4 and Fallout 4 VR. It is designed to assist mod users in avoiding detrimental conflicts, by automatically calculating a load order that satisfies all plugin dependencies and maximises each plugin's impact on the user's game.

LOOT also provides some load order error checking, including checks for requirements, incompatibilities and cyclic dependencies. In addition, it provides a large number of plugin-specific usage notes, bug warnings and Bash Tag suggestions.

libloot provides access to LOOT's metadata and sorting functionality, and the LOOT application is built using it.

Downloads

Releases are hosted on GitHub.

Snapshot builds are available as artifacts from GitHub Actions runs, though they are only kept for 90 days and can only be downloaded when logged into a GitHub account. To mitigate these restrictions, snapshot build artifacts include a GPG signature that can be verified using the public key hosted here, which means it's possible to re-upload the artifacts elsewhere and still prove their authenticity.

The snapshot build artifacts are named like so:

libloot-<last tag>-<revisions since tag>-g<short revision ID>_<branch>-<platform>.<file extension>

Building libloot

Refer to .github/workflows/release.yml for the build process.

Linux

The build process assumes that you have already cloned the libloot repository, that the current working directory is its root, and that the following applications are already installed:

• cmake
• curl
• git
• pip3 (and therefore Python 3)
• cargo and the rest of the Rust toolchain (e.g. via rustup)
• wget

The list above may be incomplete.

CMake Variables

libloot uses the following CMake variables to set build parameters:

Parameter	Values	Default	Description
BUILD_SHARED_LIBS	ON, OFF	ON	Whether or not to build a shared libloot binary.
LIBLOOT_BUILD_TESTS	ON, OFF	ON	Whether or not to build libloot's tests.
LIBLOOT_INSTALL_DOCS	ON, OFF	ON	Whether or not to install libloot's docs (which need to be built separately).
RUN_CLANG_TIDY	ON, OFF	OFF	Whether or not to run clang-tidy during build. Has no effect when using CMake's MSVC generator.
ESPLUGIN_URL	A URL	A GitHub release archive URL	The URL to get a source code archive from. This can be used to supply a local path if the archive has already been downloaded (e.g. for offline builds).
LIBLOADORDER_URL	A URL	A GitHub release archive URL	The URL to get a source code archive from. This can be used to supply a local path if the archive has already been downloaded (e.g. for offline builds).
LOOT_CONDITION_INTERPRETER_URL	A URL	A GitHub release archive URL	The URL to get a source code archive from. This can be used to supply a local path if the archive has already been downloaded (e.g. for offline builds).

You may also need to set BOOST_ROOT if CMake cannot find Boost.

Building The Documentation

The documentation is built using Doxygen, Breathe and Sphinx. Install Doxygen and Python and make sure they're accessible from your PATH, then run:

pip install -r docs/requirements.txt
sphinx-build -b html docs build/docs/html

Alternatively, you can use Docker to avoid changing your development environment, by running docker run -it --rm -v ${PWD}/docs:/docs/docs -v ${PWD}/build:/docs/build -v ${PWD}/include:/docs/include sphinxdoc/sphinx:7.3.7 bash to obtain a shell that you can use to run apt-get update && apt-get install -y doxygen and then the two commands above.

libloot's Issues

Account for BSA asset override as sorting criterion

Suggested by Lojack. If two non-conflicting plugins load BSAs that conflict, the BSAs' path lists could be compared to decide their load order so that the larger BSA loads first.

Moved from loot/loot#157.

Switch to Rust versions of libespm and libloadorder

The Rust rewrite of libespm is a separate library, esplugin, as the API is completely different. libloadorder's Rust rewrite will be released as libloadorder v10 as there are some breaking changes but the API is mostly preserved.

One of the libloadorder changes is the removal of path caching and exposure of control over plugin caching to the client, in the interests of correctness, as the internal cache invalidation was prone to not picking up on changes. While performance of loading the load order state has been greatly improved, taking < 16 ms to load a 1000 plugin / 300 active load order, the LOOT API (and LOOT, through the passthrough functions the LOOT API exposes) assumes that calls are effectively free and can make several hundred of them almost at the same time.

It may be possible to hide cache management within the LOOT API, tied to other functions, but this will need to be documented, or a new passthrough function could be added to pass responsibility for the cache to LOOT.

Use prebuilt Boost on AppVeyor

AppVeyor build environments come with a prebuilt version of Boost, using this would speed up builds for new branches. Though the MSVC 2015 environment only has Boost 1.63.0, this shouldn't be a problem, as builds were using 1.61 until quite recently.

Add a 'group' string field to plugin metadata for use as a sorting criterion

Suggested by gabba here and here.

This would add another step to sorting, after all but the tie-break edges, that would add edges so that plugins that had the same group field value would sort as close together as possible without creating a cycle.

Moved from loot/loot#659.

Avoid adding group metadata for plugins that load after other plugins that are in groups

Consider the situation with plugins A.esp, B.esp, C.esp and A - B patch.esp, where:

• A and A - B patch.esp are in the default group.
• C.esp is in the mid group.
• B.esp is in the late group.
• The mid group loads after the default group.
• The late group loads after the mid group.
• A - B patch.esp has B.esp as a master.

This causes a cycle that LOOT won't break because the group metadata is valid in group-isolation (i.e. with no other group metadata applied).

The intuitive solution is to ignore A - B patch.esp's group because it has higher-priority metadata causing it to load after a later group, but that is basically the same as priority inheritance was, and in practice that was generally confusing and unintuitive. You can end up with a chain of inherited groups/priorities (say something had metadata to load after the patch) that cause the resulting derived metadata to bear little resemblance to the input metadata.

There may be a better way to generally implement the intuitive solution, or it may be better just to decide that A - B patch.esp needs to have metadata adding it to the late group (or any group that mid doesn't load after).

Replace Boost.Log usage

Now that the LOOT API just logs to a callback set by the client, there's no need for such a heavyweight logging library as Boost.Log (which is probably doing something that causes loot/loot#835). spdlog seems like a popular, lightweight alternative.

Represent optional scalar metadata fields with std::optional

This is a more general take on #14. There are a number of scalar metadata fields that are optional, and it can currently be tricky to distinguish unspecified values from specified defaults. For example, PluginMetadata's group field defaults to default, so it's impossible to tell if user metadata that sets a value of default is overriding whatever the masterlist metadata is or has omitted the field, without comparing against the masterlist, which shouldn't be necessary. Diffing metadata is also more complicated.

One solution would be to use the C++17 std::optional<T> type to represent optional scalar fields, more accurately reflecting the metadata format in code. This does introduce large breaking API changes and requires a C++ compiler and runtime update, but there are other C++17 features that make updating attractive, and the result would be a more semantic API.

The metadata structure fields that are optional scalars are:

• PluginMetadata group
• File display
• File condition
• Group description
• Location name
• Message condition
• Tag condition
• PluginCleaningData itm
• PluginCleaningData udr
• PluginCleaningData nav

Add metadata to force a plugin to load immediately after another

Discussed in loot/skyrim#190 and loot/skyrim#245, the idea is to have something like

- name: B.esp
  directly_after: A.esp

which has the effect that LOOT will sort B.esp after A.esp with no other plugins between them.

Open Questions

• Should directly_after be invalid for regex entries? It only makes sense for one match to use the value, but the regex entry may be for a set of exclusive plugins anyway.
• How would this interact with the plugin aliasing feature proposed in #46?
• How to handle merging metadata entries that specify different directly_after values?
• How to handle the case where plugin masters and 'hard' metadata like after and req entries form a cycle with directly_after?
• How to implement this - should it be part of the graph sort, or done as a post-topological sort "override" step?

@Kaenguru-Knobel-Kette has noted that it would be good idea to get the opinions of other compatibility patch authors on this, to see if their expectations or use cases differ.

Allow absolute file paths to plugins

It is possible to make GameInterface::IsValidPlugin and GameInterface::SortPlugins functions to accept absolute file paths not only Data-relative?

Metadata syntax: clean and dirty in regex plugin metadata entries

I just added the following entry to the FO4 masterlist:

  - name: 'Deeper Thoughts (\(Curie\)|- (Expressive |)Curie)\.esp'
    url: [ 'https://www.nexusmods.com/fallout4/mods/25050' ]
    clean:
      # Deeper Thoughts (Curie).esp; version 1.1
      - crc: 0xA93E1500
        util: 'FO4Edit v3.2.3v'
      # Deeper Thoughts - Curie.esp; version 1.1a, with Affinity-Sorted Dialogue
      - crc: 0xB15CFBE9
        util: 'FO4Edit v3.2.3v'
      # Deeper Thoughts - Expressive Curie.esp; version 1.1b, with Affinity-Sorted Dialogue - Expressive
      - crc: 0xF39776A0
        util: 'FO4Edit v3.2.3v'
    msg:
      - <<: *useOnlyOneX
        condition: 'many("Deeper Thoughts (\(Curie\)|- (Expressive |)Curie)\.esp")'
        subs: [ 'Deeper Thoughts - Curie plugin' ]

And got a message that the syntax was wrong. So I scratched my head and went to try and validate the YAML… which came out perfectly fine. So I went to see Travis, and Travis let me know that the error was because there was a clean entry in a regex plugin object.

I actually think this should be allowed. If we trust users to not mess around with the plugins to cause hash collisions, any plugin with a CRC32 of "B15CFBE9" should be "clean".

Or is it not allowed due to performance concerns?

Slow start

LOOT takes a significant time to start up. Attached is the debug log:

LOOTDebugLog - slow start.zip

Make constructors with args explicit

I've recently hit a few type ambiguities that could be resolved by making constructors that take args explicit, but that's a breaking change to implement for API classes, so make the change for v0.15.

Sort plugins closer to their masters

Proposal: Any mod that has a non-ESM master should be placed as close to it's master as possible.

Reasoning: There are several advantages to this. When it comes to patches, I have detailed why proper patch placement is immediately after the last of its masters here, with an example of how not placing them that way can cause errors: https://www.nexusmods.com/skyrimspecialedition/articles/625

I initially thought to request this only for conflict resolution patches, and identify those as mods with two or more non-vanilla masters, but it occurs to me that the principle holds even for non-patch mods that have only a single non-vanilla, non USSEP master. Take a mod like Enhanced Blood Textures. If you choose all the options in the installer, you'll get a host of sub-plugins (like "no screen blood"), all of which have a main EBT plugin as a master. Currently, LOOT will litter these sub-plugins all over the list, which is terrible from an organizational perspective, makes it impossible to synchronize installation and plugin order, and leads to possible conflicts with intervening mods that change the same record being overridden by a sub-plugin of EBT but not EBT itself. With my proposal, all the EBT plugins will be grouped together nice and neatly in the plugin list, which wouldn't just prevent conflicts, it would also just look better and be easier to review and adjust their position all together as a single unit, since they really should be considered and treated as such.

Or take ACE (Armor and Clothing Extension), which has one master, WACCF (Weapons, Armor, Clothing and Clutter Fixes). The author - any author really - would certainly design these with the assumption that ACE would immediately overwrite WACCF in every shared record. An intervening mod C between the two that shared a record with them would be only partially overwritten by ACE, while possibly also partially overwriting WACCF in some other regard. This can only cause havoc. With ACE naturally placed immediately after WACCF, the combination will be either wholly overwritten by Mod C, or Mod C will wholly ovewrite WACCF+ACE, but there should generally exist no good circumstance in which any mod (other than patches for WACCF) should come between WACCF and ACE, partially overwriting one and being partially overwritten by the other. This is also a good example of why a patch for WACCF and some other Mod X should come immediately after WACCF, as placing that patch below ACE will revert ACE's intentional changes back to WACCF's values in any shared record.

Note that in my methodology below, I'm going to treat this as if it should only apply to masters without the ESM flag checked, but it may be better to just exclude the vanilla ESMs and USSEP. I'm still trying to decide if there ARE any pros or cons in regards to whether it should just be vanilla ESMs and USSEP excluded, or all ESMs excluded. For now I'll just say all ESMs since it makes the below more readable. Please treat the words "ESP master" below to mean "non-ESM flagged" masters.

Methodology:

Pass 1: Go through all plugins and count the number of ESP masters each plugin has.
Pass 2: Apply LOOT's sorting algorithm only to mods with zero ESP masters.
Pass 3: Insert into the result of Pass 2 all mods with 1 ESP master, placing them immediately after its master or as close to it as strict metadata allows.
Pass 4: Insert into the result of Pass 3 all mods with 2 ESP masters, placing them immediately after the last of its masters or as close to it as strict metadata allows.
Pass X: Insert into the result of Pass X all mods with X-1 ESP masters, placing them immediately after the last of its masters or as close to it as strict metadata allows.

Steps 3-X are really all one step, the point being to order all base mods with no masters first, then insert mods with 1 master, then insert mods and patches with 2 masters, etc. till you're done.

By "strict metadata" I mean specific "Mod X must come after Mod Y" ordering, not just "Group B must come after Group A" ordering. If this turns out not to work for all mods that have only one master, I won't mind this proposal being reduced to only applying to actual conflict patches (2 or more masters), which should always be possible. I just decided to propose this for all mods with any masters until we can find a solid reason not to and because it might actually be easier to implement this way.

Possible complications and solutions:

Problem A) What if Mod A has a master B which itself has more masters than A does. This could be handled in two ways:

1. During pass 1, build an expanded master list, where if Mod A has a master B, and master B itself has masters C and D, then C and D are also counted as masters of Mod A for the purpose of determining which pass it should be inserted in.
2. Probably easier, if the current pass is attempting to insert a plugin whose master hasn't itself been inserted yet, simply skip Mod A in that current pass. Eventually master B will be inserted, and any mod A that was skipped can then be inserted in a later pass.

Problem B) How do you order patches that would be inserted into the same position via this method?

Take mods A, B and C. There exist patches for A-B, for B-C, and for A-C. My method would result in this:

1: Mod A
2: Mod B
3: Patch A-B
4: Mod C
?: Patch A-C
?: Patch B-C

Those last two patches would be inserted in the same pass, and placed after C because that's the lower of their masters. How should they be ordered? Answer: They should be ordered by the position of their other master. Since Mod A should come before Mod B, then patch A-C should come before patch B-C.

Please throw me any questions or arguments you have against this, I've given a lot of thought to it for a year now and I plan to monitor this request. To be honest, that LOOT doesn't do this is the only reason I have not used LOOT for actual ordering until now... so far I just use it to advise me of required patches. If it did this, I would be very likely to adopt and recommend its ordering highly.

DLL doesn't have correct FileVERSION and ProductVERSION field values

src/api/resource.rc has the correct values for FILEVERSION and PRODUCTVERSION but not their equivalent text fields, the script to set versions must not cover them.

Sort out case insensitivity discrepancies

The LOOT API uses a mix of

• boost::locale::to_lower (which is all Boost.Locale is used for)
• boost::ilexicographical_compare
• boost::iequals
• boost::iends_with
• boost::istarts_with

to do case-insensitive string comparison, but only the first is correct for non-ASCII strings. That's fine for boost::iends_with, which is only used for comparison of plugin file extensions against hardcoded ASCII strings, but the others can give inaccurate results. The other usages will needed to be evaluated on a case by case basis, as it may be important not to have false positives/negatives for some usages, and others may not actually require case sensitivity, or play by different rules. Changing this will involve API breakage.

boost::locale::to_lower

Operating on filenames:

• File::operator<: Should be case insensitive, but can't access filesystem to check. Don't change.
• File::operator==: Should be case insensitive, but can't access filesystem to check. Don't change.
• Game::LoadPlugins(): Change to compare canonical file paths.
• Game::IdentifyMainMasterFile(): Case doesn't matter if Game::LoadPlugins() is changed to compare canonical paths.
• GameCache::GetPlugin(): Should be case insensitive, but can't access filesystem to check. Don't change.
• PluginMetadata::GetLowercasedName(): Needs to be exposed for case-insensitive uniqueness.
• PluginInterface::GetLowercasedName(): Remove. The Plugin::GetLowercasedName() implementation is used by the plugin cache and for storing objects in a set, but calling to_lower() on Plugin::GetName() is more appropriate for the former, and case insensitivity isn't necessary for the latter since plugins are the output of a single directory scan (and a hashset is more appropriate).
• PluginSorter::AddHardcodedPluginEdges() Since all plugins exist at sorting time, checks should probably use fs equivalence instead.
• PluginSorter::ComparePlugins(): to_lower() is only used to lexicographically compare basenames for human friendly sorting, which is fine.
• shouldIgnorePlugin(): No need for case insensitivity if ignorePlugin() is made case sensitive.
• ignorePlugin(): No need for case insensitivity.

Using any of the Boost functions to emulate Windows' path case insensitivity is incorrect as it behaves differently, and probably the only way to match Windows' behaviour correctly is to use the filesystem functions it provides (or std::filesystem). std::filesystem::equivalent() can be used to check if two paths that point to existing files point to the same file, respecting Windows' case insensitivity, but throws if the paths don't exist. Also, std::filesystem::canonical() lowercases filenames on Windows. Because they involve filesystem access, it would be good to avoid using them in hot paths, but it probably doesn't matter given LOOT's current performance.

On the other hand, boost::locale::to_lower probably comes very close to matching the Windows filesystem behaviour, and maybe it's not worth getting any more accurate.

Operating on condition strings:

• GameCache::CacheCondition()
• GameCache::GetCachedCondition()

Condition strings are tricky to get right, as they can contain file paths, so part of them can have the same issues as described above, and version strings should also not be treated case insensitively. Resolving this overlaps with #34, and it's probably best to just make caching case sensitive, treating all the variations in case handling correctly isn't really worth it.

boost::ilexicographical_compare

• Location::operator<: Operates on URL, which can have a path that is case sensitive or not. It's best to treat as case sensitive to be safe.
• MessageContent::operator<: Operates on message text, there's no point being case insensitive.
• Message::operator<: Operates on message text, there's no point being case insensitive.
• Tag::operator<: Operates on the tag name, which is case sensitive.

boost::iequals

• Location::operator==: Operates on URL, which can have a path that is case sensitive or not. It's best to treat as case sensitive to be safe.
• MessageContent::operator==: Operates on message text, there's no point being case insensitive.
• Tag::operator==: Operates on the tag name, which is case sensitive.
• Plugin::LoadsArchive(): Operates on file extensions, should match filesystem case sensitivity.

boost::istarts_with

• Plugin::LoadsArchive(): Operates on the plugin name, should match filesystem case sensitivity.

Allow specifying whether to use ProductVersion or FileVersion for a version() condition

SkyrimSE.exe has "1.0.0.0" in its FileVersion which is what LOOT currently uses for its version() condition, and has the "real" version ("1.5.50.0") in its ProductVersion field. This makes it so that there is currently no good way of comparing game versions for Skyrim SE.

Maybe add product_version() and file_version() in addition to the current version()? Or add another optional parameter to version() to specify which to look at for EXE/DLLs (e.g., version(file, version, comparator{, field}))? (The latter approach could maybe also be used to handle how different mods use different "versioning" methods which could maybe also be named and included there? Not sure if this is currently an issue though, so maybe not worth considering for now.)

See loot/skyrimse@2df100a for an example of what is not currently possible for SSE.

product_version uses wrong path?

Since updating Vortex to libloot version 0.14 I've started seeing error reports like this:

"Failed to evaluate condition "product_version("../SkyrimSE.exe", "1.5.62.0", <) and file("../SkyrimSE.exe") and version("Unofficial Skyrim Special Edition Patch.esp", "4.1.6", >=)". Details: An error was encountered while reading the version fields of "D:\Program Files (x86)\The Elder Scrolls V Skyrim Special Edition\Data../SkyrimSE.exe": entry not found"

or

"Failed to evaluate condition "version("Unofficial Fallout 4 Patch.esp", "2.0.5", >=) and not version("../Fallout4.exe", "1.10.111.0", >=)". Details: An error was encountered while reading the version fields of "C:\Games\Fallout 4\Data../Fallout4.exe": zero fill"

Not sure if this may be something I'm doing wrong but looking at the error message it appears like the relative path "../SkyrimSE.exe" is appended directly to the data path, without a (back-)slash so that would explain "entry not found" but why does it happen in the first place, why only for some users and what does "zero fill" mean?

Replace priorities with groups

Priorities in LOOT's metadata have always been a pain point, as:

• they're a bit abstract
• local vs. global can be confusing
• inheritance of priority values is difficult to reason about

Priorities were introduced to solve the problem of "how do I say plugin X needs to load late, after most but perhaps not all plugins, without having an enormous after list?" and similar for loading plugins early (which is actually the messier problem, as there's no before). In BOSS this was manually solved by dividing the masterlist into themed sections, and while this wasn't a suitable solution for LOOT, perhaps it's worth taking some inspiration from it.

It may be possible to replace priorities with groups, which would act a bit like global priorities but be more accessible. For example:

groups:
  - name: default
    after:
      - Unofficial patches
  - name: &mapMarkersGroup Map Markers # Create an anchor to the group name
    after:
      - default
  - name: Unofficial patches Has an implicit after: []

plugins:
  - name: Unofficial Patch.esp
    group: Unofficial patches
  - name: foo.esp # Belongs to the 'default' group
  - name: map marker.esp
    group: Map Markers
  - name: other map marker.esp
    group: *mapMarkersGroup # To prevent typos, references can be used

this is equivalent to:

plugins:
  - name: Unofficial Patch.esp
  - name: foo.esp
    after:
      - Unofficial Patch.esp
  - name: map marker.esp
    after:
      - Unofficial Patch.esp
      - foo.esp
  - name: other map marker.esp
    after:
      - Unofficial Patch.esp
      - foo.esp

and is similar to but not the same as:

plugins:
  - name: Unofficial Patch.esp
    global_priority: -1
  - name: foo.esp # global_priority is 0
  - name: map marker.esp
    global_priority: 1
  - name: other map marker.esp
    global_priority: 1

Defining the above:

• Groups are identified by name strings.
• Groups can load after other groups.
• Groups load after nothing by default, i.e. [] is the default value for the after key.
• Groups don't need to be defined before they're referenced.
• A plugin can belong to a single group.
• All plugins would belong to the default group by default.
• The default group doesn't need to be declared in groups.
• The default group can be declared in groups with a non-empty after array.
• A plugin group value that doesn't match a group named in groups would cause a sorting error.
• A group in a group's after list that doesn't match a named group would cause a sorting error.
• During sorting, groups would recursively resolve to sets of plugins, and a plugin would add all the plugins sets its group loads after to its after metadata. Resolution needs to be recursive as without it if (using the above example) foo.esp is not installed, map marker.esp wouldn't know to load after Unofficial Patch.esp.
• Userlists would be able to change a plugin's group, add new groups and extend existing groups' after lists. In keeping with userlist behaviour for other metadata, it would not be possible to remove entries from a group's after list.

Behavioural differences vs. priorities:

• Groups ignore whether or not plugins conflict, i.e. they're more like global priorities than local priorities.
• Priorities can't cause cycle errors as they are only applied if they don't contradict a plugin's masters, req or after metadata, but groups resolve to after metadata so can cause cycles. Using the above example, if foo.esp had an after for map marker.esp, that would cause an error.

A few other points worth noting:

• As group identifiers are strings, there's no need to think about making good use of the available range of values, it's always trivial to insert groups.
• Because all the groups (apart from default) are named and declared in the same place (in a given metadata file), it's easy to see what groups are available and get an idea of what they're for.

Make groups optionals

Every plugin belongs to the default group by default, but a user can also explicitly assign a plugin to the default group to override masterlist metadata placing it in another group. To distinguish these cases, PluginMetadata keeps track of whether a group value was explicitly set or not, but this is fragile as the explicit flag could go out of sync, and it also complicates usage.

It would be more useful if the plugin metadata's group data type could indicate if the group was specified or not. In pseudocode, this would look something like:

enum PluginGroup {
  ImplicitDefault,
  Explicit(string),
}

One way to represent this could be to use C++17's std::optional<std::string>, where a none value is equivalent to implicit default. Alternatively, LOOT could supply its own data type that provides a more problem-specific API.

File metadata's display field isn't used in messages

Reported by @CptMcSplody in loot/skyrimse#113 and replicated with requirement and incompatibility messages.

product_version evaluation returning incorrect result

I did some testing of the product version check with the latest bintray build, and it doesn't seem to be evaluating correctly.
I was testing if SkyrimSE.exe was the latest version, and that was not showing when true.
I was also testing if it was an old version, and that message was showing.

Tested using:
Bintray snapshot loot_0.13.6-28-gb8badfc_dev
SkyrimSE.exe

---

  - <<: *hasOldVersionOfX
    subs: [ 'Skyrim SE' ]
    condition: 'product_version("../SkyrimSE.exe", "1.5.62.0", <) and file("../SkyrimSE.exe")'
  - <<: *hasLatestVersionOfX
    subs: [ 'Skyrim SE' ]
    condition: 'product_version("../SkyrimSE.exe", "1.5.62.0", ==) and file("../SkyrimSE.exe")'

hasOldVersionOfX showing with above conditions.

---

If I switch the conditions.

  - <<: *hasOldVersionOfX
    subs: [ 'Skyrim SE' ]
    condition: 'product_version("../SkyrimSE.exe", "1.5.62.0", ==) and file("../SkyrimSE.exe")'
  - <<: *hasLatestVersionOfX
    subs: [ 'Skyrim SE' ]
    condition: 'product_version("../SkyrimSE.exe", "1.5.62.0", <) and file("../SkyrimSE.exe")'

The hasLatestVersionOfX message will show.

Experiment with reducing the number of edges added to the plugin graph

During sorting, as adding many edges between all the plugins in a large load order takes longer than loading the plugins.

Moved from loot/loot#761.

Use C++17 std::optional<T> in return type of unexceptionally-fallible functions

For example, functions like GetPluginMetadata() and GetPluginUserMetadata() currently return name-only PluginMetadata objects if the requested metadata doesn't exist, which isn't obvious. Worse, GetPlugin() throws if you ask for a plugin that isn't loaded, requiring a try-catch block to handle what is a reasonable and unexceptional failure case.

std::optional<T> is perfectly suited to representing such cases in a meaningful and type-safe way, so it would be good to adopt it. The following API functions could be improved with its adoption:

• DatabaseInterface::GetPluginMetadata()
• DatabaseInterface::GetPluginUserMetadata()
• GameInterface::GetPlugin()
• PluginInterface::GetVersion()
• PluginInterface::GetCRC()

This doesn't include metadata accessor methods, which are to be handled by #30, and there are probably a bunch of internal functions that could also benefit.

Rename the LOOT API to something else

It's confusing to talk about the library and call it the "LOOT API": it's not an API, it's a library that provides an API. I propose we rename it to the boring but obvious "libloot", though I'm open to other suggestions. I'd want to respect platform naming conventions with that name though, so the library DLL's filename would be loot.dll, and on Linux it would be libloot.so.

@Freso What's your opinion / any suggestions for a new name?

Sorting doesn't account for plugin hoisting

Plugin hoisting occurs when a master plugin has a non-master plugin as a master. In this case, the game will load the non-master immediately before the earliest master that depends on it (or between it and the previous master if there are multiple non-masters to hoist).

In this scenario, LOOT will encounter a cyclic interaction error as it tries to enforce both the masters-load-before-non-masters rule, and the plugin-masters-before-plugin rule. While this is an edge case, LOOT shouldn't error out unnecessarily.

Supporting this will require updating to libloadorder 12, which introduces support for plugin hoisting and requires that edits to the load order represent the hoisting that the game would otherwise do at runtime.

Refactor language handling out of the API

The Language class and LanguageCode enum being part of the API mean that adding a language to LOOT needs the API to be updated too, which isn't strictly necessary. They could be moved to the GUI if the API just used locale strings instead, with a hardcoded constant for referencing the English language.

Don't rely on LOOT metadata to sort Creation Club plugins correctly

As loot/loot#956 shows, there can be a large number of CC plugins, and to get them all in the right load order using masterlist metadata requires very verbose lists of plugins to load after for each one. Given that these lists are just repeated sub-lists of what appears in Fallout4.ccc or Skyrim.ccc, LOOT should be smart enough just to pull the information from there.

As the logic for parsing the .ccc file is already present and tested in libloadorder, it makes sense to take advantage of that. Doing so introduces slightly circular logic in that libloadorder will be checking the load order partially using data that LOOT partially used to produce the load order, but the correctness of the data obtained is verified by libloadorder's unit tests, and libloadorder would be checking that the data was used correctly.

This will need a libloadorder update to expose a C API for getting the list of implicitly active plugins, which can then be looped over when sorting to add edges between them and from them to other plugins as necessary. The list will also include hardcoded plugins, so they won't need LOOT metadata to sort correctly either, though the original Skyrim's Update.esm is a special case since it isn't hardcoded to load in a particular position, just hardcoded to load, so it'll need to be ignored by the looping.

Version comparison: `2.0.1 == 2.0.1a` is True

I was updating version information for the UFO4P in the FO4 masterlist, and wasn't sure how to enter the version, so I played around a bit. It turns out that the version string 2.0.1 evaluates to be the same as 2.0.1a, which is wrong. However, it might be the better option there is, since "2.0.1a" can be used both to say "2.0.1-alpha" (ie., a pre-release to 2.0.1) or "2.0.1 patch A" (ie., a follow-up release to 2.0.1). I'm pretty sure @Arthmoor uses the latter meaning, but I don't see a way LOOT will be able to automatically make this distinction, so maybe this is an unsolvable issue. :/ (Other than maybe urging @Arthmoor and others using such a scheme to do something like "2.0.1.1" instead.)

Maybe the LOOT(-API) docs need to have a note somewhere about this that can be referenced when contacting mod authors?

Add 'description' key for group structures

Right now it's possible to name groups structures, but beyond the name, there's no way to explain what each group is supposed to contain directly in the metadata. You can write it in comments in the masterlist, but that won't be shown to end users adding e.g., local mods to groups.

This would also be a backwards compatible change: no description should default to empty string and 0.13 compatible software should ignore unknown keys, so adding descriptions wouldn't change anything for those, similar to how the urls don't do anything.

Once this has been implemented (if agreed on/with), description should also be shown in the LOOT UI, but that's a separate issue for then. :)

Masterlist update process is error-prone

LOOT gets a lot of issues reported involving updating the masterlist that turn out to be fixed by deleting the LOOT/<game> folder. loot/loot#801 and loot/loot#947 are two examples of this that are currently open.

Most issues seem to be about the game folder not being empty when it's expected to be (AFAIK that's only when cloning), or something preventing files being removed/deleted as part of ensuring the game folder is empty.

The masterlist update code is an overly-complicated mess, and simplifying it will make it more maintainable and reduce the surface area for bugs. Pseudo-code for the necessary functionality involving a minimal number of operations is:

update masterlist {
    if repo does not exist {
        clone repo
        checkout branch
    } else {
        set remote url
        fetch updates

        if local branch exists {
            if local and remote tips match and local tip is checked out and working copy has no changes {
                return false
            } else {
                delete local branch
            }
        }

        checkout new branch tip
    }

    loop {
        if parsing masterlist fails {
            checkout previous revision
        } else {
            return true
        }
    }

    // Should never be reached, if checkout previous revision
    // reaches the start of history it'll throw an exception
    return true 
}

The current code has a lot more complexity because it tries to do things like merge branches, when throwing away the current local branch is just as good for LOOT's purposes.

Overhaul test fixture setup to create operate in independent temporary directories

As part of #24 I have to write a whole bunch of new tests to check character encoding conversion is happening correctly wherever paths and strings are used, and the way the test fixtures are currently implemented makes this a pain.

At the moment CommonGameTestFixture and others create and remove files in the build directory for each test, but if one test fails that means the directory can be left in a bad state that can cause subsequent tests to fail.

Instead, create a unique temporary directory for each test, copy the necessary test plugins / metadata files into it, and create whatever files and directory structures the fixture sets up in there. This is slower, since more files need to be copied for each test, but it does allow for tests to be run in parallel, which should be faster overall, and solves the issue with tests not operating in isolation.

Master List Group Position for "Alternate Start" causes issues

The master list's position for the "Alternate Start" group is all the way at the end, right before dynamic patches. However, putting Alternate Start mods at the beginning tends to cause problems as their scripts then start later and cause problems in games with long load orders.

Problems range from mod items being added a new game being cleared from inventory due to race conditions relating to the vault 111 scripts, vault 111 not opening, and load screen issues on Start Me Up when starting outside the vault.

I tried to modify the order myself, but it looks like groups that came with the master list cannot be edited.

Add condition to SimpleMessage

SimpleMessage doesn't have a condition field, but it would be more useful if it did.

Simplify Travis execution environment

Travis images have been updated to Ubuntu Trusty, and also have Clang 3.5 and CMake 3.2 (upgrading to Clang 3.9 today), if they are used there's no need to install GCC 6 and CMake 3.8 ourselves, which should speed builds up by a couple of minutes according to current build timings.

Non Master Plugins Group Affecting Master Sorting Order

LOOTDebugLog.txt

Sorting with a non ESM plugin in a group before ESM's.

Dragonborn.esm (DLC Group)
RSkyrimChildren.esm (Early Group)
BSAssets.esm (Default Group)
BSHeartland.esm (Default Group)
Lanterns Of Skyrim - All In One - Main.esm (Early Group)
Unofficial Skyrim Special Edition Patch.esp (Fixes Group)
Skyrim Project Optimization - Full Version.esm (Fixes Group)
UnlimitedBookshelves.esp (CC Group) non ESM plugin

DLC -> CC -> (Pre-)Early -> (Pre-)Fixes -> Default

What I expected

Dragonborn.esm (DLC Group)
RSkyrimChildren.esm (Early Group)
Lanterns Of Skyrim - All In One - Main.esm (Early Group)
Unofficial Skyrim Special Edition Patch.esp (Fixes Group)
Skyrim Project Optimization - Full Version.esm (Fixes Group)
BSAssets.esm (Default Group)
BSHeartland.esm (Default Group)
UnlimitedBookshelves.esp (CC Group)

Version condition caching lowercases version string to compare against

This is not correct behaviour, as semver states that

identifiers with letters or hyphens are compared lexically in ASCII sort order

so case should be preserved. This is unlikely to cause issues in reality as it's a small corner case, but good fixes would be:

• Don't cache conditions case-insensitively. Simple, but increases the chance of false negatives when checking the cache.
• Parse conditions into data structures that can provide per-token granularity when checking equality, so that version strings can be compared case-sensitively while the rest of the string is compared case-insensitively. A lot more work, but better design than how conditions are currently handled.

Feature Request: plugin aliasing.

That's probably not very clear as to what I mean so i'll give an example.

Say that I decided to convert SkyUI.esp to .esl using SSEEDIT. Loading up LOOT understandably treats SkyUI.esl as a completely new mod and lacks the masterlist info it had as an .esp.

Undefined behaviour if a DatabaseInterface pointer is used after discarding its creator GameInterface

A DatabaseInterface (implementation) keeps a reference to the GameInterface (implementation) it was created for, so if the latter is destroyed, the former enters an invalid state, leading to undefined behaviour. I encountered this while trying to adapt the python wrapper with minimal breaking changes (I had a function that constructs a local game handle and returns its database).

The implementations should be refactored so that there is no storage of a Game object in the ApiDatabase object, or if an object is stored, it should be in a smart pointer.

0.13.1: Cyclic interaction on previously working plugins

I wasn't sure whether to put this issue here or in libloadorder.

I'm working on a preliminary update of lootcli for the latest API versions. It's working pretty well, but I'm running into cyclic interaction errors with a certain set of plugins that never used to cause an error. It's possible this is a masterlists problem since when I was still using the older masterlists it was not happening (though the load orders were wrong).

Here is the logged error:

LOOT failed: Cyclic interaction detected between plugins "RLO - IC Patch.esp" and "Alternate Start - Live Another Life.esp". Back cycle: Alternate Start - Live Another Life.esp, RAO - Alternate Start Patch.esp, Immersive Citizens - AI Overhaul.esp, RLO - IC Patch.esp

API code contains translatable text

There are some uses of boost::local::translate() that I only spotted after splitting the API and GUI repositories, and the GUI translation won't pick up on them, so they should get refactored into the GUI code.

Adding *noAutomaticFix, a warning for mods with deleted navmeshes

Currently, LOOT's masterlists either mark mods with deleted navmeshes as 'clean' or show a message like this:

· SSEEdit found 2 deleted navmeshes. A cleaning guide is available here.

Both approaches are not ideal:

• Marking them as 'clean' is clearly wrong - they're about as far away from 'clean' as they could get.
• The cleaning guide the other approach links is the xEdit guide for cleaning ITMs and UDRs, which will obviously not help with fixing deleted navmeshes.

It was decided on the Discord that LOOT should use a third approach: Showing a warning to users that strongly discourages them from using the mod and also tells them to report the error to the mod author so that they can fix it properly.

This will require adding a new message to the masterlist with support from the LOOT API. Instead of this:

dirty:
  - <<: *quickClean
    crc: 0x4768B826 # v1.6
    itm: 39
    nav: 2
  - <<: *dirtyPlugin
    crc: 0xCBA83065 # v1.6
    nav: 2

the mod would be marked like this:

dirty:
  - <<: *quickClean
    crc: 0x4768B826 # v1.6
    itm: 39
    nav: 2
  - <<: *noAutomaticFix
    crc: 0xCBA83065 # v1.6
    nav: 2

Open Questions (and some potential answers)

• What should the actual message's text be?
  • It should be direct and upfront about the danger of using the mods, while also telling the user to report to the mod's author. A WIP phrasing the Discord came up with was

    This mod contains deleted navmeshes, which are known to cause crashes. It is strongly recommended not to use such plugins. Please also let the mod's author know about this, so that they can fix it properly.

• Is *noAutomaticFix an accurate name for this?
  • Elminster came up with *noAutomaticFix on the fly. It could easily be changed to, for example, *delNavPresent.
• Should this be shown only after all other issues with the plugin have been fixed, or straight from the start?
  • In my opinion, it doesn't make much sense to only show this once the other issues have been fixed. We'd be telling users 'yeah the mod has issues, but you can just clean them!' only to then go 'nope, you can't really clean this mod and you shouldn't have been using it to begin with'.

If the message ends up being shown from the start, then it should also link to the 'quick auto cleaning' guide. We would then end up with something like this:

dirty:
  - <<: *noAutomaticFix
    crc: 0x4768B826 # v1.6
    itm: 39
    nav: 2
  - <<: *noAutomaticFix
    crc: 0xCBA83065 # v1.6
    nav: 2

Errors building on Arch Linux

I'm trying to make loot-api build on Arch Linux, the cmake succeeds, but the make fails:

/bin/sh: /tmp/makepkg/loot-api-git/src/loot-api/external/src/libloadorder/docs/latex/make.bat: No such file or directory
make[6]: *** [CMakeFiles/loadorder.dir/build.make:384: libloadorder.a] Error 127
make[6]: *** Deleting file 'libloadorder.a'
make[5]: *** [CMakeFiles/Makefile2:218: CMakeFiles/loadorder.dir/all] Error 2
make[4]: *** [CMakeFiles/Makefile2:230: CMakeFiles/loadorder.dir/rule] Error 2
make[3]: *** [Makefile:238: loadorder] Error 2
make[2]: *** [CMakeFiles/libloadorder.dir/build.make:115: external/src/libloadorder-stamp/libloadorder-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:142: CMakeFiles/libloadorder.dir/all] Error 2
make: *** [Makefile:152: all] Error 2

I'm currently doing this:

mkdir build; cd build
cmake ..
make all

pdbs?

I'm currently trying to debug a crash and while I think there is a good chance the bug is in my code, the dumps always contain only loot_api.dll so I was wondering if you had pdbs for the published releases.
In particular I'm interested in v0.12.2

The stack I'm getting looks like this:

	ntdll.dll!RtlReportCriticalFailure
 	ntdll.dll!RtlpHeapHandleError
 	ntdll.dll!RtlpLogHeapFailure
 	ntdll.dll!RtlFreeHeap
 	ucrtbase.dll!_free_base
	loot_api.dll!00007ffaca6052bc
 	loot_api.dll!00007ffaca75c8e8
 	ntdll.dll!LdrpCallInitRoutine
 	ntdll.dll!LdrShutdownThread
 	ntdll.dll!RtlExitUserThread
 	kernel32.dll!BaseThreadInitThunk
 	ntdll.dll!RtlUserThreadStart

Error is Unhandled exception at 0x00007FFAEF9F7AEB (ntdll.dll) in crash-renderer-1516704349052.dmp: 0xC0000374: A heap has been corrupted (parameters: 0x00007FFAEFA5C6F0).

Sorting can't handle a light master that depends on a non-master

It's possible to create a light master that depends on a non-master, and the game will load it, but in doing so the non-master will be moved so that it loads before the light master. The sorting algorithm doesn't currently handle this case, and will instead produce a cyclic interaction error.

A fix would probably require a little pre-processing to get the non-masters that any ESLs depend on, and flag them so they're treated as masters by the rest of the sorting algorithm.

Use C++17's std::filesystem::path to represent paths

This is a possibility (along with #14) now that MSVS 15.7 and GCC 8 have brought good support for C++17.

This would swap the Boost.Filesystem library for the standard library implementation, and the API could expose usage of std paths. Updating to MSVC 2017 would be a breaking change as a new runtime is required, so it makes sense to break the API at the same time.

The standard library implementation needs to be checked to ensure that opening a file stream given a std path will work when using UTF-8-encoded non-ASCII paths, as Unicode path handling is a big reason why LOOT uses Boost.Filesystem paths throughout.

Also, AppVeyor haven't yet updated to VS 15.7, but they're usually pretty quick with updates.

Issue with Regex Capture Groups

Having an issue with regex, capture groups don't seem to be working.

Could be causing these issues:
loot/loot#1077
https://github.com/loot/loot/issues/1074#issuecomment-455860099

I've tried the following

active("RUSTIC SOULGEMS - (Sorted|Unsorted)\.esp") and not active("GIST-RusticSoulGems-patch.esp")

active("(RUSTIC SOULGEMS - (Sorted|Unsorted))\.esp") and not active("GIST-RusticSoulGems-patch.esp")

active("RUSTIC SOULGEMS - (Sorted|Unsorted)?.*\.esp") and not active("GIST-RusticSoulGems-patch.esp")

active("RUSTIC SOULGEMS - (Sorted|Unsorted).*\.esp") and not active("GIST-RusticSoulGems-patch.esp")

active("(RUSTIC SOULGEMS - (Sorted|Unsorted)).*\.esp") and not active("GIST-RusticSoulGems-patch.esp")

active("(RUSTIC SOULGEMS - (Sorted|Unsorted))?.*\.esp") and not active("GIST-RusticSoulGems-patch.esp")

but the warning message is still displayed.

Used for reference.
https://docs.rs/regex/1.0.5/regex/#syntax

Used for testing.
https://regex101.com/r/fN7xaL/1

Speed up builds

Builds have gotten very slow again, due to the Rust dependencies introduced. The libraries themselves build in about a minute, but each library also builds its own copy of cbindgen to generate C++ headers with, and that takes the remaining time.

cbindgen only needs to be built once, which could be done by:

• building cbindgen separately and running it as a CLI utility for each Rust dependency, instead of as part of the dependencies' build process
• having a single Rust dependency that depends on each of the current Rust dependencies and re-exports their APIs.

The two approaches can be combined. The second has the advantage that any dependencies that are common between the Rust dependencies will also only be built once, but introduces another layer into the dependency tree, and is more complex. I don't think it's worth pursuing, at least for now, because the time saved will be relatively insignificant.

Provide edge source information in cyclic interaction errors

This is necessary for loot/loot#999. This would involve adding an attribute to edges that indicates what type of source they have, which could be an enum:

enum EdgeType {
  hardcoded,
  masterFlag,
  master,
  masterlistRequirement,
  userRequirement,
  masterlistLoadAfter,
  userLoadAfter,
  masterlistGroup,
  userGroup,
  overlap,
  tieBreak,
}

This would then be exposed through CyclicInteractionError through some kind of structure that gives the plugins in the cycle and the types of interaction between them.

Figure out how to expose metadata interfaces without exposing implementation

The API's handling of metadata structures is not ideal, because extending them with new fields is not a breaking change for the YAML metadata but is a breaking change for the API.

One option is to hide implementation detail using the pimpl pattern, another is to only expose metadata in serialised form and providing a schema to define the serialisation.

While the metadata is written in YAML because it's a good format for humans writing structured data, it's a relatively poor format for data transfer. There are also few if any good YAML schema parsers. I'm currently evaluating the following:

• JSON: it's pretty much everywhere, simple and has a schema language that's on the way to being standardised. However, there doesn't seem to be anything for C++ that will generate code given a schema.
• Protocol Buffers: it's pretty heavyweight compared to FlatBuffers, with no obvious advantages for LOOT's use case.
• FlatBuffers: it seems mature, has official implementations in C++, JavaScript and Python, and its schemas can be used to generate code to de/serialise an efficient binary format (and JSON in C++). It's also apparently possible to generate JSON schemas from FlatBuffer schemas.
• Cap'n Proto: It's very like FlatBuffers, but its only official implementation is C++, and its most recent release was v0.6 after two and a half years, so it seems like not as good a bet. It also includes an RPC implementation, which isn't necessary for LOOT.

Refactoring the util: statements to avoid copy-pasting

The Idea

As it stands, util: statements take the entire tool name:

- name: 'Something.esp'
  clean:
    - crc: 0x832B2C99 # v1.0
      util: 'SSEEdit 3.2.2'

This leads to copious amounts of copy-pasting, more than 500 times in the skyrim masterlists alone.
While the information given by the version itself is useful (3.2.2 vs 3.2.84, for example), the prelude (SSEEdit / TES5Edit) is already implied by the masterlist we're in (the skyrimse masterlist clearly implies SSEEdit, etc.).

The idea is to place a definition at the beginning of each masterlist that specifies the xEdit tool to be used:

- &xEdit
  name: SSEEdit
  display: '[SSEEdit](https://www.nexusmods.com/skyrimspecialedition/mods/164/)'

(the definition given above is just a mockup, the final one could obviously look different)

That way, the tools do not have to be hardcoded and future masterlists can specify their xEdit tool without any code changes.

The *dirtyPlugin, *quickClean and *noAutomaticFix messages would then be edited to accept just a version instead of the entire tool name in the util: field:

- name: 'Something.esp'
  clean:
    - crc: 0x832B2C99 # v1.0
      util: '3.2.2'

The tool name would be read from the *xEdit definition by LOOT and automatically prepended in front of the version given by util:.

Upsides & Downsides (with some rebuttals from the discussion below them)

Upsides

• Avoids copypasting in the future, as only one definition of the xEdit tool will have to exist now
• By extension, avoids typos in the future
• Allows the removal of more than 500 occurrences of the words SSEEdit, TES5Edit, etc.
  • That might not be a big enough removal to justify editing all master lists

Downsides

• Would have to be implemented within LOOT's code
• Each master list would have to be edited to add the &xEdit definition and to remove the tool from each occurrence of util:
  • The latter can be accomplished through a search and replace, as each util: statement starts with the same prelude
• xEdit would have to be updated to generate definitions with this new style
  • Elminster has already noted that this is no problem for him, as he has to change that part of xEdit regardless to add support for *quickClean