"Previous override" support?

[focustense]

I just learned about Mutagen and have started working to convert a big pile of XEditLib P/Invoke code. One important thing that I haven't been able to find - though that doesn't mean it's not there, I might just be missing it - is the PO pointer that XEdit provides. I assume that the Parent property actually points to the master record, since there isn't an actual Master property.

Specifically, I use this to check for ITPO, which is important for my use case. I suppose I could do something like scan through the entire load order searching for the last previous record with the same key, but it seems inefficient by comparison... I have to do this on potentially several thousand records.

AFAICT the Equals operator already does a deep comparison, so all I need is an efficient way to find the PO record to compare to. Does it already exist and I'm missing the obvious, or is it not supported yet/ever?

[theSkyseS]

For finding PO you could use something like this snippet:

//this is some formlink that you can get either from record or from FormKeys
var formLink = ... 
var overrides = formLink.ResolveAll(state.LinkCache).Take(2).ToList();
//overrides then contains winning override at 0 position and previous override at 1st.

If you also want to know which override comes from which mod you can use ResolveAllContexts<> instead of ResolveAll.
You can find "FormKeys" here.

[Noggog]

Hey! Good question! Going to convert this issue to a discussion FAQ, so others can see it in the future.

Thanks @theSkyseS, you're right on the money. LinkCaches + ResolveAll is what you're looking for.

I've been filling out the docs, so I just took the opportunity to get the relevant ones complete:
https://github.com/Mutagen-Modding/Mutagen/wiki/LinkCache
https://github.com/Mutagen-Modding/Mutagen/wiki/Previous-Override-Iteration

[theSkyseS]

For finding a master mod you should use ResolveAllContexts call. It returns you a ModContext objects that contains fields:

• Record - the record itself;
• Parent - which is containing record if one exists;
• Modkey - which is what you looking for, e.g. Modkey of a mod the override came from.
  You can find article about it on wiki here. It's kinda unfinished but has some info.

[focustense]

I am already using ResolveAllContexts to get the previous override - see my snippet above. However, that seems like the least efficient way to find the master, which should be O(1) as the necessary information is already encoded in the form key, whereas ResolveAllContexts requires iteration to the end of the list. In xEdit, it's a single call to Records.GetMaster.

My ugly hack should not iterate any records other than the master, I'm just assuming there's a better way to write it - without changing the behavior.

[Noggog]

which is not necessarily the mod I started at

This phrasing confuses me. What is the mod you "started at"? Just a mod in the middle of the LO that isn't the master, and isn't the winning override? This isn't too typical of a pattern, that I know about. Usually people are either interested in:

• The winning override
• The original definition
• All overrides front to back.

I haven't seen a use case where someone wants to start in the middle arbitrarily and go from there. Are you nesting some resolves like this?

// Resolving once
foreach (var context in formLink.ResolveAllContexts<ISkyrimMod, ISkyrimModGetter, INpc, INpcGetter>(Environment.LinkCache))
{
      // Resolving twice, to find the previous override
      var overrides = formLink.ResolveAllContexts<ISkyrimMod, ISkyrimModGetter, INpc, INpcGetter>(Environment.LinkCache);
      var previousOverride = overrides.SkipWhile(x => x.ModKey != context.ModKey).Skip(1).FirstOrDefault();
      
      // Now i can process the current `context` with its `previousOverride`
}

This would not be the best pattern, if that's what you're doing. I'd recommend doing ONE ResolveAllContexts call, saving it to an array, which will give you the Length/Indexing power that can help you out:

var overrides = formLink
    .ResolveAllContexts<ISkyrimMod, ISkyrimModGetter, INpc, INpcGetter>(Environment.LinkCache)
    .ToArray();
// Iterate from end
for (int i = overrides.Length - 1; i > 0; i--)
{
    var currentRecordToLookAt = overrides[i];
    var previousOverride = overrides[i - 1];
    // Now i can process the current `context` with its `previousOverride`
}

But yeah, I think outlining what your end goal is would be helpful to know how to approach it.

---

So if I wanted to get the actual master... that can't be the best way.

Currently that would be very close to the best way. You can do a direct index instead of the GetIfEnabled call:
env.LoadOrder[weapon.FormKey.ModKey].Mod?.Weapons.TryGetValue(npcRecord.FormKey, out var npc);

But yeah, definitely a crazy call to get at the master.

I can see about adding a call to LinkCache to get at the master more easily. It would be offering some TryResolveOriginalDefinition or something which would match the patterns that already exist pretty well. Ill see if i can slip it into the next Mutagen prerelease (#205)

[focustense]

This is code that already works against XEditLib - I'm only trying to convert it to Mutagen's APIs.

The loading stage has to scan every NPC in every plugin. Any plugin could potentially define its own brand new NPCs, so there's no canonical source of FormIDs (or Form Keys) it can start at to work "down" rather than "up". It needs to know what every plugin is doing to every NPC, that's actually the point. Part of that process is to check for ITPO, another is to compare certain traits with the master record. ITPO is so we don't select a default that doesn't provide any new behavior (and consequently over-specify the required plugin masters), master record is to see if it should be treated as a customization/overhaul.

Because it needs this for all of the records in all of the plugins, regardless of position in the load order, there generally isn't going to be anything more efficient in terms of the overall algorithm than just iterating through all of the plugins and all of the records in those plugins. (I already use a separate cache for the data it reads.) It's what happens inside that main loop that's important.

I suppose I could store the actual master record or "context" in the cache for later reference, since the master is always going to be seen first when I iterate plugins themselves from top to bottom, and parts of the code already rely on exactly that assumption - but I'd rather not store direct references to Mutagen data, especially when they might have lazily-evaluated properties and when I'm in the middle of a refactoring effort when I might need to revert to the old plugin. I only want to store my own app data in long-lived objects, and since looking up a master record should be O(1) anyway, dumping it in an additional cache seems wasteful.

Of course I could also use ResolveAllContexts and store the entire array, as you suggest, but that's not really better - it still has to iterate the whole list in order to do that, and incur extra time/memory to store the results in an array. Very little extra time/memory, to be fair, but still. I guess this might fall under the category of premature optimization, I was just surprised I had to write something that ugly to do what was a single call (Record.GetMaster) in XEdit. But if that is indeed the most efficient call as things are today, then I'll just hide it away in a utility method somewhere so it's easy to clean up if someday there's a different mechanism.

Thanks!

[Noggog]

Cool! Yeah, hopefully I'll just be adding a formLink.TryResolveOrigin(linkCache, out var originatingRecord) pretty soon.

Also, could add a parameter to ResolveAll(Direction.ListedOrder) so that it would return the master first, and then enumerate to the winning override. Due to the way linkcaches are optimized, this would frontload a lot of work (it would still process everything in PriorityOrder internally), but the API would be more what you're after?

Also, perhaps AllIdentifiers would help here, too

// Loop over every npc identifier in the load order, no matter what mod it's from.
foreach (IMajorRecordIdentifier npcIdentifier in linkCache.AllIdentifiers<INpcGetter>())
{
    // Loop over every version of that record, starting with the originating master definition
    foreach (var npcRecord in linkCache.ResolveAll<INpcGetter>(npcIdentifier.FormKey, Direction.ListedOrder))
    {
         // Work
    }
}

Only missing part of that API would be the Direction.ListedOrder option on ResolveAll, and could maybe clean up the API a bit to not need to specify the <INpcGetter> on the internal loop