DeltaPlugin and Habasi merged plugins

[benjaminwinger]

DeltaPlugin and Habasi merged plugins

[alvazir]

• I am sorry for my mistake. I suppose I was over enthusiastic when checking utilities in the pre-publishing fever. I saw tes3cmd and TES3Merge(? I'm not sure now :-/) work with merged plugins and probably skipped or misinterpreted DeltaPlugin check results. I will change relevant descriptions to something like that:
  "DeltaPlugin processes records the same way as both engines, e.g. discards different variants of mergeable records except the last one. To be able to use both utilities consider using additional openmw.cfg file ...<here goes description to run DeltaPlugin first with additional config, then Habasi>"

• I've looked at both crates actually. Both were easy to plug in. I've had some concerns with performance, but the main deciding factor was the license. I've based my licensing decision on rust's defaults. I'd probably use GPL next time because it looks great for programs. Libraries are a bit different. I didn't like being forced to change my program's license from MIT to GPL by using those crates :-\

[benjaminwinger]

No problem.

the main deciding factor was the license. I've based my licensing decision on rust's defaults. I'd probably use GPL next time because it looks great for programs. Libraries are a bit different. I didn't like being forced to change my program's license from MIT to GPL by using those crates :-\

Yes I suppose that's true. bsatool_rs is I think based heavily on the openmw BSA code, so its license follows openmw's. openmw-cfg I'd licensed GPL mostly because that's what I usually use, but for a small library I don't feel strongly about the license. I could make the bsa part of openmw-cfg a feature and re-license it MIT so that you don't pull in any GPL code if that feature isn't enabled.

I've had some concerns with performance

What sort of performance concerns? For the most part, I don't think it's particularly performance-intensive, with maybe the exception of the VFS lookups if you're doing a lot of work with the VFS (does it take a noticeable amount of time just to look up the plugin paths?). There's also a small but important optimization I'd included in the python version of that case insensitive lookup algorithm which never made it to openmw-cfg, so it could certainly be improved.

[alvazir]

Yes I suppose that's true. bsatool_rs is I think based heavily on the openmw BSA code, so its license follows openmw's. openmw-cfg I'd licensed GPL mostly because that's what I usually use, but for a small library I don't feel strongly about the license. I could make the bsa part of openmw-cfg a feature and re-license it MIT so that you don't pull in any GPL code if that feature isn't enabled.

That'd be awesome!

What sort of performance concerns? For the most part, I don't think it's particularly performance-intensive, with maybe the exception of the VFS lookups if you're doing a lot of work with the VFS (does it take a noticeable amount of time just to look up the plugin paths?). There's also a small but important optimization I'd included in the python version of that case insensitive lookup algorithm which never made it to openmw-cfg, so it could certainly be improved.

1. openmw-cfg
   As far as I remember it should be related to VFS. I've tested against ~600 plugins/1000-2000 directories. It took ~2 seconds on my machine. Parallel read_dir helped me to reduce the time to ~20ms.
2. bsatool-rs
   Bsatool-rs makes excessive calls when reading BSA metadata. It may be reduced ~20x. I may look and make PR to fix the issue. I've had a pretty specific request to read multiple BSA as fast as possible and so finished creating my own "lib" with the help of that awesome UESP page and linked Interntet Archive page.

[alvazir]

And done. Made PR to bsatool_rs with improved offset reading time.

[benjaminwinger]

As far as I remember it should be related to VFS. I've tested against ~600 plugins/1000-2000 directories. It took ~2 seconds on my machine. Parallel read_dir helped me to reduce the time to ~20ms.

That is kind of slow. I did a quick benchmark and for me it's running in about 100ms, but that's with a much smaller load order on a fast ssd. I tried on a networked hard drive and it increased to 1.7 seconds. But with parallel reads that went down to ~200ms (and 15ms on the ssd).

Still slower than what you're doing in jobasha because it's doing multiple passes, but I only have so much time right now. I think I'd rather write a generic find_files function than optimize just for plugins, that way other stuff can benefit (I think it would mostly just be groundcoverify via delta-plugin right now).

[alvazir]

My information may be slightly outdated as I used openmw-cfg v0.3.0 in May. I'd probably made something wrong when testing.

I've just checked and seen GPL or MIT license choice on oenmw-cfg v0.5.0 crate. That's awesome!

I suppose the issue may be closed.