"You should have used WADPTR" - unhappily dshrunk fraggle
fragglet / wadptr Goto Github PK
View Code? Open in Web Editor NEWWAD compressor
Home Page: https://soulsphere.org/projects/wadptr
License: GNU General Public License v2.0
WAD compressor
Home Page: https://soulsphere.org/projects/wadptr
License: GNU General Public License v2.0
"You should have used WADPTR" - unhappily dshrunk fraggle
Note that this is specifically v0.99i9, no longer available(?), error does not occur on latest v1.6. I have attached the WAD.
Adding: WIP2 Squashing Column 7 overruns the end of the lump with no 0xff terminating byte
Some tools allow controlling the verbosity level by a number following the -v
switch. I'd like such a feature because when logging to a text file, you don't want the percentage progress, just the lump names. Secondly, some users may want an option where only the final result is shown and not the list of lumps.
https://www.doomworld.com/forum/topic/61701
Adding: D_RUNNIN Storing (0%), done.
Adding: MAP01 Empty (0%).
Adding: THINGS Storing (0%), done.
Adding: LINEDEFS Deferred... (0%)
Adding: SIDEDEFS Packing (-1%), done.
Adding: VERTEXES Storing (0%), done.
Adding: SECTORS Storing (0%), done.
Merging identical lumps... done.
*** __planisf2 - Copy.wad is -8910 bytes smaller (0%) ***
Notice how SIDEDEFS packing goes to -1%!
File sizes:
https://www.doomworld.com/forum/topic/115093-e1m1-multiverse-release-24/
https://www.doomworld.com/idgames/levels/doom2/Ports/m-o/multiverse
Adding: LINEDEFS Deferred... (0%)
Adding: SIDEDEFS PackingRebuildSidedefs: LINEDEFS lump (#1244) is 4634 bytes, not a multiple of 16
I get this error every time I try to use wadptr; Windows 10 x64. Doesn't matter which wad I'm trying to compress, it always fails this way.
An example level is Eviternity MAP32, which is close to the SIDEDEFS limit even when they are compressed (59483 sidedefs). wadptr will happily decompress the level and generate a corrupted map. We should add the appropriate checks to check we never overflow, like we do when decompressing BLOCKMAPs.
$ ./wadptr -u evimap32.wad
Adding: MAP32 Storing, done.
Adding: THINGS Storing, done.
Adding: LINEDEFS Deferred...
Adding: SIDEDEFS Unpacking, done.
Adding: VERTEXES Storing, done.
Adding: SEGS Storing, done.
Adding: SSECTORS Storing, done.
Adding: NODES Storing, done.
Adding: SECTORS Storing, done.
Adding: REJECT Storing, done.
Adding: BLOCKMAP Unstacking, failed.
Some BLOCKMAP lumps could not be unstacked because the decompressed
version would exceed the vanilla BLOCKMAP size limit.
$ deutex -wadir evimap32.wad
i AA17 Doom directory is /home/fraggle/doom/doom2-1.9
i WR19 Reading WAD ./doom.wad: (2306 entries)
i WR19 Reading WAD evimap32.wad: (11 entries)
i ID10 PWAD entry identification...
Listing of WAD directory for evimap32.wad
Entry Size Type
MAP32 0 Level Map 32
THINGS 24200 Level Map 32
LINEDEFS 845250 Level Map 32
SIDEDEFS 3340890 Level Map 32
VERTEXES 203224 Level Map 32
SEGS 0 Level Map 32
SSECTORS 0 Level Map 32
NODES 3733415 Level Map 32
SECTORS 187382 Level Map 32
REJECT 0 Level Map 32
BLOCKMAP 205490 Level Map 32
i AA99 Normal exit
$ python3
Python 3.11.5 (main, Aug 29 2023, 15:31:31) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 3340890 // 30
111363
>>>
I assumed CPUs were fast enough that this wasn't a huge issue nowadays, but when investigating #5, it really took a looooong time trying to compress Evi MAP32. Ultimately we are using an O(nĀ²) algorithm and this can be improved.
I wasn't aware of this when I wrote wadptr, and it can save quite a lot of space in large levels. Some nodebuilders already do it automatically.
https://www.doomworld.com/forum/topic/91306
https://www.doomworld.com/idgames/levels/doom2/deathmatch/Ports/megawads/32in2416
Adding: SIDEDEFS PackingRebuildSidedefs: LINEDEFS lump (#99) is 6594 bytes, not a multiple of 16
(This is with updated binary from the latest commit)
While I think that not packing special lines is good default behaviour, this can also be used to get some neat effects, so it would be nice if there were a switch to allow them to be packed, and the level designer can adjust their map to accommodate this. I'm not aware of any other currently-maintained sidedef compression tools for Doom or other convenient/automated ways to do this, and it would be a bit silly to have to keep two versions of wadptr around.
This is a WAD for DOOM 64 EX:
https://www.doomworld.com/forum/topic/102849-beta-64-a-31-level-megawad-for-doom64-ex/
Adding: MAP32 Empty (0%).
Adding: THINGS Storing (0%), done.
Adding: LINEDEFS Deferred... (0%)
Adding: SIDEDEFS PackingRebuildSidedefs: SIDEDEFS lump (#3) is 12768 bytes, not a multiple of 30
wadptr -u *.wad
and for d in *.wad; wadptr -u $d; done
don't produce the same result, and the resulting WADs seem to get corrupted BLOCKMAP lumps.
commit 3ddf4b7 added this optimization to graphic squashing and we can do the same thing for BLOCKMAPs to make prefix matching work better
Also discovered when investigating #5. If sidedef packing doesn't produce a smaller result than the original, we should just use the original.
$ ./wadptr -c evimap32.wad
Adding: MAP32 Storing (0%), done.
Adding: THINGS Storing (0%), done.
Adding: LINEDEFS Deferred... (0%)
Adding: SIDEDEFS Packing (-2%), done.
Adding: VERTEXES Storing (0%), done.
Adding: SEGS Storing (0%), done.
Adding: SSECTORS Storing (0%), done.
Adding: NODES Storing (0%), done.
Adding: SECTORS Storing (0%), done.
Adding: REJECT Storing (0%), done.
Adding: BLOCKMAP Stacking (0%), failed.
Merging identical lumps... done.
*** evimap32.wad is 0% smaller ***
I'm running wadptr through doomkid's "Big Vanilla Wad Pack"; 2.2 GB of vanilla and limit-removing WADs. You'll probably want to download this yourself and run through it -- just search inside the folder for EGYPT.WAD
https://www.doomworld.com/forum/topic/114671
Adding: PLATFORM Squashing Column 0 overruns the end of the lump with no 0xff terminating byte
Even though I've seen this error message before, this is using the latest binary so perhaps is a different root cause?
https://www.doomworld.com/idgames/levels/doom2/Ports/d-f/entropy
Adding: BLOCKMAP Stacking BLOCKMAP lump too short: 0 < 4 header size
https://www.doomworld.com/idgames/themes/TeamTNT/daedalus/daedalus
Adding: BLOCKMAP Stacking BLOCKMAP lump too short: 0 < 4 header size
Discussed here on Doomworld.
If multiple scrolling linedefs share a sidedef, the normal scrolling speed gets multiplied by the number of linedefs because of Doom's internal data structures. It should be possible to avoid this by flagging sidedefs used by a scrolling linedef and excluding them from merge.
This should cover the original 48 scroll left type, plus also the Boom 85 scroll right type and 255 scroll-by-offsets type:
https://github.com/fragglet/mbf/blob/48b629dda9e1fc07c035ce5558fc4cd67b86f2ff/p_spec.c#L2672
This could be used to make sidedef packing more effective
Boom and other source ports support extensions to the SIDEDEFS and BLOCKMAP limits to make use of the full 16-bit range (vanilla uses signed integers, only giving 15-bits). So we should add options to support these.
For automation purposes, an unexpected input prompt is not good. WadPtr will ask confirmation for IWADs. Does quiet mode (-q
) remove this and what default does it take? (presumably, "no") I'd like an option to force this to "yes", e.g. -y
so that all IWADs are processed without confirmation.
https://www.doomworld.com/forum/topic/98149-cryogenics-a-small-collection-of-random-cl9-slaughterish-maps-now-on-idgames/
https://www.doomworld.com/idgames/levels/doom2/Ports/a-c/cryogenics
Adding: MAP01 Empty (0%).
Adding: THINGS Storing (0%), done.
Adding: LINEDEFS Deferred... (0%)
Adding: SIDEDEFS Packing (62%), done.
Adding: VERTEXES Storing (0%), done.
Adding: SEGS Empty (0%).
Adding: SSECTORS Empty (0%).
Adding: NODES Storing (0%), done.
Adding: SECTORS Storing (0%), done.
Adding: REJECT Empty (0%).
Adding: BLOCKMAP Stacking BLOCKMAP lump too short: 0 < 4 header size
A declarative, efficient, and flexible JavaScript library for building user interfaces.
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ššš
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ā¤ļø Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.