daemonengine/urcheon

An asset builder and package manager for Dæmon based games

urcheon's Issues

add a -pakpath equivalent

When I run urcheon from within map-parpax_src.dpkdir (with PATH="$PATH:/opt/netradiant" urcheon build to work around #28), then q3map2 does not locate any of the shared texture packages:

--- InitPaths ---
VFS Init: /home/eris/src/map-parpax_src.dpkdir

--- BSP ---

The map then looks like this:

Note that the only light comes from shaders that are inside map-parpax_src.dpkdir, as all other light shaders could not be loaded. The weird tiling is a result of q3map2 not finding the texture files.

How do I make Urcheon pass additional paths to q3map2?

rewrite the file detection using a tree

to avoid detection collision

Pillow doesn't properly support 8-bit grayscale PNG

When loading 8-bit grayscale PNGs with Pillow, and converting them to RGB, they are converted like if they were 1-bit PNGs.

So basically, white,lightgrey,midgrey,darkgrey,black becomes white,white,white,white,black.

For now I'm implementing a workaround by converting all PNGs to lossless WebP as a transient format because cwebp is known to properly convert PNG to lossless WebP before loading the bitmap with Pillow.

For example I do PNG to WebP to TGA to convert from PNG to TGA.

This becomes crazy because we already did PNG to TGA to CRN to convert from PNG to CRN because crunch doesn't read all PNG formats. So, since we cannot do PNG to TGA in one go, I now do PNG to WebP to TGA to CRN to convert from PNG to CRN. This is really crazy!

Some code may now be highly unoptimized like PNG to WebP since the sanitization of the image is converted first to WebP before being converted to WebP again.

I plan to introduce a specific ImageConverter module to handle image conversion properly with the “shortest” (ahem) path for known formats.

This workaround is good enough to build Unvanquished 0.54 release.

---

Note: in the past we used ImageMagick convert tool to convert from anything to TGA but then ImageMagick broke TGA conversion (both conversion from TGA and to TGA) and maintainer is considering there is no bug, so this other tool is not an option anymore.

Something is wrong with PNG…

When building using -r (since reference), unmodified moved files are untracked.

When building using -r (since reference), unmodified but moved files are untracked.

pak_mallet does not convert compiled external lightmaps

pak_mallet does not convert compiled external lightmaps

Error needed when trying to build a delta pak from an unknown reference

Error needed when trying to build a delta pak from an unknown reference, it still builds something with a DEPS referencing a package version that is likely to not exist.

make generated files (lightmap, minimap…) same date source map

make generated files (lightmap, minimap…) same date source map, not only the bsp

add support for an env variable setting a TMPFS build dir

add support for an env variable setting a TMPFS build dir, to speed up build tasks and save SSD lifetime.

rebuild everything from action file if action file modified

TODO: rebuild everything from action file if action file modified

rewrite "script" as "tool" in readme

rewrite "script" as "tool" in readme.

port pak_profiles to a non-python format like TOML

to allow people to write profiles in their own packages easily, to allow a build server to read them safely.

import lightmaps from a directory

bsp_cutter.sh is able to dump embedded lightmaps to a directory, it needs to be able to import them from a directory to embed them inside the lightmaps bsp lump.

This behavior is needed to be able to convert a .bspdir directory to a .bsp file.

add a network module to send and receive build tasks

add a network module to send and receive build tasks, to parallelize asset conversion across the network.

map compilation: light, minimap and navmesh stages can be done in parallel

map compilation: light, minimap and navmesh stages can be done in parallel

Do not copy files when q3map2 fails

Currently, Urcheon copy (incomplete) files when q3map2 fails.

In the following example, q3map2 faced a segfault but produced files were copied.

--- IlluminateRawLightmap ---
0...1...2...3...4...5...6...7.
Error: command failed: 'q3map2' '-game' 'unvanquished' '-v' '-light' '-fast' '-fastallocate' '-shade' '-dirty' '-patchshadows' '-samples' '3' '-samplesize' '8' '-bouncegrid' '-bounce' '16' '-bouncescale' '2' '-deluxe' '-lightmapsize' '1024' '-external' '-fs_nobasepath' '-fs_nohomepath' '-fs_nomagicpath' '-fs_pakpath' '/home/illwieckz/Mapping/Unvanquished/UnvanquishedAssets/src/map-chasm_src.dpkdir' '-fs_pakpath' '/home/illwieckz/Mapping/Unvanquished/UnvanquishedAssets/src' '-srffile' '/mnt/tmpfs/tmptuysep0j_q3map2.srf' '-bspfile' '/mnt/tmpfs/tmprmt00gx7_chasm.bsp_transient.dir/maps/chasm.bsp' '-lightmapdir' '/mnt/tmpfs/tmprmt00gx7_chasm.bsp_transient.dir/maps/chasm' '/home/illwieckz/Mapping/Unvanquished/UnvanquishedAssets/src/map-chasm_src.dpkdir/maps/chasm.map'
minimaps/chasm.minimap: MiniMap sidecar found, will copy file.
minimaps/chasm.tga: MiniMap image found, will convert to crn format.
maps/chasm-level0.navMesh: Navigation Mesh found, will copy file.
maps/chasm-human_naked.navMesh: Navigation Mesh found, will copy file.
maps/chasm-human_bsuit.navMesh: Navigation Mesh found, will copy file.
maps/chasm-level3upg.navMesh: Navigation Mesh found, will copy file.
maps/chasm-level4.navMesh: Navigation Mesh found, will copy file.
maps/chasm-level3.navMesh: Navigation Mesh found, will copy file.
maps/chasm-level1.navMesh: Navigation Mesh found, will copy file.
maps/chasm-level1upg.navMesh: Navigation Mesh found, will copy file.
maps/chasm-builder.navMesh: Navigation Mesh found, will copy file.
maps/chasm-level2.navMesh: Navigation Mesh found, will copy file.
maps/chasm-level2upg.navMesh: Navigation Mesh found, will copy file.
maps/chasm.bsp: BSP File found, will keep file.
maps/chasm.map: Map found, will keep file.
Convert to crn: minimaps/chasm.tga
Copy: minimaps/chasm.minimap
Copy: maps/chasm-level0.navMesh
Copy: maps/chasm-human_naked.navMesh
Copy: maps/chasm-human_bsuit.navMesh
Copy: maps/chasm-level3upg.navMesh
Copy: maps/chasm-level4.navMesh
Copy: maps/chasm-level3.navMesh
Copy: maps/chasm-level1.navMesh
Copy: maps/chasm-level1upg.navMesh
Copy: maps/chasm-builder.navMesh
Copy: maps/chasm-level2.navMesh
Copy: maps/chasm-level2upg.navMesh
Keep: maps/chasm.bsp
Keep: maps/chasm.map
translating DEPS for testing
clean dust file: maps/chasm/lm_0003.webp
clean dust file: maps/chasm/lm_0002.webp
clean dust file: maps/chasm/lm_0001.webp
clean dust file: maps/chasm/lm_0000.webp

do not keep deleted files in build dir

• build new test pk3dir into temporary directory
• copy from older one if latest
• delete older pk3dir
• rename temporary pk3dir as test pk3dir

build partial package since commit or tag

• build partial package since commit or tag

use argparse in bsp_cutter.py like in map_cutter.py

use argparse in bsp_cutter.py like in map_cutter.py

Export preview editor images with some lowres lossy webp instead of lowres jpg

Export preview editor images with some lowres lossy webp instead of lowres jpg, this to keep alpha channel (example: blood materials in res-ambient).

splitlines are not safe

If a binary file is opened instead of a plain text file, it fails at splitlines(), so it must be checked.

use ordered dict to handle entity key-value pairs

Current implementation in bsp_cutter use unordered python dictionaries, so each time we write down an “entities” lump, the file is different even if nothing changed (so, user can't use diff to compare them for example).

add a version file in bspdir

this way merge_bsp knows which format to merge to.

Implement `urcheon init`

So that's yet another thing I want to implement in Urcheon since the very beginning but that always got de-prioritized because of other things like things actually needed to build a release.

The idea is that someone could do this:

mkdir package_src.dpkdir
cd package_src.dpkdir
urcheon --game unvanquished init

Or:

urcheon --game unvanquished init package_src.dpkdir

This would set up the basic configuration of an urcheon-driven dpkdir, here it's just about writing unvanquished in package_src.dpkdir/.urcheon/game.txt and may be write a default .gitignore.

This would also be doable:

urcheon --game unvanquished init --git package_src.dpkdir

This would create the dpkdir folder, write the configuration, call git init and commit the basic things.

For collections (like UnvanquishedAssets or InterstellarOasis parent directories), one would just do that:

mkdir MyAssets
cd MyAssets
urcheon init --collection

or

urcheon init --collection MyAssets

And of course that would be possible as well:

urcheon init --collection --git MyAssets

Writing the game name in the collection configuration would allow to not have to retype it when creating dpkdirs.

So a complete git-based scenario would be:

urcheon --game unvanquished init --collection --git MyAssets
cd MyAssets
urcheon init --git src/package0_src.dpkdir
urcheon init --git src/package1_src.dpkdir
urcheon init --git src/package2_src.dpkdir

This would be the same as:

mkdir MyAssets
cd MyAssets
urcheon --game unvanquished init --collection --git
urcheon init --git src/package0_src.dpkdir
urcheon init --git src/package1_src.dpkdir
urcheon init --git src/package2_src.dpkdir

add cached conversion

use cached file if it was already converted in the past

• for pictures, checksum the rgba data
• for sounds, checksum the wav

Regression in argument parsing with `--version-suffix '~n'`

Since a few weeks ago, Nightly is broken. The command below used to work, but now gives a help message saying… that I should do the exact same thing as I did?

$ urcheon package --version-suffix '~n' src/map-antares_src.dpkdir src/map-chasm_src.dpkdir src/map-forlorn_src.dpkdir src/map-parpax_src.dpkdir src/map-perseus_src.dpkdir src/map-plat23_src.dpkdir src/map-spacetracks_src.dpkdir src/map-station15_src.dpkdir src/map-thunder_src.dpkdir src/map-vega_src.dpkdir src/map-yocto_src.dpkdir src/res-ambient_src.dpkdir src/res-buildables_src.dpkdir src/res-legacy_src.dpkdir src/res-players_src.dpkdir src/res-soundtrack_src.dpkdir src/res-voices_src.dpkdir src/res-weapons_src.dpkdir src/tex-all_src.dpkdir src/tex-common_src.dpkdir src/tex-ej01_src.dpkdir src/tex-ex_src.dpkdir src/tex-exm_src.dpkdir src/tex-pk01_src.dpkdir src/tex-pk02_src.dpkdir src/tex-space_src.dpkdir src/tex-tech_src.dpkdir src/tex-trak5_src.dpkdir src/tex-vega_src.dpkdir
usage: urcheon [-h] [-D] [-v] [-l] [-g GAMENAME] [--build-prefix DIRNAME] [--test-prefix DIRNAME] [--pak-prefix DIRNAME] [--test-dir DIRNAME] [--pak-file FILENAME]
               [--version-suffix STRING] [-np]
               {discover,prepare,build,package,clean} ...
urcheon: error: unrecognized arguments: --version-suffix

README: The urcheon examples have to be moved before the detailed explanations about stages

I just read that:

<freem> to use Urcheon is like asking a dev to fully understand the stages of the build process
        and to do them manually, one after the other
<freem> that's what the doc suggests at least
<afontain> it's a bit like cmake && make && make install
<freem> the doc does not makes this obvious

There is no need to fully understand the stages to follow the examples, but displaying the stage documentation before the examples can give the false impression this is actually needed to understand the examples.

Also some things can be updated, Urcheon evolved a lot since that README has been written.

while converting some jpg files to crn, they end upside down

I discovered the bug while toying with metro and cruz Tremulous map (trying to port them to Unvanquished) and I noticed some textures were upsede-down. I first tought it may be a bug in engine in the crunch tool, then after some investigations I discovered it is a bug in the convert tool by the ImageMagick project.

See https://bugs.launchpad.net/ubuntu/+source/imagemagick/+bug/1838860

You can have a look at the bug there:

Image samples that are known to fail can be found there:
https://dl.illwieckz.net/b/imagemagick/bugs/convert-jpg-to-tga-upside-down/convert-jpg-to-tga-upside-down-samples.zip

Basically, the urcheon tool converts using convert any textures to a tga file before feeding crunch, this allows urcheon to convert any image format to crn as long as imagemagick groks the format, even if crunch itself does not support the given format.

I used the tga format as a transient format because in the past I discovered that crunch had some trouble with some png file. Since convert is known to convert properly those jpg to png then those png to tga, we may use a png transient file to produce the tga file, that would make Urcheon doing:

jpg → png → tga → crn

This to be sure to workaround all the bugs from any software stack…

What a mess!

pak_mallet does not handle empty lines in paklist

pak_mallet does not handle empty lines in paklist

Delta pak DEPS rewriting don't work with config file-based versionning.

Delta pak DEPS rewriting don't work with config file-based versioning.

We would have to read the version in .pakinfo from the given commit for that.

Using git repositories ist still the recommended way to fully unlock Urcheon abilities anyway.

add a generic patch command

When bspdir will be usable (see #1) add a generic patch command to merge all existing files from bspdir into the bsp file.

allow to ignore bspdir sub files

do not merge it if there is an ignore rule on it even if there is other files to merge.

DEPS is not deleted on test dir if deleted on source dir

DEPS is not deleted on test dir if deleted on source dir.

add "bootstrap" option to (re)build common makefiles

• for standalone pakdir
• for multiple pakdir parents

add a tool to package maps

why not naming it pk3_hammer.py ?

Urcheon does not delete files that were deleted but not committed in source dir

Urcheon does not delete files that were deleted but not committed in source dir, or just not built by Urcheon.

It means a temporary file that never get committed but has been built once can be mistakenly packaged.

See also:

• #25

Urcheon does not fail properly when the map compiler cannot be located.

When I run urcheon build I see the following lines in quick succession:

Building /home/eris/src/map-parpax_src.dpkdir/maps/parpax.map, stage: vis
Building /home/eris/src/map-parpax_src.dpkdir/maps/parpax.map, stage: nav
Building /home/eris/src/map-parpax_src.dpkdir/maps/parpax.map, stage: minimap
Creating MiniMap for: /home/eris/src/map-parpax_src.dpkdir/maps/parpax.map
Building /home/eris/src/map-parpax_src.dpkdir/maps/parpax.map, stage: light

This is immediately followed by FileNotFoundError: [Errno 2] No such file or directory: '/home/eris/src/map-parpax_src.dpkdir/build/test/map-parpax_test.dpkdir/maps/parpax.bsp'.

All is fine when I make q3map2 available by running PATH="$PATH:/opt/netradiant" urcheon build instead. Urcheon should give a proper error message and terminate immediately when the map compiler cannot be found.

Mistake in Urcheon/Pak.py when processing list of built files

I spotted this in Urcheon/Pak.py:

			head = unit["head"]
			body = unit["body"]
[…]
			# if multiple calls produce the same files (like merge_bsp)
			if head in unit:
				continue

That can't work, this is probably a leftover or we may have to do if head in body instead.

compute pak version on git history

• use tag if latest commit is tagged
• use tag+date~commit if there is commits since last tag
• use 0+date~commit if there is no tag at all in the whole history

date must be the commit date

Mistake in q3legacybrush_plane_pattern pattern?

According to all docs I've read (including the comment here) rotation is before scale but the pattern puts rotation after scale. I haven't yet used the parsed data so can't confirm, just don't want this to be forgotten.

add a tool to handle layout files

to convert from bsp / map entitites to layout files and vice versa.

use a bspdir

Currently, the bsp_cutter dump entities and textures lumps as files alongside the bsp file, and lightmaps like that:

mapname.bsp
mapname.textures
mapname.entities
mapname/lm_0000.tga

For the future, we must do this:

mapname.bsp
mapname.bspdir/textures.csv
mapname.bspdir/entities.txt
mapname.bspdir/lightmaps/lm_000.tga

…and add command line option to be able to dump elsewhere

compute pak version from all pakdir in PAKPATH

and rewrite versions in DEPS on the fly

note: keep also in DEPS the reference from the past on current pakdir on partial build

make it optional (opt-out?)