dfhack/df_misc

miscellanous stuff related to dwarf fortress hacking

DFHack Readme

DFHack is a Dwarf Fortress memory access library, distributed with scripts and plugins implementing a wide variety of useful functions and tools.

The full documentation is available online here. If you have DFHack installed, it is also accessible as raw text in the hack/docs folder. If you're an end-user, modder, or interested in contributing to DFHack -- go read those docs.

If the docs are unclear or you need more help, please check out our support page for ways to contact the DFHack developers.

df_misc's Issues

scan_twbt.rb doesn't work on osx

logging this as an issue so it's easier to track

the script works perfectly on windows and linux, but fails in two different ways on osx32 and osx64

osx32 has no results while osx64 has wrong results

DFHack/df-structures@59d4f82

/cc @jjyg @lethosor

ghidra importer needs to be updated to support new codegen types

DFHack/df-structures#622 added support for std::mutex (as stl-mutex) and std::condition_variable (as stl-condition-variable), and a pending update will add support for std::future<void> (as stl-future). the ghidra importer needs to be updated to understand these types and provide reasonable definitions for them to support analysis

codegen_c_hdr.pl generates invalid header

$ perl codegen_c_hdr.pl --linux ~/projects/DFHack/dfhack/library/include/df/codegen.out.xml 
Use of uninitialized value $tgst in string eq at codegen_c_hdr.pl line 560.
Use of uninitialized value $tgst in concatenation (.) or string at codegen_c_hdr.pl line 561.
Use of uninitialized value $tgst in string eq at codegen_c_hdr.pl line 560.
Use of uninitialized value $tgst in concatenation (.) or string at codegen_c_hdr.pl line 561.
    // TODO in poetic_form_part: struct stl_vector_global-struct-type line_subject_target;
        // TODO in T_location: struct stl_vector_global-struct-type deity_data;

Then when importing in IDA

Error /home/clement/src/df_misc/codegen.h,15526: Undefined type name 'stl_vector_'
Error /home/clement/src/df_misc/codegen.h,15542: Syntax error near: }
Error /home/clement/src/df_misc/codegen.h,15543: Syntax error near: }
Error /home/clement/src/df_misc/codegen.h,39038: Undefined type name 'stl_vector_'
Error /home/clement/src/df_misc/codegen.h,39039: Syntax error near: }

Here are the codegen xml for the two vectors with the invalid types.

            <ld:field ld:meta="container" ld:level="2" ld:subtype="stl-vector" name="potential_corrupt_target" ld:is-container="true">
                <ld:item ld:anon-compound="true" is-union="true" union-tag-field="potential_corrupt_skill" ld:level="3" ld:meta="compound" ld:typedef-name="T_potential_corrupt_target" ld:in-union="true">
                    <ld:field init-value="-1" name="LYING" ld:level="3" ld:meta="number" ld:subtype="int32_t" ld:bits="32"/>
                    <ld:field ref-target="historical_entity" name="INTIMIDATION" ld:level="3" ld:meta="number" ld:subtype="int32_t" ld:bits="32"/>
                    <ld:field ld:subtype="enum" type-name="value_type" base-type="int32_t" name="CONVERSATION" ld:level="3" ld:meta="global"/>
                </ld:item>
            </ld:field>

        <ld:field ld:meta="container" ld:level="1" ld:subtype="stl-vector" name="interrogate_status" since="v0.47.01" ld:is-container="true">
            <ld:item ld:subtype="bitfield" ld:level="2" ld:meta="compound" ld:typedef-name="T_interrogate_status">
                <ld:field name="interview_scheduled" ld:level="3" ld:meta="number" ld:subtype="flag-bit" ld:bits="1"/>
                <ld:field name="already_interviewed" ld:level="3" ld:meta="number" ld:subtype="flag-bit" ld:bits="1"/>
            </ld:item>
        </ld:field>

Versions:

• DFHack: 7809c6094afe98037ef20ef5ff3d816070d9564c
• df-structures: e26ec29d29be0f8a3dab5df583e05f6875f11a64
• IDA: 7.0.191002
• perl: 5.32.1-474 (from fedora 34)

ghidra: some 16-bit enums should have 32-bit versions

There's any number of enums (e.g. job_type) that we have defined as 16-bit but which are often handled internally as 32-bit. Having them defined only as 16-bit enums makes marking up decompiled code harder.

import_df_structures should therefore probably create a 32-bit version for any enum that is officially sized as 16-bit.

df_import_structures does not properly adduce minimum sizes of enums with negative values

df_import_structures identifies the building_type enum as requiring 32 bits even though it can fit in 16 (indeed probably 8) because the enum includes a value NONE with value -1. the enum imported into ghidra encodes this value as 0xffffffff instead of as -1 (which itself isn't necessarily wrong) and thus concluded from this that the enum required 4 bytes (which is)

stl-map support in `import_df_structures.java`

adding stl-map to codegen (DFHack/df-structures#502, DFHack/df-structures#510) means that import_df_structures.java needs to be updated to do something appropriate when it encounters this tag in codegen.xml

`import_df_structures.java` clobbers any datatype assigned to `texture`

texture ought to be assigned texture_handlerst by the importer because it's specified as a typed global in df-structures, but instead not only does this not happen, but the importer removes any type previously assigned to texture

this doesn't appear to be happening to other globals (if it is, i haven't noticed it) so there must be an edge condition of some sort

import_df_structures should import comments and initial values

When importing into Ghidra, structure definitions should add any comment attributes to the corresponding structure fields; additionally, any init-value attributes should be included there as well.

This would be especially useful when realigning structures against a new version, because some fields can be conclusively identified by their initial values.

See DFHack/dfhack#2489.

add global size checking to ghidra importer

when an extended global table is detected, compare the size in the extended table with the size imported from codegen, and issue a diagnostic if they are different

codegen_c_hdr.pl doesn't understand vectors of unions

This script has never properly supported vectors of unions, such as the "line_subject_target" field in the structure "poetic_form_part" - a recent change to DFHack/df-structures made said union externally-defined, causing the script to emit a TODO comment instead of a syntax error that I could track down and manually fix.

There are a few other things that need to be fixed as well - the size of stl-fstream on 64-bit Windows is now 280 bytes (int64_t[35]), and $vecpad should have been removed entirely since vectors aren't padded in 32-bit or 64-bit Windows (ever since Toady switched to Visual C++ 2015).

automate new-release ingest

about 95% of the process of generating symbols.xml for a new DF release is currently being handled with scan_vtable.rb and dumpdf_globals.rb

the remaining 5% should also be automated if possible

automatically extracting the PE header timestamp should be easy

automatically identifying the locations of steam_mod_manager and game_extra may prove more difficult but we may be able to roll back the whole gamest split after 50.08 rendering this point moot

and assembling the collected bits into a symbols.xml stanza is obviously trivial

ghidra: df_import_structures does not appear to handle base-type on enum instances

For example, intrigue_corruption::offered_relationship is an enum of type vague_relationship_type with an overridden base type of int32_t but the type is imported into ghidra as 2 bytes long, matching the underlying type of the enum but ignoring the override.

the specific xml for this example is:
<enum name='offered_relationship' type-name='vague_relationship_type' base-type='int32_t'/>

this should result in a 4-byte allocation for that field, but in the structure actually generated by df_import_structures the allocation is 2 bytes. in this particular instance, padding insertion prevents an issue, but this is happenstance

add support for `std::function` to ghidra importer

once this has been done in df-structures XML (ref: DFHack/df-structures#654)