dfhack/df_misc

miscellanous stuff related to dwarf fortress hacking

df_misc's Issues

add global size checking to ghidra importer

when an extended global table is detected, compare the size in the extended table with the size imported from codegen, and issue a diagnostic if they are different

stl-map support in `import_df_structures.java`

adding stl-map to codegen (DFHack/df-structures#502, DFHack/df-structures#510) means that import_df_structures.java needs to be updated to do something appropriate when it encounters this tag in codegen.xml

automate new-release ingest

about 95% of the process of generating symbols.xml for a new DF release is currently being handled with scan_vtable.rb and dumpdf_globals.rb

the remaining 5% should also be automated if possible

automatically extracting the PE header timestamp should be easy

automatically identifying the locations of steam_mod_manager and game_extra may prove more difficult but we may be able to roll back the whole gamest split after 50.08 rendering this point moot

and assembling the collected bits into a symbols.xml stanza is obviously trivial

`specific_ref_type` is sometimes 16-bit rather than 32-bit

toady defines this as an enum without a specific base type so the underlying type is 32 bits, but he then does typedef int16_t Reference; and usually uses Reference so most of the time these are 16-bit values

we thus need to generate both a 16-bit and 32-bit version of this enum

ghidra: df_import_structures does not appear to handle base-type on enum instances

For example, intrigue_corruption::offered_relationship is an enum of type vague_relationship_type with an overridden base type of int32_t but the type is imported into ghidra as 2 bytes long, matching the underlying type of the enum but ignoring the override.

the specific xml for this example is:
<enum name='offered_relationship' type-name='vague_relationship_type' base-type='int32_t'/>

this should result in a 4-byte allocation for that field, but in the structure actually generated by df_import_structures the allocation is 2 bytes. in this particular instance, padding insertion prevents an issue, but this is happenstance

import_df_structures should import comments and initial values

When importing into Ghidra, structure definitions should add any comment attributes to the corresponding structure fields; additionally, any init-value attributes should be included there as well.

This would be especially useful when realigning structures against a new version, because some fields can be conclusively identified by their initial values.

See DFHack/dfhack#2489.

scan_twbt.rb doesn't work on osx

logging this as an issue so it's easier to track

the script works perfectly on windows and linux, but fails in two different ways on osx32 and osx64

osx32 has no results while osx64 has wrong results

DFHack/df-structures@59d4f82

/cc @jjyg @lethosor

df_import_structures does not properly adduce minimum sizes of enums with negative values

df_import_structures identifies the building_type enum as requiring 32 bits even though it can fit in 16 (indeed probably 8) because the enum includes a value NONE with value -1. the enum imported into ghidra encodes this value as 0xffffffff instead of as -1 (which itself isn't necessarily wrong) and thus concluded from this that the enum required 4 bytes (which is)

ghidra: some 16-bit enums should have 32-bit versions

There's any number of enums (e.g. job_type) that we have defined as 16-bit but which are often handled internally as 32-bit. Having them defined only as 16-bit enums makes marking up decompiled code harder.

import_df_structures should therefore probably create a 32-bit version for any enum that is officially sized as 16-bit.

codegen_c_hdr.pl generates invalid header

$ perl codegen_c_hdr.pl --linux ~/projects/DFHack/dfhack/library/include/df/codegen.out.xml 
Use of uninitialized value $tgst in string eq at codegen_c_hdr.pl line 560.
Use of uninitialized value $tgst in concatenation (.) or string at codegen_c_hdr.pl line 561.
Use of uninitialized value $tgst in string eq at codegen_c_hdr.pl line 560.
Use of uninitialized value $tgst in concatenation (.) or string at codegen_c_hdr.pl line 561.
    // TODO in poetic_form_part: struct stl_vector_global-struct-type line_subject_target;
        // TODO in T_location: struct stl_vector_global-struct-type deity_data;

Then when importing in IDA

Error /home/clement/src/df_misc/codegen.h,15526: Undefined type name 'stl_vector_'
Error /home/clement/src/df_misc/codegen.h,15542: Syntax error near: }
Error /home/clement/src/df_misc/codegen.h,15543: Syntax error near: }
Error /home/clement/src/df_misc/codegen.h,39038: Undefined type name 'stl_vector_'
Error /home/clement/src/df_misc/codegen.h,39039: Syntax error near: }

Here are the codegen xml for the two vectors with the invalid types.

            <ld:field ld:meta="container" ld:level="2" ld:subtype="stl-vector" name="potential_corrupt_target" ld:is-container="true">
                <ld:item ld:anon-compound="true" is-union="true" union-tag-field="potential_corrupt_skill" ld:level="3" ld:meta="compound" ld:typedef-name="T_potential_corrupt_target" ld:in-union="true">
                    <ld:field init-value="-1" name="LYING" ld:level="3" ld:meta="number" ld:subtype="int32_t" ld:bits="32"/>
                    <ld:field ref-target="historical_entity" name="INTIMIDATION" ld:level="3" ld:meta="number" ld:subtype="int32_t" ld:bits="32"/>
                    <ld:field ld:subtype="enum" type-name="value_type" base-type="int32_t" name="CONVERSATION" ld:level="3" ld:meta="global"/>
                </ld:item>
            </ld:field>

        <ld:field ld:meta="container" ld:level="1" ld:subtype="stl-vector" name="interrogate_status" since="v0.47.01" ld:is-container="true">
            <ld:item ld:subtype="bitfield" ld:level="2" ld:meta="compound" ld:typedef-name="T_interrogate_status">
                <ld:field name="interview_scheduled" ld:level="3" ld:meta="number" ld:subtype="flag-bit" ld:bits="1"/>
                <ld:field name="already_interviewed" ld:level="3" ld:meta="number" ld:subtype="flag-bit" ld:bits="1"/>
            </ld:item>
        </ld:field>

Versions:

• DFHack: 7809c6094afe98037ef20ef5ff3d816070d9564c
• df-structures: e26ec29d29be0f8a3dab5df583e05f6875f11a64
• IDA: 7.0.191002
• perl: 5.32.1-474 (from fedora 34)

ghidra importer needs to be updated to support new codegen types

DFHack/df-structures#622 added support for std::mutex (as stl-mutex) and std::condition_variable (as stl-condition-variable), and a pending update will add support for std::future<void> (as stl-future). the ghidra importer needs to be updated to understand these types and provide reasonable definitions for them to support analysis

add support for `std::function` to ghidra importer

once this has been done in df-structures XML (ref: DFHack/df-structures#654)

codegen_c_hdr.pl doesn't understand vectors of unions

This script has never properly supported vectors of unions, such as the "line_subject_target" field in the structure "poetic_form_part" - a recent change to DFHack/df-structures made said union externally-defined, causing the script to emit a TODO comment instead of a syntax error that I could track down and manually fix.

There are a few other things that need to be fixed as well - the size of stl-fstream on 64-bit Windows is now 280 bytes (int64_t[35]), and $vecpad should have been removed entirely since vectors aren't padded in 32-bit or 64-bit Windows (ever since Toady switched to Visual C++ 2015).

`import_df_structures.java` clobbers any datatype assigned to `texture`

texture ought to be assigned texture_handlerst by the importer because it's specified as a typed global in df-structures, but instead not only does this not happen, but the importer removes any type previously assigned to texture

this doesn't appear to be happening to other globals (if it is, i haven't noticed it) so there must be an edge condition of some sort