xoreos/xoreos-tools

Tools to help the development of xoreos

xoreos-tools README

A collection of tools to help with the reverse-engineering of BioWare's Aurora engine games. xoreos-tools is part of the xoreos project; please see the xoreos website and its GitHub repositories for details, especially the main README.md.

The tools included here are licensed under the terms of the GNU General Public License version 3 or (at your option) any later version.

Currently, the following tools are included:

• gff2xml: Convert BioWare GFF to XML
• tlk2xml: Convert BioWare TLK to XML
• ssf2xml: Convert BioWare SSF to XML
• fev2xml: Convert FMOD FEV to XML
• xml2gff: Convert XML back to BioWare GFF
• xml2tlk: Convert XML back to BioWare TLK
• xml2ssf: Convert XML back to BioWare SSF
• convert2da: Convert BioWare 2DA/GDA to 2DA/CSV
• fixpremiumgff: Repair BioWare GFF files in NWN premium module HAKs
• fixnwn2xml: Convert Obsidian NWN2 XML to valid XML
• unerf: Extract BioWare ERF archives
• unherf: Extract BioWare HERF archives
• unrim: Extract BioWare RIM archives
• unnds: Extract Nintendo DS roms
• unnsbtx: Extract Nintendo NSBTX textures into TGA images
• unkeybif: Extract BioWare KEY/BIF archives
• unobb: Extract Aspyr's OBB virtual filesystem
• untws: Extract CDProjectRed's TheWitcherSave archives
• erf: Create BioWare ERF archives
• rim: Create BioWare RIM archives
• tws: Create CDProjectRed TheWitcherSave archives
• keybif: Create BioWare KEY/BIF archives
• desmall: Decompress "small" (Nintendo DS LZSS, types 0x00 and 0x10) files
• xoreostex2tga: Convert BioWare's texture formats into TGA
• nbfs2tga: Convert Nintendo's raw NBFS images into TGA
• ncgr2tga: Convert Nintendo's NCGR images into TGA
• cbgt2tga: Convert CBGT images into TGA
• cdpth2tga: Convert CDPTH depth images into TGA
• ncsdis: Disassemble NWScript bytecode
• ncsdecomp: Decompile NWScript bytecode

CI Status

• 
• 
• 
• 
• 
• 
• 
• 

Getting xoreos-tools

You can get xoreos-tools in multiple ways:

You can download an archive with a binary of the latest release from our downloads page. This includes binaries for Microsoft Windows, Mac OS X and GNU/Linux, as well as packages for various GNU/Linux distributions. All of them are available for both 32- and 64-bit x86 architectures.

Or, if you're running Arch Linux, you can install xoreos-tools directly from the AUR.

Or, if you're running Gentoo Linux, you can install xoreos-tools directly from our overlay.

Lastly, you can compile xoreos-tools yourself; either from a release source package, found on our downloads page, or a fresh repository checkout. For details on how to compile xoreos on various operating system, please read the Compiling xoreos-tools page on our wiki.

TLK language IDs and encodings

Aurora games use numerical language IDs to identify which language a TLK file holds. Unfortunately, those language IDs vary between games, and so does the encoding used for strings in those TLK files. There is no way to autodetect this information, so it has to be provided to tools handling those files, in one way or another.

For the tools tlk2xml and xml2tlk, you can specify this encoding either directly, or by giving the game the TLK is from. Please note that this does not work for Sonic Chronicles: The Dark Brotherhood, because its TLK files do not provide a language ID.

Neverwinter Nights, Neverwinter Nights 2, Knights of the Old Republic, Knights of the Old Republic 2:

Language ID	Language	Encoding
0	English	CP-1252
1	French	CP-1252
2	German	CP-1252
3	Italian	CP-1252
4	Spanish	CP-1252
5	Polish	CP-1250
128	Korean	CP-949
129	Chinese (Traditional)	CP-950
130	Chinese (Simplified)	CP-936
131	Japanese	CP-932

Jade Empire:

Language ID	Language	Encoding
0	English	UTF-8
1	French	UTF-8
2	German	UTF-8
3	Italian	UTF-8
4	Spanish	UTF-8
5	Polish	UTF-8
6	Czech	UTF-8
7	Hungarian	UTF-8
130	Chinese (Simplified)	UTF-8
132	Russian	UTF-8

The Witcher:

Language ID	Language	Encoding
0	"Debug"	UTF-8
3	English	UTF-8
5	Polish	UTF-8
10	German	UTF-8
11	French	UTF-8
12	Spanish	UTF-8
13	Italian	UTF-8
14	Russian	UTF-8
15	Czech	UTF-8
16	Hungarian	UTF-8
20	Korean	UTF-8
21	Chinese (Traditional)	UTF-8
22	Chinese (Simplified)	UTF-8

Sonic Chronicles: The Dark Brotherhood:

Language ID	Language	Encoding
-	English	CP-1252
-	French	CP-1252
-	German	CP-1252
-	Italian	CP-1252
-	Spanish	CP-1252
-	Japanese	UTF-8

Dragon Age: Origins, Dragon Age II:

Language ID	Language	Encoding
0	English	UTF-16LE
1	French	UTF-16LE
2	Russian	UTF-16LE
3	Italian	UTF-16LE
4	German	UTF-16LE
5	Polish	UTF-16LE
6	Spanish	UTF-16LE
7	Czech	UTF-16LE
8	Hungarian	UTF-16LE
9	Korean	UTF-16LE
10	Japanese	UTF-16LE

Links

• xoreos website
• xoreos wiki
• Main source repository
• All xoreos repositories

Contact

To contact us, please either write to mailing list, or join our IRC channel #xoreos on Libera IRC.

xoreos-tools's Issues

UNOBB: Add support for KotOR2 Android obb archives

KotOR2 for Android from Aspyr is out. Our unobb tool doesn't handle those obb data files.

To recap: obb files are big data files used by Android apps. They don't have a set format; what's inside them is handled by the apk itself.

For the first KotOR game, Aspyr just used normal PKZIP files. For Jade Empire, they made them a virtual filesystem, with zlib compressed blocks of 4096 bytes.

Looking at the files in KotOR2, they seem to be still that virtual filesystem Aspyr introduced with Jade Empire. With a key difference: some files are not zlib compressed, but uncompressed. Might have been that Jade Empire already supports that too in principle, but that just no file there was uncompressed.

This has two implications for our unobb reader:

• The file identification fails, because we depend on there being a zlib header (78 9C)
• The compressed resource index list can't be found, since we depend on there being a compressed chunk beforehand

Manually ripping a resource list block from the KotOR2 obb files yields a zlib compressed resource list of the usual format.

This means, I need to revise our resource index list searching for obb files. Instead of searching backwards for a zlib header and then checking that the previous block metadata looks okay, why not just go by the metadata right at the end of the file (the last 16 bytes)? That should contain the offset of the resource list. Throw on inconsistent data there, and we don't need to check for the a zlib header at the start either.

Another thing I need to fix: check if compressed size == uncompressed size. If so, it's an uncompressed file. Read it directly instead of running it through zlib. Open question: what if the file is larger than 4096 bytes? Is there a block split somewhere or can we just read the whole thing as is?

There's also liblzma in the apk. At first I suspected Aspyr uses that in this obb format version, but I might be wrong. Open question: are all files in the obb either uncompressed or zlib? Or is there lzma too?

In either case, that looks like a nice little puzzle to solve/fix over the holidays for me :)

xml2gff not included in xoreos-tools-0.0.5 release?

Hi :)

Thanks for creating such an awesome set of tools!

I was just wondering whether it's intentional that xml2gff.exe is not included with xoreos-tools-0.0.5 on xoreos.org - the tool would be very useful for a project I'm working on so I'm hoping to get a copy of it.

Thanks :)

Lachlan

Bug with xml2tlk

I'm getting a strange error when trying to use it with --dragonage and --version40 flags together on the file that was previously unpacked with tlk2xml tool.

If you need an example of tlk, let me know.

NCSDIS: Control flow analysis failure in do-loop nested in while-loop

Hi :)

There appears to be an issue in either the control flow analysis in ncsdis or (potentially) a bug in nwnnsscomp.exe (which was used to compile the file). The bug happens in a rather strange situation, where you have a do loop nested within a while loop. Ncsdis seems to disassemble the ncs alright, but the control flow analysis throws the following warning:

"WARNING: Control flow analysis failed
Because: Loop blocks out of order: 00000073, 00000073, 000000A9"

Again, I'm not necessarily saying this is a bug with ncsdis as it might be an error with nwnnsscomp.exe (this is a fringe case, and was likely never tested); it's probably worth investigation either way. I've attached the script (apologies for the .txt extension but I can't submit .nss files).

ncsdis_fail.txt

xoreos 0.0.5 tools won't run on Windows 7

I did some poking around and according to what I've found, this is probably a Windows problem rather than a xoreos problem and might not be worth fixing. But I figured I ought to make it known that there is an issue in case it becomes a problem for someone else.

Whenever I try to use at tool from the 0.0.5 release, I get the following error message:
The procedure entry point _create_locale could not be located in the dynamic link library msvcrt.dll.
I found some old error reports for other software that suggest this issue is specific to Visual C++ on Windows 7. According to users from ages past, it shouldn't happen on Windows 8 or 10. And of course I know people who can run the 0.0.5 release tools on Windows 10 successfully, and they work for me on Linux. The 0.0.4 release tools also do run on Windows 7 for me, so the issue is probably limited to the most recent update. I can run the tools just fine on Linux or by downgrading for Windows 7, so no pressure or anything to fix them on my account.

Issue running configure to build project: Syntax error near unexpected token `FT2

I keep getting an error related to FT2:

checking for PTHREAD_PRIO_INHERIT... yes
./configure: line 23447: syntax error near unexpected token `FT2,'
./configure: line 23447: `	PKG_CHECK_MODULES(FT2, freetype2 >= 11.0.5, , as_fn_error $? "FreeType2 (>= 11.0.5) is required and could not be found!" "$LINENO" 5)'

when running ./configure

I have freetype updated:

> brew install freetype2                                                                                                                                                                                                                               

Warning: freetype 2.10.4 is already installed and up-to-date.
To reinstall 2.10.4, run:
  brew reinstall freetype

CONVERT2DA: Code and documentation mismatch for the parameters

I'm having a few problems with the command line parameters for convert2da.

I downloaded a few versions to test with:

wget https://github.com/xoreos/xoreos-tools/releases/download/v0.0.4/xoreos-tools-0.0.4-linux64.tar.gz
tar xvf xoreos-tools-0.0.4-linux64.tar.gz

wget https://github.com/xoreos/xoreos-tools/releases/download/v0.0.5/xoreos-tools-0.0.5-linux64.tar.gz
tar xvf xoreos-tools-0.0.5-linux64.tar.gz

wget https://github.com/xoreos/xoreos-tools/releases/download/v0.0.6/xoreos-tools-0.0.6-linux64.tar.gz
tar xvf xoreos-tools-0.0.6-linux64.tar.gz

0.0.4 works as expected. Starting with 0.0.5 (including 0.0.6) I observe the following behaviour:

1. The -o parameter doesn't work as advertised

   Here's the output of the help text:

   ./xoreos-tools-0.0.5-linux64/convert2da | grep output
     -o      --output  <file>    Write the output to this file
   

   Here's what happens when I try to use -o:

   $ ./xoreos-tools-0.0.5-linux64/convert2da ~/.steam/steam/steamapps/workshop/content/208580/485537937/override/appearance.2da -o appearance.csv
   ERROR: Failed reading GDA file
      Because: Failed reading GFF4 file
      Because: Not a GFF4 file
   

   I was able to do this as a workaround:

   $ ./xoreos-tools-0.0.5-linux64/convert2da ~/.steam/steam/steamapps/workshop/content/208580/485537937/override/appearance.2da > appearance.csv
   

2. Related to the previous point, the man page doesn't match the help text

   Here's a snippet of the output of running convert2da without any parameters:

            --2da               Convert to ASCII 2DA (default)
            --2dab              Convert to binary 2DA
            --cvs               Convert to CSV
   

   But the man page instead lists -a, -b, and -c:

   $ man xoreos-tools-0.0.5-linux64/man/convert2da.1 | grep -A 8 '\-a'
      -a
      --2da
            Convert the 2DA or GDA file into an ASCII 2DA file.  This is the default mode of opera‐
            tion.
      -b
      --2dab
            Convert the 2DA or GDA file into a binary 2DA file.
      -c
      --csv
   

   However, these options don't work any more:

   $ ./xoreos-tools-0.0.5-linux64/convert2da -a ~/.steam/steam/steamapps/workshop/content/208580/485537937/override/appearance.2da
   ERROR: Can't open file "-a"
   $ ./xoreos-tools-0.0.5-linux64/convert2da -b ~/.steam/steam/steamapps/workshop/content/208580/485537937/override/appearance.2da
   ERROR: Can't open file "-b"
   $ ./xoreos-tools-0.0.5-linux64/convert2da -c ~/.steam/steam/steamapps/workshop/content/208580/485537937/override/appearance.2da
   ERROR: Can't open file "-c"
   

3. --cvs should be --csv?

   $ ./xoreos-tools-0.0.5-linux64/convert2da | grep -i csv
   BioWare 2DA/GDA to 2DA/CSV converter
            --cvs               Convert to CSV
   

Thanks!

CMAKE Build - xml issue / error

The cmake build can't link the tools while the combination of autogen.sh , configure , make can.
Here is the log file

buildlog.txt

System: Archlinux current

XML2GFF: Tool needs to be game/encoding aware

For embedded LocStrings, the tool needs to be aware of which game the GFF is created for, so that string can be written with the proper encoding.

FEATURE: Add tool to convert TGA back to TPC

I regularly use xoreostex2tga in order to unfold kotor files, but I didn't any command line tool which was capable of converting back tga files to tpc.

It would be very interesting to have such a tool to make mods. At the moment only "tga2tpc" GUI is capable of doing such a thing but we can't pipe it in the command line chain!

FEATURE: Make more tools read stdin for input

(Split from #58)

Which tools exactly do you need to be able to read from stdin?

Many (Most? All? of the tools that can read textual data should already be able to read from stdin.

Many (Most? All?) of the tools that read binary data currently don't, because they need SeekableReadStreams, and stdin obviously isn't seekable. However, often this is only used for SeekableReadStream::skip() and only in a forward direction.

So I could probably fix that by:

1. Introduce a companion method to skip() that seeks backwards. Though I'm not sure what to call it. rewind()? Or rather rewind() is seek(0, kOriginBegin), while rewind(size_t n) is seek(-n, kOriginCurrent)?
2. Change all uses of skip() with a negative parameter to use that new method
3. Change SeekableReadStream::skip() to take an unsigned value and mandate it to only skip forwards
4. Add a virtual method skip in ReadStream that implements it using read(). SeekableReadStream still overrides it with the seek()-using-implementation

That would give us a ReadStream that can do skip() and we might be able to downgrade some uses of SeekableReadStream to ReadStream instead, which means we can use StdInStream there.

I would need to evaluate that on a case-by-case basis, though.

Thoughts?

UNOBB: OBB files are actually not Aspyr related

I read a bit about android development and found out that obb files do not look like they are not specifically related to Aspyr but rather a common android container for large amounts of files (See also), since android play requires apk to not exceed a size of 100Mb. Maybe it would be wirth to change some notes to make clear it is something android specific to avoid confusion, like it did for me.

xml2tlk gives error when trying convert back cp1251 xml

When i trying to convert cp1251 xml back to tlk programm returns:
WARNING: iconv() failed: Illegal byte sequence!
and just making empty tlk file.

I tried to not making cp1251 encoding - this works fine, but it is not supporting cyrillic symbols in xml then.

UNERF: Encrypted ERF (v2.2 blowfish) not working

It seems that the unerf tool is testing the password's md5 instead of the decimal string corresponding to the password, which prevents it from extracting even when using a correct password.
ERF 2.2 Format
ERF Blowfish decryption / encryption steps
Attaching an example.
Created in Dragon Age: Origins Toolset - ERF Editor

ERF version: v2.2
Encryption: Blowfish
Module Id (decimal, as expected by DA:O Toolset): 123
Password (decimal, as expected by DA:O Toolset): 1234567890

DA:O Toolset ERF Editor opens the file, shows its content and extracts just fine (hello_world.txt)
Running unerf results in error:

unerf --pass 499602D2 e HelloWorld.erf
ERROR: Failed reading ERF file
    Because: Password digest does not match

HelloWorld.zip

FEATURE: "File picker" for RIM/ERF/MOD files

Hi :)

As mentioned in #59, it would be useful if xoreos-tools had a straightforward RIM/ERF/MOD packer/unpacker that had the following features (given a particular archive):

• Add/replace/remove a file in the archive based on binary/xml input (only binary output is sufficient if #59 is implemented as then piping can be used, though an xml output would probably be straightforward and useful either way)
• Get a file from the archive in binary/xml format (with same caveat as above)
• Unpack the archive into files, and vice versa

Most of these features are already available in different xoreos-tools executables; the difficult part would probably be the first part (add/replace/remove a file). I imagine the command-line might look like:

command_name {--get/--add/--replace/--remove/--unpack/--pack} {--erf/--rim/...} {--kotor, --kotor2, ...} {--inplace/--newfile} --input infile --output outfile

If infile isn't given, read from stdin; if outfile isn't given, write to stdout (so -i and -o flags are necessary as it can't rely on positioning alone).

I feel pretty bad that I keep asking for new features but don't contribute to the project, so I'm wondering whether I might take this opportunity (if you're open to it) to contribute this feature myself (once #59 is dealt with, as the binary reading will rely on that).

another error while compiling

it give me this error in log

Determining if the include file pthread.h exists failed with the following output:
Change Dir: C:/Users/sebas/OneDrive/Desktop/xoreos-tools/out/build/x64-Debug/CMakeFiles/CMakeTmp

Run Build Command(s):C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/CommonExtensions/Microsoft/CMake/Ninja/ninja.exe cmTC_689b7 && [1/2] Building C object CMakeFiles\cmTC_689b7.dir\CheckIncludeFile.c.obj

FAILED: CMakeFiles/cmTC_689b7.dir/CheckIncludeFile.c.obj

C:\PROGRA1\MICROS3\2022\COMMUN1\VC\Tools\MSVC\14301.307\bin\Hostx64\x64\cl.exe /nologo /DWIN32 /D_WINDOWS /W3 /MDd /Zi /Ob0 /Od /RTC1 /showIncludes /FoCMakeFiles\cmTC_689b7.dir\CheckIncludeFile.c.obj /FdCMakeFiles\cmTC_689b7.dir\ /FS -c C:\Users\sebas\OneDrive\Desktop\xoreos-tools\out\build\x64-Debug\CMakeFiles\CMakeTmp\CheckIncludeFile.c

C:\Users\sebas\OneDrive\Desktop\xoreos-tools\out\build\x64-Debug\CMakeFiles\CMakeTmp\CheckIncludeFile.c(1): fatal error C1083: Cannot open include file: 'pthread.h': No such file or directory
ninja: build stopped: subcommand failed.

Performing C++ SOURCE FILE Test ICONV_SECOND_ARGUMENT_IS_CONST failed with the following output:
Change Dir: C:/Users/sebas/OneDrive/Desktop/xoreos-tools/out/build/x64-Debug/CMakeFiles/CMakeTmp

Run Build Command(s):C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/CommonExtensions/Microsoft/CMake/Ninja/ninja.exe cmTC_00236 && [1/2] Building CXX object CMakeFiles\cmTC_00236.dir\src.cxx.obj

FAILED: CMakeFiles/cmTC_00236.dir/src.cxx.obj

C:\PROGRA1\MICROS3\2022\COMMUN1\VC\Tools\MSVC\14301.307\bin\Hostx64\x64\cl.exe /nologo /TP -DICONV_SECOND_ARGUMENT_IS_CONST -IC:\dev\cpp\vcpkg\packages\libiconv_x86-windows /DWIN32 /D_WINDOWS /W3 /GR /EHsc /MP /bigobj /wd4250 /wd4100 /wd4127 /wd4189 /wd4245 /wd4435 /wd4510 /wd4512 /wd4610 /wd4706 /wd4710 /wd4714 /wd4305 /wd4244 /wd4267 /wd4996 /MDd /Zi /Ob0 /Od /RTC1 /showIncludes /FoCMakeFiles\cmTC_00236.dir\src.cxx.obj /FdCMakeFiles\cmTC_00236.dir\ /FS -c C:\Users\sebas\OneDrive\Desktop\xoreos-tools\out\build\x64-Debug\CMakeFiles\CMakeTmp\src.cxx

C:\Users\sebas\OneDrive\Desktop\xoreos-tools\out\build\x64-Debug\CMakeFiles\CMakeTmp\src.cxx(2): fatal error C1083: Cannot open include file: 'iconv.h': No such file or directory
ninja: build stopped: subcommand failed.

Source file was:

#include <iconv.h>
int main(){
iconv_t conv = 0;
const char* in = 0;
size_t ilen = 0;
char* out = 0;
size_t olen = 0;
iconv(conv, &in, &ilen, &out, &olen);
return 0;
}

GFF2XML -> XML2GFF (invalid base64 length)

Hi,

it looks like xml2gff doesn't work for XMLs that contains longer base64 as it fails because of "Invalid length for a base64-encoded string".

I checked the string itself and it's correct, just contains whitespaces from XML formatting (and newlines).

BUILD: Move Travis CI config from artful to bionic?

The Ubuntu release Artful Aardvark we're updating the Travis CI VM (to get C++11 support) has reached end-of-life earlier this year. I.e. the repos might vanish, and then everything will break.

The next LTS version is Bionic Beaver (18.04). Maybe we should adapt our Travis CI config to update to that.

Also relevant for Phaethon, and xoreos proper in the near-ish future.

Missing xml2gff

Hi, @DrMcCoy!

Is xml2gff tool still under development? I can see its source code, but can't find it anywhere in the release bundles.

XML2GFF: Serialize color codes

The Aurora games has a weird way to do color codes: <cRGB>Foo</c> where R, G and B are raw byte values (!).

When reading in LocStrings, we demunge those into <cRRGGBBAA>Foo</c> where RR, GG, BB and AA are textual hexadecimal values.

When converting XML files back to GFF, we need to reverse this as well.

ERF: Recursion leads to infinite loop

Hi :)

When packing an ERF file into itself (e.g. with "erf --mod output.mod files ... output.mod"), it seems an infinite loop is triggered which is only exited when the 4GB file size limit is reached. This issue might exist in the other archive tools as well but I have observed it for ERF at least.

NCSDIS: Analyse control flow for recursive functions, and functions with incompatible fork merging

Hi :)

My ncs2nss decompiler is nearly complete, but I've run into a bit of a roadblock. I use the output from ncsdis's analysis to create function signatures, which greatly simplifies the data flow analysis (because I can do the analysis in a single pass). However, the control flow fails in the following cases:

1. Unbalanced block fork merging detected
2. Recursion is detected

From looking at the decompiled code (attached to the issue - don't take it as gospel though, as there are certainly mistakes in it) for the file "cp_end_trasksp_d.ncs", which causes the block fork issue (as discussed on the DeadlyStream forums), I have a suspicion that the problem is due to some compiler issue with returning from a subroutine, where it doesn't clean up the stack properly. If I had to guess, I'd say that this issue was never found because it is caught and handled by the interpreter at runtime, but I could be completely wrong. In any case, hopefully the attached decompiled code will help out.

The recursion issue is more difficult - however, I think I have an idea for how it can be solved. When a recursive function is detected, you can parse the function and do the following in a single pass:

• Whenever an access is made to the stack below the bottom of the stack using a cptopsp command (i.e. an argument access), record the argument and its type (which can hopefully be determined from the commands which subsequently use this argument after copying it to the top of the stack). Record this in the "list of argument types".
• When we write down below the bottom of the stack using cpdownsp, that indicates that we are writing a return value. Record this type in the "set of below-stack assignments".

After this analysis, the final step is to find the position on the stack of the lowest "below-stack assignment". If this position is not contained by any of the arguments, it's a return value. If the "below-stack assignment" set is empty, or all of the assignments were to positions known to be arguments, there is no return type and it's a void subroutine.

The above algorithm works on non-recursive functions with no issues. If the function calls itself directly (direct recursion), a simple heuristic algorithm I can suggest is to run the previous algorithm multiple times, with the number of arguments hard-set to 0, 1, 2, 3, ..., until you find a number for which the stack is consistent throughout and at the end of the analysis (basically, guessing). Once you have the number of arguments, the return value can also be calculated as above.

If the function does not call itself, but is rather recursively called by another function, I think you would have to run the "guessing" algorithm but with guesses for each of the different subroutines in the recursion graph, which expands the number of possibilities exponentially with the number of functions in the loop. Although the analysis might be relatively expensive, I think it should still be quick enough for any scripts requiring recursion (I'd imagine that most scripts would have fairly small chains).

This should allow you to infer the arguments and return value of a subroutine without having to rely on other subroutines calling it. You could also implement this as a separate check and then compare the result of this to the analysis already implemented, and ensure they are the same as a sanity check.

I might implement this in my ncs2nss code as well, but my preference would be to rely on ncsdis (because there's no point in duplicating code and ncsdis is already most of the way there).

Please let me know if you have any thoughts on this. Thanks :)

cp_end_trasksp_d.txt

XOREOSTEX2TGA: Failure to convert many TPC files

(This was introduced with @Nostritius's ecfb314)

xoreostex2tga now fails to convert many KotOR TPCs, with a "Invalid UTF-8" error thrown. C_Wraid02.tpc is one example.

That example file has no attached TXI data, and I guess it tries to read other binary data as TXI, thus throwing. I.e. the offset after reading the mipmap data, before reading the TXI data, is not at the end of the file.

ERF: MOD files have incorrect prefix

Hi :)

The first eight bytes of a mod file saved with ERF v1.0 should read "MOD V1.0", but they read "ERF V1.0" instead. This apparently causes TSL to be unable to open the archive, where manually hex-editing the first eight bytes to the correct value allows the archive to be opened.

NCSDIS: --list is not default

In ncsdis, the NCS disassembler, if none of --list, --assembly or --dot are given, ncsdis just spits out "Invalid command".

Instead, if none are given, --list should be the default, and ncsdis should provide a full listing of the disassembly.

xml2gff: NWN:EE invalid file format

We migrated to NWN:EE and our servervault (converted by xml2gff) completely broke with following issue:

*** ValidateGFFResource sent by user.
*** FAULT *** ValidateSafePointersInData.  struct (1) was referenced more than once.

I believe that's because of deduplication of referenced fields that's apparently not allowed in NWN:EE. Can I ask at least for pointers how to fix this issue?

Clarify XML format for xml2gff

Hi :)

I was hoping it might be possible to develop an XML schema for what is expected by xml2gff, as it appears to be quite picky (especially regarding the string formats). I could attach an initial draft of such a scheme to this issue in a comment in the next day or two - @DrMcCoy would you be willing to verify this and help me fix any errors in the schema?

Thanks :)

Write an NWScript assembler and compiler

Currently, we have a disassembler for NWScript bytecode, ncsdis. For an explanation for what NWScript is and how it works internally, please see here: https://xoreos.org/blog/2016/01/12/disassembling-nwscript-bytecode/

However, we could also use an NWScript assembler. For that, the existing NWScript IR structures could be leveraged.

As a first step, a roundtrip from NWScript bytecode -> IR -> NWScript bytecode would be possible. Afterwards, NWScript disassembly as produced by the disassembler could be read, parsed, and stuffed into the IR structures to be turned into bytecode.

The IR itself might also need to be modified, to bring it more in line with usual compiler IR.

The next step after that would be to expand the assembler into a compiler. Read and parse the C-like NWScript source, compile it into IR, and then write it to disk as bytecode.

Owning the different BioWare games would be useful here, since more functionality was added to the script over the years. And testing the assembler/compiler over a wide variety of scripts is paramount. As a starter, though, a few (Neverwinter-Nights-era) script sources and their bytecode are here: scripts.zip

Like the rest of the xoreos-tool, the assembler and compiler should be written in C++. xoreos-tools is currently fully C++03, but I'm opening it up to C++11 now. I don't necessarily want to see a PR with thousands line diffs changing everything in xoreos-tools the C++11 way, but feel free to use C++11 features in new code.

NWN2: Module Corruption Checker

One of the persistent issues we have while building with the NWN2 toolset is the (occasional) corruption of a module as a result of a save. I'm speculating that a useful utility could be built using this code, and the source code at xoreos/xoreos, that could process a NWN2 module file or directory and look for signs of corrupted elements. The tool could do things like check for invalid offsets, look for invalid fields, find missing textures, etc. Possibly other issues could be identified if we could obtain some corrupt module files to practice upon. If the specific corrupt element could be readily identified, that would quickly help remediate the issue.

Such a tool would be of immediate value to the NWN2 community.

Static compilation?

I want to be able to compile this statically, so I can include it in an app on Android. However, when I try to do so, a bunch of files give the error

/usr/bin/ld: /usr/lib/arm-linux-gnueabihf/libc.a(libc-start.o): relocation R_ARM_THM_MOVW_ABS_NC against `_dl_starting_up' can not be used when making a shared object; recompile with -fPIC            /usr/lib/arm-linux-gnueabihf/libc.a: error adding symbols: Bad value

Googling the issue tells me that it is trying to compile dynamically, despite me specifically saying to compile statically only (the full configure was

./configure --enable-static --disable-shared CPPFLAGS="-fPIC -fPIE -static" LDFLAGS="-static /usr/lib/arm-linux-gnueabihf/libz.a /usr/lib/arm-linux-gnueabihf/libxml2.a /usr/lib/gcc/arm-linux-gnueabihf/6/libstdc++.a /usr/lib/arm-linux-gnueabihf/libm.a /usr/lib/gcc/arm-linux-gnueabihf/6/libgcc.a /usr/lib/arm-linux-gnueabihf/libc.a /usr/lib/arm-linux-gnueabihf/libdl.a /usr/lib/arm-linux-gnueabihf/libicui18n.a /usr/lib/arm-linux-gnueabihf/libicuuc.a /usr/lib/arm-linux-gnueabihf/libicudata.a /usr/lib/arm-linux-gnueabihf/liblzma.a /usr/lib/arm-linux-gnueabihf/libpthread.a"

). Is something wrong with my configure, does it just not build statically, or what? I am building on an Android tablet in Termux, using Debian For Termux.

error while compiling

i get this error while compiling can somebody help please

Severity Code Description Project File Line Suppression State
Error LNK1104 cannot open file 'external_mspack.lib' C:\Users\sebas\Downloads\xoreos-tools-master\xoreos-tools-master\out\build\x64-Debug\xoreos-tools-master C:\Users\sebas\Downloads\xoreos-tools-master\xoreos-tools-master\out\build\x64-Debug\LINK 1

Upgrade the NWScript disassembler to a decompiler

Currently, we have a disassembler for NWScript bytecode, ncsdis. For an explanation for what NWScript is and how it works internally, please see here: https://xoreos.org/blog/2016/01/12/disassembling-nwscript-bytecode/

What we could use is a full-fledged decompiler. For that, the existing NWScript IR produced by the disassembler could be leveraged. More analysis on said IR would be necessary, of course.

The IR itself might also need to be modified, to bring it more in line with usual compiler IR. And the disassembler could probably also profit from this task.

Owning the different BioWare games would be useful here, since more functionality was added to the script over the years. And testing the decompiler over a wide variety of scripts is paramount. As a starter, though, a few (Neverwinter-Nights-era) script sources and their bytecode are here: scripts.zip

Like the rest of the xoreos-tool, the decompiler should be written in C++. xoreos-tools is currently fully C++03, but I'm opening it up to C++11 now. I don't necessarily want to see a PR with thousands line diffs changing everything in xoreos-tools the C++11 way, but feel free to use C++11 features in new code.

FEATURE: Compile xoreos-tools as a DLL/library

Hi :)

I've been using xoreos-tools for several of my projects, and up until now I've been mainly doing so through subprocess calls and writing files to disk. For example, I use it in my Python-based dialog editor (which I admittedly could write in C++ but that would be a significant changeover as I use several Python libraries for the project) as well as a Unity-based KOTOR level editor (which is in C# and can't be ported to C++ without changing game engines). This subprocess-based solution is less than ideal, and on machines without an SSD it could even become a significant bottleneck (although on my relatively fast SSD-based machine it's been fine so far).

Rather than moving to C++ (which is difficult and even not really possible in some cases), it seems that having "xoreos-tools come to us" through a DLL (which I could then write bindings for in all relevant languages) would be a better solution.

I took a look through the codebase and it seems that a lot of the code for the executables is also written in a way conducive to compiling as a DLL. Minor refactoring would be required for some files which include necessary logic in the main function (for example, eff and keybif would need a slight refactor). I think the ideal starting point would be the ability to do anything you can do through running the .exe files through a call to the DLL instead. I'm sure that there are other interesting/useful things that your code does behind the scenes which might be useful for other projects too though, so this is also worth considering down the track.

Is making xoreos-tools available as a DLL something that you'd consider?

Thanks :)

Lachjames

NWN2's TLK mixes encodings, breaks tlk2xml

The TLK file in Neverwinter Nights 2 mixes encodings. This breaks our tlk2xml utility.
Spawned off of this thread on the Neverwinter Vault: https://neverwintervault.org/comment/36764

When interpreting the TLK as CP-1252 (either using the command line switch --cp1252 or --nwn2), iconv will complain about some of these (and output "[!?!]") and produce garbage for others. When interpreting the TLK as UTF-8 (using the command line switch --utf8), our UTF-8 string class will throw for some of them.

What follows is a rundown of all problematic strings (with IDs and the raw data of the offending characters in hex notation) found in the unmodified English dialog.tlk from the GOG version of Neverwinter Nights 2. The natural encoding should be Windows Codepage 1252, but some strings are UTF-8 instead, a few Polish strings are probably CP-1250 instead, and some very broken strings are even double-UTF-8.

• û in Faerûn, Faerûnian, Selûne, Selûnian is encoded as UTF-8 [C3 BB]:

  • 252
  • 12891
  • 13008
  • 13010
  • 13022
  • 13034
  • 13038
  • 13040
  • 13050
  • 13056
  • 13410
  • 13424
  • 13446
  • 13906
  • 13921
  • 13961
  • 13978
  • 13993
  • 14017
  • 14030
  • 14076
  • 14088
  • 40083
  • 41024
  • 41128
  • 47128
  • 47173
  • 48907
  • 48937
  • 60920
  • 60963
  • 68839
  • 68850
  • 75624
  • 75631
  • 79701
  • 79747
  • 83492
  • 86778
  • 91185
  • 91248
  • 91258
  • 94988
  • 95023
  • 95081
  • 107813
  • 112085
  • 127464
  • 128485
  • 129198
  • 131466
  • 131937
  • 131938
  • 131939
  • 132428
  • 132681
  • 132683
  • 132686
  • 132827
  • 132830
  • 133552
  • 133553
  • 136564
  • 138234
  • 138266
  • 138941
  • 139484
  • 142353
  • 142830
  • 142831
  • 142847
  • 142848
  • 142855
  • 142876
  • 143298
  • 145320
  • 146031
  • 146607
  • 150041
  • 151576
  • 151969
  • 151970
  • 151986
  • 151987
  • 151994
  • 152163
  • 152184
  • 152185
  • 152193
  • 152194
  • 158868
  • 158913
  • 159535
  • 159541
  • 159549
  • 159575
  • 159668
  • 160377
  • 161440
  • 161613
  • 161913
  • 162188
  • 162225
  • 162394
  • 164349
  • 165035
  • 168344
  • 173752
  • 175833
  • 176226
  • 176228
  • 176452
  • 176463
  • 176797
  • 178078
  • 178232
  • 179492
  • 179646
  • 182325
  • 183608
  • 183610
  • 183613
  • 183616
  • 183619
  • 183621
  • 184288
  • 185484
  • 190706
  • 192763
  • 194275
  • 206565
  • 228547
  • 230125
  • 231209
  • 234736
  • 234737
  • 234746

• é in fiancé, décor and protégé is encoded as UTF-8 [C3 A9]:

  • 14074
  • 14101
  • 206490

• ï in naïve is encoded as UTF-8 [C3 AF]:

  • 144840

• Double-UTF-8 (UTF-8 data interpreted as Windows CP-1252 and then encoded as UTF-8 again):

  • 155891
  • 159776
  • 159836
  • 174702
  • 176247
  • 176670
  • 176671
  • 177416
  • 181507
  • 176832 ([C3 A2 E2 82 AC E2 80 9D], that's em-dash [0xE2 0x80 0x94] in double-UTF-8)

• ½ is encoded as CP-1252 [BD]:

  • 241
  • 6075
  • 185809
  • 185819

• © is encoded as CP-1252 [A9]:

  • 3038

• … (ellipsis) as CP-1252 [85]:

  • 75941
  • 75947
  • 75949
  • 75950
  • 76064
  • 76066
  • 76068
  • 76070
  • 76071
  • 76072
  • 76080
  • 76082
  • 76114
  • 76124
  • 76135
  • 76136
  • 76144
  • 76153
  • 76182
  • 76356
  • 76367

• CP-1252 smart single quotes ‘ [91] and ’ [92]:

  • 53031

• CP-1252 smart apostrophe ’ [92]:

  • 13770
  • 25203
  • 90857
  • 90861
  • 91117
  • 91125
  • 91129
  • 91145
  • 91179
  • 91262

• UTF-8 smart double quotes “ [E2 80 9C] and ” [E2 80 9D]

  • 137098
  • 155799
  • 161607
  • 162084 (also with UTF-8 Faerûn)
  • 177903
  • 178401
  • 180942 (also with UTF-8 … (ellipsis) [E2 80 A6])
  • 205916
  • 218438 (also with UTF-8 ’ (apostrophe) [E2 80 99])
  • 232910
  • 232945
  • 232952
  • 232958
  • 232962

• Polish strings with the letter ł:

  • 3114
  • 3116

• Polish strings with unknown letter (ż?):

  • 3143
  • 3145

• Unknown strings with unknown encoding, two unknown letters (Polish? One of the letters might be ą?):

  • 3127
  • 3128
  • 3129
  • 3134
  • 3137
  • 3138

• Unknown strings with unknown encoding, single letters (might be CP-1252?):

  • 3154
  • 3155
  • 3157