tyulis/3dskit

A multi-purpose and pluggable program to extract and repack files found in 3DS (and some other Nintendo consoles) games

3dskit's Issues

No Support for packing BFFNT v4.1

I can unpack and make changes, but I can not package. I'm new to this and I do not know if I'm doing it right.

I unpacked with these arguments
python 3DSkit.py -xf BFFNT -O reverse=true -O origin=NX "MessageFont38.bffnt"

And I try to pack with these arguments
python 3DSkit.py -pf BFFNT -O reverse=true -O meta="MessageFont38_meta.json" "MessageFont38_sheet0.png" "MessageFont38_sheet1.png"

Returns this message
UnsupportedFormatError: 3DSkit is currently unable to pack this format (101)

FontData.zip

These are files from the Super Mario Odyssey game of the Nintendo Switch

Forgive me for my lack of knowledge.

Error unpacking new BFLAN Nintendo Switch

I don't want to overload you with issues, but here is another error.
The common.szs (another system file that contains info just like ResidentMenu.szs) also comes with an animation folder. Some of these animations can be opened in Wexos and extracted by your kit, like SystemColorFader_In.bflan, but others, like SystemColorFader_Color.bflan, result in an error. This so far only happens with files that (I assume) contain color information: BgNml_Color.bflan, BlurBg2_Color.bflan, CntrIndicatorS_Color.bflan, SystemColorFader_Color.bflan etc.
SystemColorFader_In.bflan:
https://drive.google.com/open?id=1YaYKFYTffGlzWl7kkRnddsEwO7aUiWm_

--------SystemColorFader_Color.bflan--------
No compression
BFLAN file found
Extracting...
Traceback (most recent call last):
  File "3DSkit.py", line 249, in <module>
    result = main(args, opts)
  File "3DSkit.py", line 169, in main
    extract_files(filename, args.big, args.format, args.verbose, opts)
  File "3DSkit.py", line 112, in extract_files
    unpack.extract(filename, file, format, endian, verbose, opts)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\unpack\__init__.py", line 16, in extract
    unpacker = cls(*args)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\util\utils.py", line 18, in __new__
    return self.main(*args, **kwargs)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\unpack\BFLAN.py", line 41, in main
    self.extract_sections(ptr)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\unpack\BFLAN.py", line 63, in extract_sections
    method(ptr)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\unpack\BFLAN.py", line 131, in readpai1
    data = self.unpack_from(pai1_TAG, self.file, tagpos)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\util\rawutil.py", line 101, in unpack_from
    unpacked, ptr = _unpack(stct, data, offset, byteorder, refdata)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\util\rawutil.py", line 58, in __new__
    return ins.main(*args, **kwargs)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\util\rawutil.py", line 317, in main
    return self.unpack_file(stct)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\util\rawutil.py", line 415, in unpack_file
    final.append([self.unpack_file(el.stct)[0] for i in range(count)])
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\util\rawutil.py", line 415, in <listcomp>
    final.append([self.unpack_file(el.stct)[0] for i in range(count)])
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\util\rawutil.py", line 467, in unpack_file
    final += struct.unpack(substruct, subdata)
struct.error: unpack requires a buffer of 4 bytes

Idk if Wexos' error will add anything, but I'll include it anyway

Some files return a bit different error, like FooterBtn_TextColor.bflan
FooterBtn_TextColor.bflan:
https://drive.google.com/open?id=1viUh1Xuy_WFCtLbvjWfljpKfbBaBt8Qm

--------FooterBtn_TextColor.bflan--------
No compression
BFLAN file found
Extracting...
Traceback (most recent call last):
  File "3DSkit.py", line 249, in <module>
    result = main(args, opts)
  File "3DSkit.py", line 169, in main
    extract_files(filename, args.big, args.format, args.verbose, opts)
  File "3DSkit.py", line 112, in extract_files
    unpack.extract(filename, file, format, endian, verbose, opts)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\unpack\__init__.py", line 16, in extract
    unpacker = cls(*args)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\util\utils.py", line 18, in __new__
    return self.main(*args, **kwargs)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\unpack\BFLAN.py", line 41, in main
    self.extract_sections(ptr)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\unpack\BFLAN.py", line 63, in extract_sections
    method(ptr)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\unpack\BFLAN.py", line 148, in readpai1
    tagentry['type2'] = self.gettype(type2, type)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\unpack\BFLAN.py", line 161, in gettype
    el = eval('%s_types[type]' % tagtype)
ValueError: source code string cannot contain null bytes

And here's Wexos' error for that one:

Edit: I'm adding some more info

I have executed the unpack command for every file.
Here's a hastebin: https://hastebin.com/direwirose.rb
I noticed that not every color file, BgNml_Color.bflan and BlurBg2_Color.bflan, results in an error when unpacking.
Oh whoops, I forgot that unpacking BFLAN's is only partially supported and they cannot be repacked yet.

Error packing new BFLYT Nintendo Switch

When trying to repack the LineFooter.json -> LineFooter.bflyt an error is generated. Unpacking is fine though.

Packing .\LineFooter-edit.bflyt...
Packing BFLYT
Traceback (most recent call last):
  File "3DSkit.py", line 249, in <module>
    result = main(args, opts)
  File "3DSkit.py", line 193, in main
    pack_files(files, args.out, args.compression, args.format, args.big, args.verbose, opts)
  File "3DSkit.py", line 52, in pack_files
    pack.pack(filenames, output, format, endian, verbose, opts)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit-master\pack\__init__.py", line 10, in pack
    return func(*args)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit-master\util\utils.py", line 18, in __new__
    return self.main(*args, **kwargs)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit-master\pack\BFLYT.py", line 69, in main
    self.packdata(data, output)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit-master\pack\BFLYT.py", line 87, in packdata
    self.packsection(secname, section, output)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit-master\pack\BFLYT.py", line 98, in packsection
    self.packmat1(name, data, output)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit-master\pack\BFLYT.py", line 265, in packmat1
    self.pack('3f', node['rotation'], node['x warp'], node['y warp'], output)
KeyError: 'x warp'

File:
https://drive.google.com/file/d/1n42hjXlsJS0WZGEfnwIjtsZxZtgBdWz7/view?usp=sharing

Error unpacking ALYT files

When unpacking alyt archive I get an error. There is no such problem with other alyt archives. Help me to solve the problem. I attach the file with the problem.
dec_7.zip

Dbz Supersonic warriors 2 .arc files not supported

I tried with several options and all of them give unrecognized file type

Error extracting nx_bffnt font ,I don't know why.

Font found in [Captain Toad：Treasure Tracker] NS version
https://drive.google.com/open?id=1X4yCZtTADGWygFNK0ELZhMU5h2Id1sjW

Error Log

D:\TEST\3DSkit>3DSkit.py -x -V TitleFont32.bffnt

--------TitleFont32.bffnt--------
No compression
BFFNT file found
Extracting...
Version 4.1.0 (NX)
Texture format: BC4
Sheet width: 1024
Sheet height: 1024

Extracting wrapped BNTX
BNTI info for sheet 0:
 - Format: BC4
 - Width: 1024
 - Height: 1024
Traceback (most recent call last):
  File "D:\TEST\3DSkit\3DSkit.py", line 239, in <module>
    result = main(args, opts)
  File "D:\TEST\3DSkit\3DSkit.py", line 161, in main
    extract_files(filename, args.big, args.format, args.verbose, opts)
  File "D:\TEST\3DSkit\3DSkit.py", line 111, in extract_files
    unpack.extract(filename, file, format, endian, verbose, opts)
  File "D:\TEST\3DSkit\unpack\__init__.py", line 16, in extract
    unpacker = cls(*args)
  File "D:\TEST\3DSkit\util\utils.py", line 16, in __new__
    return self.main(*args, **kwargs)
  File "D:\TEST\3DSkit\unpack\BFFNT.py", line 84, in main
    self.readTGLP(data, self.tglpoffset - 8)
  File "D:\TEST\3DSkit\unpack\BFFNT.py", line 133, in readTGLP
    self.extract_underlying_BNTX(data)
  File "D:\TEST\3DSkit\unpack\BFFNT.py", line 189, in extract_underlying_BNTX
    self.extract_sheet_c3DSkit(data, outname, width, height, sheetsize, BNTI_FORMAT_NAMES[format], swizzlesize)
  File "D:\TEST\3DSkit\unpack\BFFNT.py", line 148, in extract_sheet_c3DSkit
    format = c3DSkit.getTextureFormatId(format)
AttributeError: module 'c3DSkit' has no attribute 'getTextureFormatId'

No support for unpacking BFFNT v4.1

There is no support for unpacking BFFNT version 4.1.0. This version can be found in Super Mario Odyssey.

Here is included one example font:
NumberFont26.zip

Here is the same font but extracted as png (credits to Random Talking Bush and SciresM - https://www.spriters-resource.com/nintendo_switch/supermarioodyssey/sheet/103323/):

Slow performance makes extraction unwieldy

I attempted to extract a GARC file, but found 3DSKit to be unexpectedly slow, and during this time, a folder was created but was not being populated. Investigating further, I noted extracting a 55MB GARC took almost 2 minutes between when extractGARC.readheader() is called and extractGARC.extract() is even called. Given that GARCs can grow to be over a gigabyte big, this renders such large garcs effectively unextractable.

It seems the line that causes this lag in particular is
fato, ptr = self.unpack_from(GARC_FATO_SECTION, data, ptr, getptr=True).
Once this is done, all other sections of the code, such as the FATB extraction, proceed smoothly. Could rawutil.py's _unpack() be the culprit? Could the entire 55MB be being copied repeatedly through several nested function calls through the arguments when it shouldn't be? Further investigation is warranted to eliminate this unwanted lag.

Missing parenthesis

Line 64 of /unpack/GARC.py is missing a closing parenthesis. This prevents any GARCs from being unpacked.

bksnd support

I'm sorry for asking so much of you, but there seems to be a completely new filetype used on the Switch: bksnd. The folder the files are in is called 'sound' which would mean the snd stands for sound. The b probably means Binary and the second letter is part of the codename of the device. My Switch's codenumber on the back is BKEHAC001, which just is Nintendo's grantee code. The bksnd files are not the sounds themselves, because those are in the bfsar. I'm just giving these files to you because 1. I am curious and 2. I'd like to see if there's something editable.

I can provide you with the files again:
https://drive.google.com/drive/folders/1dz1myNu84X7_FPtqWifFeYrstVwlFNkX?usp=sharing

BFLIM replacement in ALYT files results in a crash

At the bottom of the ALYTtool repo, you mention "NOTE THAT ACTUALLY, IF YOU REPACK A BFLYT FILE IN AN ALYT FILE, THERE IS SOME ISSUES WITH THE ELEMENT DISPLAYING (or there is not displaying at all...)." For the last week or so, I've been trying to pack a BFLIM, replace it in an ALYT, and then pack that into a GARC. However, it looks like it's crashing every time the specific repacked BFLIM is being loaded. I'm not sure what the problem is; I've tried packing and repacking and it doesn't seem to work.

Other BFLIMS in the same ALYT file load fine - only when the game tries to load the one 180x360 BFLIM that was replaced does it cause problems.

Do you know what I could do to test or otherwise figure out what's causing this problem?

Download link?

No download?

UnsupportedDataFormatError: Unsupported texture format RGB565 (104)

Help with Pokemon Ultra Sun And Moon bflim.

Add a way to extract and convert the formats in Tank Troopers

Add a way to extract and convert the formats in Tank Troopers for Nintendo 3DS

Extracting files inside exefs Nintendo Switch

You seem quite skilled (especially because I have no idea what has to be done to extract all these binary files), so I wanted to ask you if you could also take a look at the files in the exefs partition. I was wondering if those can be unpacked an converted to a readable format aswell. I don't know though if it'll be of anything useful.
The three files inside the qlaunch's (Nintendo's system titles) exefs:

And here's the link
https://drive.google.com/drive/folders/1RixtA-_BSjd0-ihIJ1tHPQUDgxm0ctfY?usp=sharing

I'm not saying you have to do this, just asking (<- that 'just' can be quite a lot tho) if you could if you have the time available. I don't know if it is even possible, also because there isn't much documentation.

Mini repacking: "3DSkit is currently unable to pack this format"

Using 3DSkit, I managed to unpack a .bin file which is actually a BL mini format archive.
(command line: 3DSkit.py -x 0.bin)

However, the tool doesn't allow repacking, with that error message thrown.
I confirm that the mini.py is indeed in the "pack" folder.

As I'm trying to pack a directory of files, command line used is:
3DSkit.py -pdvf mini 0 (0 being the folder; I understand without options, it'll default to BL)

I've tried playing around with the command line, and can't get it to work, so it might be a bug.
(assuming it's not an error on my side; I've managed to get sarc packing, garc packing, bflim injecting to work, so this is the only thing that bugs me on my end)

packing BFFNT

I just try to pack BFFNT but there is something mess up with CMAP

I even tried to unpack and pack without changes anything to catch up if i did something wrong

these arguments i used

unpack
3DSkit.py -xf BFFNT -O origin=NX jpn_heading.bffnt

and pack
3DSkit.py -pf BFFNT jpn_heading_meta.json

original bffnt size 6,301,668
after packing 6,300,864

so i checked back in HxD
original

after packing

and here example

https://drive.google.com/file/d/1VQNGUAyiXaaMSR1oapyUwB8g1MAyK1OW/view?usp=sharing

Packing BFLIM crashes due to swizzle option

$ 3DSkit.py -p 'intro_bg_01.png' -o intro_bg_01.bflim -f BFLIM
creates this error message:

Packing BFLIM
Traceback (most recent call last):
  File "3DSkit.py", line 163, in <module>
    pack_files(files, args.out, args.compression, args.format.upper(), args.big, opts)
  File "3DSkit.py", line 34, in pack_files
    outnames = pack.pack(filenames, output, format, endian, opts)
  File "3DSkit/pack/__init__.py", line 9, in pack
    func(*args)
  File "3DSkit/util/funcops.py", line 14, in __new__
    return self.main(*args, **kwargs)
  File "3DSkit/pack/BFLIM.py", line 77, in main
    img = self.swizzle(img, opts['swizzle'])
KeyError: 'swizzle'

Opts is an empty dict, so it has no swizzle option. Also, self.swizzle doesn't seem to be a function.

Show message if program is run without any arguments

Right now, if the program is run without any arguments, it quits. This isn't very helpful, and may make people think the program is broken. You already have a message written out, displayed upon --help, so perhaps that message should show up if no arguments are given?

Contributing

Hello, @hlixed sent me, I figured I'd help out since python and Nintendo is my expertise, I started by looking at the Yaz0 decompressor which could probably use some rework (I haven't tested it vs. my implementation), but I wanted to send this your way
https://gist.github.com/NWPlayer123/05dd55dd1cce6eb5457741ce14f48587
some things I learned/would suggest:
ord + bit-manipulation is far faster than using struct
could probably use BytesIO instead of an array (I use wb+ to seek back and forth, might be faster to write in one go instead of allocating all the zeroes )
I dunno what you're doing here, but I'd recommend doing the masking like I do.
I dunno how the cycles compare on what I do vs. what you're doing, but I'd also do this so we don't have to catch anything, one less compare (I made the same mistake before)
My fancy buffer trick would also save some time since you don't have to keep a track of the full amount back you can go (0x111, 0xFF on third byte + the 0x12 added)
Also probably a slight waste to re-read the first two bytes.
One fun fact: you have to read byte-by-byte cuz their algo lets stuff overlap (aka, data isn't already there if you just copy the whole size at once), but you don't have to every time, only if size is bigger than how far back you go.

BFLIM unpacking/repacking of non-multiple of 8px images results in weirdly distorted images

(Firstly, unpacking an alyt file seems to produce two folders: one x folder containing the alyt folder, and one x_ folder with the timg and other folders.)

I've downloaded this nice cat off the internet to serve as a test image (downscaled here).

I then packed it into a BFLIM:
python3 3DSkit.py -p testcat.png -f BFLIM -o testcat_packed.bflim
and took it back out again:
python3 3DSkit.py -x testcat_packed.bflim

However, in the process, 3DSkit horribly mangled this poor kitty. You wouldn't want to leave this poor innocent kitty like this, would you?

However, I've noticed this doesn't happen with all images: This quick test Kirby, for example, passes through extraction and repacking just fine. (excuse his appearance; he only had a few seconds to compose himself)

Do you know why this might be happening? Is it something to do with the dimensions of the image?

Error extracting .bflyt "KeyError: '__parent'"

Trying to extract a .bflyt to JSON and I'm getting KeyError: '__parent'.

Terminal Output:
PS D:\Switch Modding\romfs\lyt\ResidentMenu\blyt> python C:\Users\imado\Downloads\3DSkit-master\3DSkit-master\3dSkit.py -x RdtBase.bflyt

--------RdtBase.bflyt--------
No compression
BFLYT file found
Extracting...
Traceback (most recent call last):
File "C:\Users\imado\Downloads\3DSkit-master\3DSkit-master\3dSkit.py", line 249, in
result = main(args, opts)
File "C:\Users\imado\Downloads\3DSkit-master\3DSkit-master\3dSkit.py", line 169, in main
extract_files(filename, args.big, args.format, args.verbose, opts)
File "C:\Users\imado\Downloads\3DSkit-master\3DSkit-master\3dSkit.py", line 112, in extract_files
unpack.extract(filename, file, format, endian, verbose, opts)
File "C:\Users\imado\Downloads\3DSkit-master\3DSkit-master\unpack_init_.py", line 16, in extract
unpacker = cls(*args)
File "C:\Users\imado\Downloads\3DSkit-master\3DSkit-master\util\utils.py", line 18, in new
return self.main(*args, **kwargs)
File "C:\Users\imado\Downloads\3DSkit-master\3DSkit-master\unpack\BFLYT.py", line 66, in main
self.readdata(data)
File "C:\Users\imado\Downloads\3DSkit-master\3DSkit-master\unpack\BFLYT.py", line 107, in readdata
self.curparent = self.curparent['__parent']
KeyError: '__parent'
PS D:\Switch Modding\romfs\lyt\ResidentMenu\blyt>

Error unpacking new BFLYT Nintendo Switch

Trying to extract a BFLYT from the Nintendo Switch ResidentMenu.szs. Files in ResidentMenu are mostly layout information, which the one I'm trying to decompress is one of. I am trying to convert it to readable text to change the order of a few items, and I am searching for a tool that enables me to do that. Wexos is the layout editor I'm using but it can't change the order of panels (and other items). This is what I am trying to accomplish:

My file:
https://drive.google.com/file/d/1lCrVG1bG0chnk7p8ypakEahvAkFW9h7Q/view?usp=sharing
I thought it wouldn't hurt to try, but the following error message is stopping me:

--------RdtBtnShop.bflyt--------
No compression
BFLYT file found
Extracting...
Version: 00000000
Section count: 47
Traceback (most recent call last):
  File "3DSkit.py", line 249, in <module>
    result = main(args, opts)
  File "3DSkit.py", line 169, in main
    extract_files(filename, args.big, args.format, args.verbose, opts)
  File "3DSkit.py", line 112, in extract_files
    unpack.extract(filename, file, format, endian, verbose, opts)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\unpack\__init__.py", line 16, in extract
    unpacker = cls(*args)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\util\utils.py", line 18, in __new__
    return self.main(*args, **kwargs)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\unpack\BFLYT.py", line 133, in main
    write(dump(self.parsedata()), outfile)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\unpack\BFLYT.py", line 192, in parsedata
    method(chunk)
  File "S:\Mijn Drive\Things\NSW\Themes\Very nice theme\tools\3DSkit\unpack\BFLYT.py", line 508, in readusd1
    secname = 'usd1-%s' % self.actnode['__prevname']
KeyError: '__prevname'```

BFFNT OverflowError: signed integer is greater than maximum

When extracting a large BFFNT (8 MB), the program returns

OverflowError: signed integer is greater than maximum

It extracts many other, smaller BFFNT's successfully but not the larger one.

EDIT: This is a Switch BFFNT