binref/refinery

High Octane Triage Analysis

refinery's Issues

`peek -m` does not show metadata, only peeked output

The peek unit states, in its help output, that the -m, --meta flag does the following:

Only display attached metadata, do not add the peek value.

However, it seems that the peek unit currently does not display the metadata at all if the -m flag is specified; it behaves the same as if -r, --bare is specified, and only shows the peeked data:

$ emit yosemite.png | peek
-------------------------------------------------------------------------------------------------------------------------------------------------------------
peek = 5.441 MB; PNG image data, 2560 x 1440, 8-bit/color RGB, non-interlaced
-------------------------------------------------------------------------------------------------------------------------------------------------------------
000000: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52 00 00 0A 00 00 00 05 A0 08 02 00 00 00 1D 62 8D 88 00 00 00 09  .PNG........IHDR..............b......
000025: 70 48 59 73 00 00 0B 13 00 00 0B 13 01 00 9A 9C 18 00 00 0A 4F 69 43 43 50 50 68 6F 74 6F 73 68 6F 70 20 49 43  pHYs................OiCCPPhotoshop.IC
00004A: 43 20 70 72 6F 66 69 6C 65 00 00 78 DA 9D 53 67 54 53 E9 16 3D F7 DE F4 42 4B 88 80 94 4B 6F 52 15 08 20 52 42  C.profile..x..SgTS..=...BK...KoR...RB
00006F: 8B 80 14 91 26 2A 21 09 10 4A 88 21 A1 D9 15 51 C1 11 45 45 04 1B C8 A0 88 03 8E 8E 80 8C 15 51 2C 0C 8A 0A D8  ....&*!..J.!...Q..EE...........Q,....
000094: 07 E4 21 A2 8E 83 A3 88 8A CA FB E1 7B A3 6B D6 BC F7 E6 CD FE B5 D7 3E E7 AC F3 9D B3 CF 07 C0 08 0C 96 48 33  ..!.........{.k........>...........H3
0000B9: 51 35 80 0C A9 42 1E 11 E0 83 C7 C4 C6 E1 E4 2E 40 81 0A 24 70 00 10 08 B3 64 21 73 FD 23 01 00 F8 7E 3C 3C 2B  Q5...B..........@..$p....d!s.#...~<<+
0000DE: 22 C0 07 BE 00 01 78 D3 0B 08 00 C0 4D 9B C0 30 1C 87 FF 0F EA 42 99 5C 01 80 84 01 C0 74 91 38 4B 08 80 14 00  ".....x.....M..0.....B.\.....t.8K....
000103: 40 7A 8E 42 A6 00 40 46 01 80 9D 98 26 53 00 A0 04 00 60 CB 63 62 E3 00 50 2D 00 60 27 7F E6 D3 00 80 9D F8 99  @z.B..@F....&S....`.cb..P-.`'........
000128: 7B 01 00 5B 94 21 15 01 A0 91 00 20 13 65 88 44 00 68 3B 00 AC CF 56 8A 45 00 58 30 00 14 66 4B C4 39 00 D8 2D  {..[.!.......e.D.h;...V.E.X0..fK.9..-
00014D: 00 30 49 57 66 48 00 B0 B7 00 C0 CE 10 0B B2 00 08 0C 00 30 51 88 85 29 00 04 7B 00 60 C8 23 23 78 00 84 99 00  .0IWfH.............0Q..)..{.`.##x....
-------------------------------------------------------------------------------------------------------------------------------------------------------------

$ emit yosemite.png | peek --meta
-------------------------------------------------------------------------------------------------------------------------------------------------------------
000000: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52 00 00 0A 00 00 00 05 A0 08 02 00 00 00 1D 62 8D 88 00 00 00 09  .PNG........IHDR..............b......
000025: 70 48 59 73 00 00 0B 13 00 00 0B 13 01 00 9A 9C 18 00 00 0A 4F 69 43 43 50 50 68 6F 74 6F 73 68 6F 70 20 49 43  pHYs................OiCCPPhotoshop.IC
00004A: 43 20 70 72 6F 66 69 6C 65 00 00 78 DA 9D 53 67 54 53 E9 16 3D F7 DE F4 42 4B 88 80 94 4B 6F 52 15 08 20 52 42  C.profile..x..SgTS..=...BK...KoR...RB
00006F: 8B 80 14 91 26 2A 21 09 10 4A 88 21 A1 D9 15 51 C1 11 45 45 04 1B C8 A0 88 03 8E 8E 80 8C 15 51 2C 0C 8A 0A D8  ....&*!..J.!...Q..EE...........Q,....
000094: 07 E4 21 A2 8E 83 A3 88 8A CA FB E1 7B A3 6B D6 BC F7 E6 CD FE B5 D7 3E E7 AC F3 9D B3 CF 07 C0 08 0C 96 48 33  ..!.........{.k........>...........H3
0000B9: 51 35 80 0C A9 42 1E 11 E0 83 C7 C4 C6 E1 E4 2E 40 81 0A 24 70 00 10 08 B3 64 21 73 FD 23 01 00 F8 7E 3C 3C 2B  Q5...B..........@..$p....d!s.#...~<<+
0000DE: 22 C0 07 BE 00 01 78 D3 0B 08 00 C0 4D 9B C0 30 1C 87 FF 0F EA 42 99 5C 01 80 84 01 C0 74 91 38 4B 08 80 14 00  ".....x.....M..0.....B.\.....t.8K....
000103: 40 7A 8E 42 A6 00 40 46 01 80 9D 98 26 53 00 A0 04 00 60 CB 63 62 E3 00 50 2D 00 60 27 7F E6 D3 00 80 9D F8 99  @z.B..@F....&S....`.cb..P-.`'........
000128: 7B 01 00 5B 94 21 15 01 A0 91 00 20 13 65 88 44 00 68 3B 00 AC CF 56 8A 45 00 58 30 00 14 66 4B C4 39 00 D8 2D  {..[.!.......e.D.h;...V.E.X0..fK.9..-
00014D: 00 30 49 57 66 48 00 B0 B7 00 C0 CE 10 0B B2 00 08 0C 00 30 51 88 85 29 00 04 7B 00 60 C8 23 23 78 00 84 99 00  .0IWfH.............0Q..)..{.`.##x....
-------------------------------------------------------------------------------------------------------------------------------------------------------------

$ emit yosemite.png | peek --bare
-------------------------------------------------------------------------------------------------------------------------------------------------------------
000000: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52 00 00 0A 00 00 00 05 A0 08 02 00 00 00 1D 62 8D 88 00 00 00 09  .PNG........IHDR..............b......
000025: 70 48 59 73 00 00 0B 13 00 00 0B 13 01 00 9A 9C 18 00 00 0A 4F 69 43 43 50 50 68 6F 74 6F 73 68 6F 70 20 49 43  pHYs................OiCCPPhotoshop.IC
00004A: 43 20 70 72 6F 66 69 6C 65 00 00 78 DA 9D 53 67 54 53 E9 16 3D F7 DE F4 42 4B 88 80 94 4B 6F 52 15 08 20 52 42  C.profile..x..SgTS..=...BK...KoR...RB
00006F: 8B 80 14 91 26 2A 21 09 10 4A 88 21 A1 D9 15 51 C1 11 45 45 04 1B C8 A0 88 03 8E 8E 80 8C 15 51 2C 0C 8A 0A D8  ....&*!..J.!...Q..EE...........Q,....
000094: 07 E4 21 A2 8E 83 A3 88 8A CA FB E1 7B A3 6B D6 BC F7 E6 CD FE B5 D7 3E E7 AC F3 9D B3 CF 07 C0 08 0C 96 48 33  ..!.........{.k........>...........H3
0000B9: 51 35 80 0C A9 42 1E 11 E0 83 C7 C4 C6 E1 E4 2E 40 81 0A 24 70 00 10 08 B3 64 21 73 FD 23 01 00 F8 7E 3C 3C 2B  Q5...B..........@..$p....d!s.#...~<<+
0000DE: 22 C0 07 BE 00 01 78 D3 0B 08 00 C0 4D 9B C0 30 1C 87 FF 0F EA 42 99 5C 01 80 84 01 C0 74 91 38 4B 08 80 14 00  ".....x.....M..0.....B.\.....t.8K....
000103: 40 7A 8E 42 A6 00 40 46 01 80 9D 98 26 53 00 A0 04 00 60 CB 63 62 E3 00 50 2D 00 60 27 7F E6 D3 00 80 9D F8 99  @z.B..@F....&S....`.cb..P-.`'........
000128: 7B 01 00 5B 94 21 15 01 A0 91 00 20 13 65 88 44 00 68 3B 00 AC CF 56 8A 45 00 58 30 00 14 66 4B C4 39 00 D8 2D  {..[.!.......e.D.h;...V.E.X0..fK.9..-
00014D: 00 30 49 57 66 48 00 B0 B7 00 C0 CE 10 0B B2 00 08 0C 00 30 51 88 85 29 00 04 7B 00 60 C8 23 23 78 00 84 99 00  .0IWfH.............0Q..)..{.`.##x....
-------------------------------------------------------------------------------------------------------------------------------------------------------------

Reproduced with binary-refinery version 0.4.30, Python 3.10.4.

xtzip raises when password is a variable

Love this tool and currently learning to use it. ❤️

I followed one tutorial linked in the readme and noticed it crashes as soon as you try to decrypt the zip with a variable.

Here's the stack trace:

Traceback (most recent call last):
  File "/home/<redacted>/Tools/refinery/refinery/units/__init__.py", line 1256, in normalized_action
    yield from (x for x in result if x is not None)
  File "/home/<redacted>/Tools/refinery/refinery/units/__init__.py", line 1256, in <genexpr>
    yield from (x for x in result if x is not None)
  File "/home/<redacted>/Tools/refinery/refinery/units/__init__.py", line 681, in <genexpr>
    return (self.labelled(r) for r in result)
  File "/home/<redacted>/Tools/refinery/refinery/units/formats/__init__.py", line 114, in process
    results: List[UnpackResult] = list(self.unpack(data))
  File "/home/<redacted>/Tools/refinery/refinery/units/formats/archive/xtzip.py", line 36, in unpack
    archive.setpassword(self.args.pwd)
  File "/usr/lib/python3.10/zipfile.py", line 1449, in setpassword
    raise TypeError("pwd: expected bytes, got %s" % type(pwd).__name__)
TypeError: pwd: expected bytes, got ByteStringWrapper

There seems to be a typecheck inside the zipfile library code. Might be due to version changes, not sure. Anyway, I fixed it for you. PR incoming. :)

Usage example for Push does not work

I can't get the usage example for push to work:

emit key=value | push [[| rex =(.*)$ $1 | pop v ]| repl var:v censored ]

First off, the rex syntax seems to have changed, the $1 will be taken literally:

❯ r.emit "key=value" | r.rex '=(.*)' '$1' 
$1

Instead it should probably be {1}:

❯ r.emit "key=value" | r.rex '=(.*)' '{1}' 
value

But even after correcting this it still fails because the variable is not defined:

> r.emit "key=value" |  r.push -vv [ [ | r.rex -vv '=(.*)' '{1}' | r.pop -vv v ] | r.repl --v 'var:v' censored ]
 
(07:30:25) verbose in rex: regular expression: re.compile(b'=(.*)', re.DOTALL)
(07:30:25) verbose in pop: buffering invisible chunk
--- Logging error ---
Traceback (most recent call last):
  File "/home/zac/.local/lib/python3.8/site-packages/refinery/lib/argformats.py", line 621, in extract
    result = meta[name]
  File "/home/zac/.local/lib/python3.8/site-packages/refinery/lib/meta.py", line 500, in __getitem__
    item = super().__getitem__(key)
  File "/home/zac/.local/lib/python3.8/site-packages/refinery/lib/meta.py", line 511, in __missing__
    raise KeyError(F'The meta variable {key} is unknown.')
KeyError: 'The meta variable v is unknown.'

Maybe I'm doing the whole variables thing wrong, because even this fails becase k is not set:

> r.emit "test" | r.put -vv k "test" | r.repl -vv var:k hello
Message: 'delayed argument initialization failed:'
Arguments: ('The variable k is not defined.',)

I'm using version Version: 0.4.6 of binary-refinery, on Ubuntu with bash.

machometa

Specification

Unit Request for machometa

Requesting a new unit to perform similar functionality as pemeta, but for macho executables. The data requested inside of this would ideally include:

• Header info (machine arch, num of load commands, etc - this is already parsed in peek)

• Dylibs (externally linked dependencies / libs - lief can parse this )

• Code-signing information (mostly just the name but if extracting other information is readily do-able, that would be great! Example parsing in Go: https://github.com/blacktop/go-macho/blob/master/pkg/codesign/codesign.go )

• Imports/Exports/Symbols (optional flag - lief can parse this information)

• Class Information (Not necessary but cool as an option flag - ObjC files only) (via class-dump - not python :( https://github.com/nygard/class-dump ))

I realize I'm not giving a ton of great info here but I'm currently using a bunch of Apple utilities or python snippets to triage Macho samples and I figured others would benefit from this being in binref :)

Example (awful) Parsing Code to show dylibs and signature data - works for most ARM and x86 bins but not FATs

import sys
import lief
from hashlib import md5
import argparse
import os

def main():

	parser = argparse.ArgumentParser(description="Macho Sig Name Extraction")
	parser.add_argument("-f", "--file", help="Specify file to parse", metavar="<file>", required=False)
	args = parser.parse_args()

	result = sig_reading(args.file)
	if result is not None:
		print("\tTarget File: \t" + result[0])
		print("\tFile MD5: \t" + result[1])
		print("\tSig Name: \t" + result[2])
		print("\tDylibs: \t")
		for i in result[3]:
			print("\t\t"+i)



def sig_reading(target):

	dylibs = []

	try:
		target_macho = open(target, mode="rb")
		contents = target_macho.read()
		file_hash = md5(contents).hexdigest()
		parsed_macho = lief.parse(target)
		for lib in parsed_macho.libraries:
			dylibs.append(lib.name.lower())
		cs_sign_dir_offset = parsed_macho.code_signature.data_offset

		# read the big CS directory & get ptr to 0th blob
		target_macho.seek(cs_sign_dir_offset)
		cs_dir_bytes = target_macho.read(0x20)
		jump_to_blob = cs_dir_bytes[19]

		# read the 0th blob and look for ident ofset
		target_macho.seek(cs_sign_dir_offset+jump_to_blob)
		first_codesign_blob = target_macho.read(0x20)
		jump_to_ident = first_codesign_blob[23]

		# read identifier string
		target_macho.seek(cs_sign_dir_offset+jump_to_blob+jump_to_ident)
		ident_str = target_macho.read(0x30)
		sig_name = str(ident_str)
		sig_name = sig_name.split('\\x00')[0]
		sig_name = sig_name[2:]
		target_macho.close()
	except:
		print("not a macho")
		return


	sig_list = [target,file_hash,sig_name,dylibs]
	return sig_list
main()

rc5 and rc6: CTR mode does not work

Description

rc5 and rc6 cannot encrypt with CTR mode since version 0.5.10.

To Reproduce

Master:

$ echo -n  "This is a secret message." | rc5 -m ctr -i 01234567 -R 0123456789abcdef
(00:44:40) failure in rc5: exception of type ValueError; Information in counter object does not align with block size.
$ echo -n  "This is a secret message." | rc6 -m ctr -i 0123456789abcdef -R 0123456789abcdef
(00:45:03) failure in rc6: exception of type ValueError; Information in counter object does not align with block size.
$

Version 0.5.9 works:

$ echo -n  "This is a secret message." | rc5 -m ctr -i 01234567 -R 0123456789abcdef | rc5 -m ctr -i 01234567 0123456789abcdef
This is a secret message.
$ echo -n  "This is a secret message." | rc6 -m ctr -i 0123456789abcdef -R 0123456789abcdef | rc6 -m ctr -i 0123456789abcdef 01
23456789abcdef
This is a secret message.
$

Environment

• Operating System: Ubuntu 20.04.6 LTS (WSL2)
• Python Version: 3.8.10
• Refinery Version: master

camellia: padding incompatibilities with OpenSSL for CFB, OFB, and CTR modes

Description

camellia command has padding incompatibilities with OpenSSL for CFB, OFB, and CTR modes. OpenSSL does not add padding and camellia command adds padding for these modes.

When a plaintext is encrypted with openssl command then the ciphertext output is decrypted with camellia command, the plaintext output is truncated. On the other hand, when a plaintext is encrypted with camellia command then the ciphertext output is decrypted with openssl command, the plaintext output has redundant trailing bytes. These behaviors do not occur for ECB and CBC modes. Truncation of plaintext outputs on decryption is an especially critical issue.

According to the article of Block cipher mode of operation in the WikiPedia, CFB, OFB and CTR modes do not require any special measures to handle messages whose lengths are not multiples of the block size. So I think it is incorrect that camellia command requires input as a multiple of the 16-byte block size for CFB, OFB and CTR modes.

I use Binary Refinery as Python modules for writing some plugins of FileInsight-plugins. Binary Refinery is so great to support a lot of encodings, crypto algorithms, and compression algorithms.

To Reproduce

Encryption with ECB mode (identical output):

$ echo "This is a secret message." | openssl enc -e -camellia-128-ecb -K 30313233343536373839616263646566 | xxd
00000000: c4eb d417 c468 8165 2367 99f1 c5e0 bcb8  .....h.e#g......
00000010: ff47 867e 39ca b441 a4c3 0679 4a48 00bb  .G.~9..A...yJH..
$

$ echo "This is a secret message." | camellia --mode ecb -R 0123456789abcdef | xxd
00000000: c4eb d417 c468 8165 2367 99f1 c5e0 bcb8  .....h.e#g......
00000010: ff47 867e 39ca b441 a4c3 0679 4a48 00bb  .G.~9..A...yJH..
$

---

Encryption with CBC mode (identical output):

$ echo "This is a secret message." | openssl enc -e -camellia-128-cbc -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: cda7 a782 bce6 4ca8 6c3a a804 b141 ce6c  ......L.l:...A.l
00000010: c1c4 8c07 fb65 357d 21d0 4da9 6a2c 0f49  .....e5}!.M.j,.I
$

$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode cbc -R 0123456789abcdef | xxd
00000000: cda7 a782 bce6 4ca8 6c3a a804 b141 ce6c  ......L.l:...A.l
00000010: c1c4 8c07 fb65 357d 21d0 4da9 6a2c 0f49  .....e5}!.M.j,.I
$

---

Encryption with CFB mode (different outputs):

$ echo "This is a secret message." | openssl enc -e -camellia-128-cfb -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 3e1a c0f8 aa74 e546 8925 eb0a 1776 1127  >....t.F.%...v.'
00000010: 6833 fe93 bb59 043d 8b3e                 h3...Y.=.>
$

$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode cfb -S 128 -R 0123456789abcdef | xxd
00000000: 3e1a c0f8 aa74 e546 8925 eb0a 1776 1127  >....t.F.%...v.'
00000010: 6833 fe93 bb59 043d 8b3e abce 0701 8677  h3...Y.=.>.....w

---

Encryption with OFB mode (different outputs):

$ echo "This is a secret message." | openssl enc -e -camellia-128-ofb -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 3e1a c0f8 aa74 e546 8925 eb0a 1776 1127  >....t.F.%...v.'
00000010: 36dd fea5 869c 171f 547c                 6.......T|

$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode ofb -R 0123456789abcdef | xxd
00000000: 3e1a c0f8 aa74 e546 8925 eb0a 1776 1127  >....t.F.%...v.'
00000010: 36dd fea5 869c 171f 547c 0363 e05a 2850  6.......T|.c.Z(P
$

---

Encryption with CTR mode (different outputs):

$ echo "This is a secret message." | openssl enc -e -camellia-128-ctr -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 3e1a c0f8 aa74 e546 8925 eb0a 1776 1127  >....t.F.%...v.'
00000010: 472b 60a4 8c98 0540 ce1a                 G+`....@..
$

$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode ctr -R 0123456789abcdef | xxd
00000000: 3e1a c0f8 aa74 e546 8925 eb0a 1776 1127  >....t.F.%...v.'
00000010: 472b 60a4 8c98 0540 ce1a dd81 0101 dc8f  G+`....@........
$

---

Encryption with openssl command and decryption with camellia command (ECB mode, works correctly):

$ echo "This is a secret message." | openssl enc -e -camellia-128-ecb -K 30313233343536373839616263646566 | camellia --mode ecb 0123456789abcdef | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574  This is a secret
00000010: 206d 6573 7361 6765 2e0a                  message..
$

Encryption with openssl command and decryption with camellia command (CBC mode, works correctly):

$ echo "This is a secret message." | openssl enc -e -camellia-128-cbc -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | camellia --iv 0123456789abcdef --mode cbc -S 128 -L 0123456789abcdef | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574  This is a secret
00000010: 206d 6573 7361 6765 2e0a                  message..
$

Encryption with openssl command and decryption with camellia command (CFB mode, the plaintext output is truncated):

$ echo "This is a secret message." | openssl enc -e -camellia-128-cfb -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | camellia --iv 0123456789abcdef --mode cfb -S 128 -L 0123456789abcdef | xxd
(22:20:55) warning in camellia: removing 10 bytes from the input to make it a multiple of the 16-byte block size
00000000: 5468 6973 2069 7320 6120 7365 6372 6574  This is a secret
$

Encryption with openssl command and decryption with camellia command (OFB mode, the plaintext output is truncated):

$ echo "This is a secret message." | openssl enc -e -camellia-128-ofb -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | camellia --iv 0123456789abcdef --mode ofb -L 0123456789abcdef | xxd
(22:23:24) warning in camellia: removing 10 bytes from the input to make it a multiple of the 16-byte block size
00000000: 5468 6973 2069 7320 6120 7365 6372 6574  This is a secret
$

Encryption with openssl command and decryption with camellia command (CTR mode, the plaintext output is truncated):

$ echo "This is a secret message." | openssl enc -e -camellia-128-ctr -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | camellia --iv 0123456789abcdef --mode ctr -L 0123456789abcdef | xxd
(22:24:28) warning in camellia: removing 10 bytes from the input to make it a multiple of the 16-byte block size
00000000: 5468 6973 2069 7320 6120 7365 6372 6574  This is a secret
$

---

Encryption with camellia command and decryption with openssl command (ECB mode, works correctly):

$ echo "This is a secret message." | camellia --mode ecb -R 0123456789abcdef | openssl enc -d -camellia-128-ecb -K 30313233343536373839616263646566 | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574  This is a secret
00000010: 206d 6573 7361 6765 2e0a                  message..
$

Encryption with camellia command and decryption with openssl command (CBC mode, works correctly):

$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode cbc -R 0123456789abcdef | openssl enc -d -camellia-128-cbc -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574  This is a secret
00000010: 206d 6573 7361 6765 2e0a                  message..

Encryption with camellia command and decryption with openssl command (CFB mode, the plaintext output has redundant trailing bytes):

$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode cfb -S 128 -R 0123456789abcdef | openssl enc -d -camellia-128-cfb -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574  This is a secret
00000010: 206d 6573 7361 6765 2e0a 0606 0606 0606   message........
$

Encryption with camellia command and decryption with openssl command (OFB mode, the plaintext output has redundant trailing bytes):

$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode ofb -R 0123456789abcdef | openssl enc -d -camellia-128-ofb -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574  This is a secret
00000010: 206d 6573 7361 6765 2e0a 0606 0606 0606   message........
$

Encryption with camellia command and decryption with openssl command (CTR mode, the plaintext output has redundant trailing bytes):

$ echo "This is a secret message." | camellia --iv 0123456789abcdef --mode ctr -R 0123456789abcdef | openssl enc -d -camellia-128-ctr -K 30313233343536373839616263646566 -iv 30313233343536373839616263646566 | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574  This is a secret
00000010: 206d 6573 7361 6765 2e0a 0606 0606 0606   message........
$

Environment

• Operating System
  Ubuntu 20.04.6 LTS (WSL2)
• Python Version
  3.8.10
• Refinery Version
  0.5.8

xtchm - extract Compiled Help Modules

Specification

It is possible to weaponize .chm files but binref can't extract this files.
There is a python lib PyCHM but this is just a wrapper for this c lib CHMLib. The c lib needs a string with the path to the chm file to open it. This is against the binref code of conduct.

I think the only solution would be to implement the algorithm in python as a new binref unit.

@huettenhain can you prove that there isn't a other way to extract chm files with binref? If so I can start to develop a new unit.

Test Cases

Malicious-CHM-Guide.md
AgentTesla Spreads Through CHM and PDF Files in Recent Attacks
Cryptowall Makes a Comeback Via Malicious Help Files (CHM)
Harmful Help: Analyzing a Malicious Compiled HTML Help File Delivering Agent Tesla

`vstack` unit fails to execute on binref installations with Python 3.12 due to failed Unicorn import

Description

When running the vstack unit on an installation of Binary Refinery in a Python virtual environment running Python 3.12, I kept seeing the following error:

failure in vstack: dependency unicorn is missing; run pip install intervaltree capstone unicorn

This was despite verifying inside the virtual environment that the listed dependencies were present.

To Reproduce

Install Binary Refinery with the binary-refinery[all] spec in an environment which uses Python 3.12, then execute the vstack unit.

Re-installing refinery inside a virtual environment with Python 3.11.6 fixes this issue.

Cause

As far as I can tell, this is the reason for this issue:

• Python 3.12 removes the distutils module from the standard library (which was deprecated starting in 3.10: https://docs.python.org/3.12/whatsnew/3.10.html#distutils-deprecated)
• The Unicorn bindings for Python still imports distutils: https://github.com/unicorn-engine/unicorn/blob/d4b92485b1a228fb003e1218e42f6c778c655809/bindings/python/unicorn/unicorn.py#L5
• This results in a failure to import unicorn when executing the vstack unit, which results in a RefineryImportMissing exception, which is reported by refinery as a missing dependency.

Environment

• Operating System: Fedora Linux 39 (which uses Python 3.12 by default)
• Python Version: 3.12.0
• Refinery Version: 0.6.24

Additional Context

Verbose error and traceback:

$ vstack -v -v
(13:31:14) failure in vstack: dependency unicorn is missing; run pip install unicorn capstone intervaltree
Traceback (most recent call last):
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 1251, in __get__
    self.module = module = self.fget()
                           ^^^^^^^^^^^
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/formats/exe/vstack.py", line 74, in _unicorn
    import unicorn
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/unicorn/__init__.py", line 4, in <module>
    from .unicorn import Uc, uc_version, uc_arch_supported, version_bind, debug, UcError, __version__
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/unicorn/unicorn.py", line 5, in <module>
    import distutils.sysconfig
ModuleNotFoundError: No module named 'distutils'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 1391, in normalized_action
    yield from (x for x in result if x is not None)
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 1391, in <genexpr>
    yield from (x for x in result if x is not None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 806, in <genexpr>
    return (Chunk.Wrap(r) for r in result)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/formats/exe/vstack.py", line 146, in process
    uc = self._unicorn
         ^^^^^^^^^^^^^
  File "/home/cxiao/.local/pipx/venvs/binary-refinery-br/lib64/python3.12/site-packages/refinery/units/__init__.py", line 1257, in __get__
    raise RefineryImportMissing(self.dependency, *args) from E
refinery.units.RefineryImportMissing

Deprecation warnings when calling update.ps1

Description

When calling ./update.ps1 I get a couple of log lines starting with DEPRECATION. I think this is the important snippet from the output:

Successfully built binary-refinery jsbeautifier
Installing collected packages: z3-solver, win-unicode-console, unicorn, unicodecsv, texttable, sortedcontainers, red-black-tree-mod, pyxlsb2, pywin32, python-magic, pysmt, pyperclip, pyelftools, pycdlib, ply, phpserialize, mulpyplexer, msgpack, mpmath, LnkParse3, lark-parser, javaobj-py3, itanium-demangler, hexdump, enum-compat, editorconfig, ebcdic, easygui, dpkt, compressed-rtf, cabarchive, brotli, asn1crypto, altgraph, aenum, xlrd2, wheel, tzdata, typing_extensions, toml, tbtrim, sympy, soupsieve, smmap, six, scapy, roman, pyzstd, python-utils, python-registry, python-magic-win64, pyppmd, pyparsing, pyonenote, pycryptodomex, pycryptodome, pycparser, pybcj, psutil, protobuf, plumbum, Pillow, pefile, olefile, numpy, networkx, multivolumefile, macholib, intervaltree, inflate64, future, et-xmlfile, dictdumper, defusedxml, decorator, CppHeaderParser, configparser, colorclass, colorama, chardet, capstone, cachetools, bitstring, archinfo, ailment, untangle, rpyc, pytz-deprecation-shim, PyPDF2, pypcapkit, py7zr, progressbar2, openpyxl, nampa, more-itertools, jsbeautifier, imapclient, gitdb, click, claripy, cffi, beautifulsoup4, zipp, xdis, tzlocal, spark-parser, pyvex, GitPython, cryptography, uncompyle6, python-evtx, msoffcrypto-tool, decompyle3, cle, XLMMacroDeobfuscator, angr, pcodedmp, oletools, RTFDE, extract-msg, binary-refinery
  DEPRECATION: win-unicode-console is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for win-unicode-console ... done
  DEPRECATION: red-black-tree-mod is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for red-black-tree-mod ... done
  DEPRECATION: mulpyplexer is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for mulpyplexer ... done
  DEPRECATION: hexdump is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for hexdump ... done
  DEPRECATION: scapy is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for scapy ... done
  DEPRECATION: intervaltree is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for intervaltree ... done
  DEPRECATION: future is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for future ... done
  DEPRECATION: CppHeaderParser is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for CppHeaderParser ... done
  DEPRECATION: pypcapkit is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for pypcapkit ... done
Successfully installed CppHeaderParser-2.7.4 GitPython-3.1.31 LnkParse3-1.2.0 Pillow-9.5.0 PyPDF2-3.0.1 RTFDE-0.0.2 XLMMacroDeobfuscator-0.2.7 aenum-3.1.12 ailment-9.2.46 altgraph-0.17.3 angr-9.2.46 archinfo-9.2.46 asn1crypto-1.5.1 beautifulsoup4-4.11.2 binary-refinery-0.6.1 bitstring-4.0.1 brotli-1.0.9 cabarchive-0.2.4 cachetools-5.3.0 capstone-4.0.2 cffi-1.15.1 chardet-5.1.0 claripy-9.2.46 cle-9.2.46 click-8.1.3 colorama-0.4.6 colorclass-2.2.2 compressed-rtf-1.0.6 configparser-4.0.2 cryptography-40.0.1 decompyle3-3.9.0 decorator-5.1.1 defusedxml-0.7.1 dictdumper-0.8.4.post2 dpkt-1.9.8 easygui-0.98.3 ebcdic-1.1.1 editorconfig-0.12.3 enum-compat-0.0.3 et-xmlfile-1.1.0 extract-msg-0.40.0 future-0.18.3 gitdb-4.0.10 hexdump-3.3 imapclient-2.3.1 inflate64-0.3.1 intervaltree-3.1.0 itanium-demangler-1.1 javaobj-py3-0.4.3 jsbeautifier-1.14.7 lark-parser-0.12.0 macholib-1.16.2 more-itertools-5.0.0 mpmath-1.3.0 msgpack-1.0.5 msoffcrypto-tool-5.0.1 mulpyplexer-0.9 multivolumefile-0.2.3 nampa-0.1.1 networkx-3.1 numpy-1.24.2 olefile-0.46 oletools-0.60.1 openpyxl-3.1.2 pcodedmp-1.2.6 pefile-2023.2.7 phpserialize-1.3 plumbum-1.8.1 ply-3.11 progressbar2-4.2.0 protobuf-4.22.3 psutil-5.9.4 py7zr-0.20.4 pybcj-1.0.1 pycdlib-1.14.0 pycparser-2.21 pycryptodome-3.17 pycryptodomex-3.17 pyelftools-0.29 pyonenote-0.0.2 pyparsing-2.4.7 pypcapkit-0.15.5 pyperclip-1.8.2 pyppmd-1.0.0 pysmt-0.9.5 python-evtx-0.7.4 python-magic-0.4.13 python-magic-win64-0.4.13 python-registry-1.3.1 python-utils-3.5.2 pytz-deprecation-shim-0.1.0.post0 pyvex-9.2.46 pywin32-306 pyxlsb2-0.0.9 pyzstd-0.15.6 red-black-tree-mod-1.20 roman-4.0 rpyc-5.3.1 scapy-2.5.0 six-1.16.0 smmap-5.0.0 sortedcontainers-2.4.0 soupsieve-2.4 spark-parser-1.8.9 sympy-1.11.1 tbtrim-0.3.1 texttable-1.6.7 toml-0.10.2 typing_extensions-4.5.0 tzdata-2023.3 tzlocal-4.2 uncompyle6-3.9.0 unicodecsv-0.14.1 unicorn-2.0.1.post1 untangle-1.2.1 wheel-0.40.0 win-unicode-console-0.5 xdis-6.0.5 xlrd2-1.3.4 z3-solver-4.10.2.0 zipp-1.0.0

To Reproduce

Open a PowerShell window, navigate to the refinery directory and call ./update.ps1.

Environment

• Operating System: Windows 10
• Python Version: 3.9.1
• Refinery Version: master

Additional Context

The log lines reference pypa/pip#8559 so I assume this is caused by an upstream change in pip behavior.

iemap does not display colour

Description

I have noticed that in my environment, the iemap unit does not produce any colour, which unfortunately makes the unit not very useful. Other units which produce colour, such as peek, do emit colour in their output as expected.

To Reproduce

This behaviour is reproducible on both Linux and MacOS. Just for reproducing this bug, I created a new Python virtual environment on both the Linux and MacOS machines, and did a fresh install of Binary Refinery inside it, via pip install binary-refinery[all].

I have included screenshots below of the output of the peek unit, as well as the iemap unit. I passed the --legend flag to iemap in this case, and expect to see colours in at least the numbered legend to the left of the entropy graph.

• Operating System: Fedora Linux 37
• Python Version: 3.11.2
• Refinery Version: 0.6.0
• Shell: Bash
• Terminal Emulator: gnome-terminal

• Operating System: MacOS Ventura
• Python Version: 3.10.1
• Refinery version: 0.6.0
• Shell: Zsh
• Terminal Emulator: iTerm2

Fish support

emit is already a command in fish.

alias emit (which emit)

seems to be a suitable workaround

MSI Extraction

Specification

MSI files are already parsed well using the xtdoc unit. However, the embedded streams are often incorrectly labeled with nonsensical Chinese characters

Example (from https://bazaar.abuse.ch/sample/a00ae529a1ce4e2eb7f988ed5adaa07e653de3fe0c01bec9452aae71b935f63c/)

$ emit Projects/Malware/AdobePDFReader.msi | xtdoc -l 
[5]SummaryInformation
㡿㪆㦍㪄㫊㮅㨁㠀㬇㫄㩅㮌㣍㠍㥁㧆䠋
䡀㬿䏲䐸䖱
䡀㼿䕷䑬㭪䗤䠤
䡀㼿䕷䑬㹪䒲䠯
䡀㽿䅤䈯䠶
䡀㿿䏤䇬䗤䒬䠱
䡀䇊䌰㮱䈻䘦䈷䈜䘴䑨䈦
䡀䇊䗹䛎䆨䗸㼨䔨䈸䆱䠨
䡀䈏䗤䕸㬨䐲䒳䈱䗱䠶
䡀䈏䗤䕸䠨
䡀䈖䌧䠤
䡀䌍䈵䗦䕲䠼
䡀䌏䈯
䡀䑒䗶䏤㮯䈻䘦䈷䈜䘴䑨䈦
䡀䒌䓰䑲䑨䠷
䡀䒖䘧䈯䌜䑪䗤䕸䠨
䡀䕙䓲䕨䜷
䡀䘌䗶䐲䆊䌷䑲

Didier Stevens wrote plugin for oledump (https://github.com/DidierStevens/DidierStevensSuite/blob/master/plugin_msi_info.py ) that handles this issue & properly displays the correct embedded names. The cleaned up stream names make it much easier to discern what the MSI will drop and easier to select what stream to examine next or dump out

I don't know if its worth introducing a new unit (like xtmsi) or to just build that encoding fix into xtdoc and use it as a flag inside of the unit :)

Streams:
  1        620 SummaryInformation
  2        360 MPB_ExternalFiles.RunAction1_File1
  3     182784 MPB_ExternalFiles.RunAction2_File2
  4    2609864 Binary.CustomActionsModuleX64
  5        728 !_Columns
  6         36 !MPB_RunActions
  7         24 !MPB_ExternalFiles
  8      12676 !_StringData
  9       1756 !_StringPool
 10         40 !_Tables
 11       2232 !_Validation
 12        420 !ActionText
 13         48 !AdminExecuteSequence
 14         24 !AdminUISequence
 15         42 !AdvtExecuteSequence
 16          4 !FeatureComponents
 17         16 !Feature
 18          4 !Binary
 19         12 !Directory
 20        144 !InstallExecuteSequence
 21         72 !InstallUISequence
 22         12 !Component
 23         16 !Upgrade
 24         96 !Error
 25         48 !Property
 26        156 !CustomAction

Test Cases

Second comparison (truncated)

$ emit f0b6d6981e06c7be2e45650e5f6d39570c1ee640ccb157ddfe42ee23ad4d1cdb.bin | xtdoc -l
[5]SummaryInformation
䌋䄱䜵㷾䚨
䌋䄱䜵㹾䚲䕨䋜䏨㼯䕦䓬㵷䘤䆱䈫䞵䏧䠯
䌋䄱䜵㾾䠳
䌋䄱䜵䄾䆬䖸䄷䗦䇾䏯
䌋䄱䜵䄾䖬䋦䇨䏸䕨䇾䏯
䌋䄱䜵䄾䖬䋦䇨䏸䕨䞂䏧䠯
䌋䄱䜵䅾䑤䈱䠵
䌋䄱䜵䆾䇰䌯䎱䕤䒵䠺
䌋䄱䜵䆾䐲䏳䗨䠬
䌋䄱䜵䆾䖸䌷䒦䠱
䌋䄱䜵䇾䄬䒯䠪
䌋䄱䜵䈾䆻䄯䌰䠦
䌋䄱䜵䌾䉱䠲
䌋䄱䜵䌾䖱䌷䒦䠱
䌋䄱䜵䕾䐨䙲䆬䠲
䌋䄱䜵䕾䓨䌤䌵䠦
䌋䄱䜵䗾䅤䄥䎦
䡀㬿䏲䐸䖱
䡀㲊㼿䋦䇨䏸䇨䄝䎶䠶
䡀㲞䈝䗻
䡀㼿䕷䑬㭪䗤䠤
䡀㼿䕷䑬㹪䒲䠯
䡀㽿䅤䈯䠶
䡀㿿䏤䇬䗤䒬䠱
䡀䄕䑸䋦䒌䇱䗬䒬䠱
䡀䄛䌧㫲䗸䒷䠱
䡀䆊䌷䑲䈝䗻
䡀䇊䌰㮱䈻䘦䈷䈜䘴䑨䈦
䡀䇊䌰㾱㼒䔨䈸䆱䠨
䡀䇊䗹䛎䆨䗸㼨䔨䈸䆱䠨
䡀䈏䗤䕸㬨䐲䒳䈱䗱䠶
䡀䈏䗤䕸䠨
䡀䈛䌪䗶䜵
䡀䈝䗻䗜䏼䠨
䡀䋌䆨㫮䛲
䡀䌋䄱䜵
䡀䌍䈵䗦䕲䠼
䡀䌍䏤䊲
䡀䑒䗶䏤㮯䈻䘦䈷䈜䘴䑨䈦
䡀䑒䗶䏤㾯㼒䔨䈸䆱䠨
䡀䒋䗲䗶䄵䓳䕨㲞䈜䘴䑨䈦
䡀䒌䓰䑲䑨䠷
䡀䒌䗱䒵㬯䑲䌧䌷䑲
䡀䒌䗱䒵㮯䈹䗱
䡀䒌䗱䒵䠯
䡀䓞䕪䇤䠨
䡀䕌䄨䈷䒏䇯䕨
䡀䕎䒵䠵
䡀䕙䓲䕨䜷
䡀䘌䗶䐲䆊䌷䑲
䡀䙎䑨㶷䓤䌳䊱

$ python3 oledump.py -p plugin_msi_info.py f0b6d6981e06c7be2e45650e5f6d39570c1ee640ccb157ddfe42ee23ad4d1cdb.bin

Streams:
  1        316 SummaryInformation
  2        318 Binary.New
  3     627864 Binary.PowerShellScriptLauncher.dll
  4        318 Binary.Up
  5     390304 Binary.aicustact.dll
  6     146072 Binary.aischeduler.dll
  7     183448 Binary.aischeduler2.dll
  8       2818 Binary.banner
  9       2862 Binary.cmdlinkarrow
 10       2998 Binary.completi
 11       2998 Binary.custicon
 12      11791 Binary.dialog
 13        766 Binary.exclamic
 14       1078 Binary.info
 15       2998 Binary.insticon
 16       2998 Binary.removico
 17       2998 Binary.repairic
 18        854 Binary.tabback
 19       1480 !_Columns
 20         32 !AI_ScheduledTasks
 21        444 !UIText
 22      84211 !_StringData
 23       7560 !_StringPool
 24         76 !_Tables
 25       4440 !_Validation
 26         16 !LaunchCondition
 27         36 !RadioButton
 28        480 !ActionText
 29         48 !AdminExecuteSequence
 30         66 !AdminUISequence
 31         72 !AdvtExecuteSequence
 32          8 !FeatureComponents
 33         16 !Feature
 34         24 !Registry
 35         48 !TextStyle
 36         12 !CheckBox
 37         68 !Binary
 38         12 !Directory
 39        616 !Dialog
 40        462 !InstallExecuteSequence
 41        180 !InstallUISequence
 42         54 !BootstrapperUISequence
 43         24 !Component
 44         72 !ControlCondition
 45       1536 !ControlEvent
 46       7280 !Control
 47         32 !Upgrade
 48          4 !CreateFolder
 49       2564 !Error
 50        332 !Property
 51        372 !CustomAction
 52        128 !EventMapping

tea and xtea: making endianness configurable

For tea and xtea commands, block operation is done with little endian.

Excerpt from refinery/units/crypto/cipher/tea.py:

def tea_block_operation(
    blk: Callable[[int, int, int, int, int, int], Tuple[int, int]]
) -> Callable[[TEA, ByteString], ByteString]:
    def wrapped(self: TEA, data: ByteString) -> ByteString:
        v0, v1 = blk(
            int.from_bytes(data[:4], 'little'),
            int.from_bytes(data[4:], 'little'),
            *self.derived_key
        )
        return v0.to_bytes(4, 'little') + v1.to_bytes(4, 'little')
    return wrapped


class TEABase(BlockCipher):

    block_size = 8
    valid_key_sizes = {16}
    derived_key: List[int]

    @property
    def key(self):
        return self.derived_key

    @key.setter
    def key(self, key):
        self.derived_key = [int.from_bytes(key[k:k + 4], 'little') for k in range(0, 16, 4)]

However, there are implementations of TEA and XTEA that block operation is done with big endian.
For example, Crypt::XTEA Perl module has the little_endian option and the default is little_endian=0 (big endian). If block operation is done with big endian on encryption, tea and xtea commands cannot decrypt correctly. It would be great if endianness is configurable for tea and xtea commands.

A sample Perl script xtea.pl for encryption with XTEA:

use Crypt::XTEA;
use Crypt::ECB;

my $key = "0123456789abcdef";

# Little endian
my $xtea = Crypt::XTEA->new( $key, 32, little_endian => 1 );
my $ecb = Crypt::ECB->new( -cipher => $xtea );

my $text = 'This is a secret message.';
my $cipher_text = $ecb->encrypt( $text );
my $cipher_hex = unpack("H*", $cipher_text);

print("Ciphertext with little endian: $cipher_hex\n");

# Big endian (default)
$xtea = Crypt::XTEA->new( $key );
$ecb = Crypt::ECB->new( -cipher => $xtea );

$text = 'This is a secret message.';
$cipher_text = $ecb->encrypt( $text );
$cipher_hex = unpack("H*", $cipher_text);

print("Ciphertext with big endian: $cipher_hex\n");

Decryption results with xtea command:

$ perl xtea.pl
Ciphertext with little endian: a3bab9180f2ff58eb6d5bfe356fc2b436c9ae1338fee4cb70b626fd037e4a797
Ciphertext with big endian: eb7a1e2fe4de2699978ddfcec5bdd3649d654c0a7bc1c5c8b343cd8763afc88
$ echo -n a3bab9180f2ff58eb6d5bfe356fc2b436c9ae1338fee4cb70b626fd037e4a797 | hex | xtea -m ecb -L 0123456789abcdef | xxd
00000000: 5468 6973 2069 7320 6120 7365 6372 6574  This is a secret
00000010: 206d 6573 7361 6765 2e                    message.
$ echo -n eb7a1e2fe4de2699978ddfcec5bdd3649d654c0a7bc1c5c8b343cd8763afc883 | hex | xtea -m ecb -L 0123456789abcdef | xxd
00000000: 03dd 14c3 a150 5bf3 ffe0 a500 cc04 8402  .....P[.........
00000010: d7af 3dfa 64da 902b 3c87 1a19 72e2 22ab  ..=.d..+<...r.".
$

Additional Compression Library

Can lzf be added to the decompression functionality as well?

Basically this project: https://github.com/teepark/python-lzf

NSIS Extraction EOF Error

Description

The xtnsis fails to properly parse some NSIS files such as this one:
https://mega.nz/file/QrsG0STY#4uAml4wfCe8-aAUHIXGuHDMK9Po7JmZgrTyQpYwCbLw
When attempting to parse, the user is returned an error stating "failure in xtnsis: exception of type EOF; Unexpected end of buffer."

To Reproduce

Attempting to use the xtnsis to list or extract from this NSIS installer will produce the error.

emit malware.exe | xtnsis -l
(07:05:29) failure in xtnsis: exception of type EOF; Unexpected end of buffer.

or

emit malware.exe | xtnsis [| dump archive/{path} ]
(07:09:29) failure in xtnsis: exception of type EOFError

Environment

• Operating System: Ubuntu 22.04.4 LTS
• Python Version: Python 3.10.12
• Refinery Version: binary-refinery-0.6.39

Additional Context

I use an adapted copy of xtnsis in my debloat tool. When debugging debloat, the error occurs in the "read_exactly" method an returns the following, which I THINK suggests the error is occurring while parsing the NSIS Script, so perhaps there is another missing instruction or something?:

Exception has occurred: EOF
End of File
  File "/home/Documents/GitHub/debloat/src/debloat/utilities/readers.py", line 329, in read_exactly
    raise EOF(data)
  File "/home/Documents/GitHub/debloat/src/debloat/utilities/readers.py", line 376, in read_integer
    result = int.from_bytes(self.read_exactly(bytecount, peek), self.byteorder_name)
  File "/home/Documents/GitHub/debloat/src/debloat/utilities/readers.py", line 468, in u32
    return self.read_integer(32, peek)
  File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 413, in <listcomp>
    self.arguments = [reader.u32() for _ in range(6)]
  File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 413, in __init__
    self.arguments = [reader.u32() for _ in range(6)]
  File "/home/Documents/GitHub/debloat/src/debloat/utilities/readers.py", line 575, in wrapped__init__
    original__init__(self, reader, *args, **kwargs)
  File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 988, in <listcomp>
    self.instructions: List[NSScriptInstruction] = [NSScriptInstruction(reader) for _ in range(self.block_header_entries.size)]
  File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 988, in __init__
    self.instructions: List[NSScriptInstruction] = [NSScriptInstruction(reader) for _ in range(self.block_header_entries.size)]
  File "/home/Documents/GitHub/debloat/src/debloat/utilities/readers.py", line 575, in wrapped__init__
    original__init__(self, reader, *args, **kwargs)
  File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 1148, in __init__
    self.header = NSHeader(header_data, size=header_size)
  File "/home/Documents/GitHub/debloat/src/debloat/utilities/readers.py", line 575, in wrapped__init__
    original__init__(self, reader, *args, **kwargs)
  File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 1309, in unpack
    raise _error
  File "/home/Documents/GitHub/debloat/src/debloat/utilities/nsisParser.py", line 1309, in unpack
    raise _error
  File "/home/Documents/GitHub/debloat/src/debloat/processor.py", line 114, in check_and_extract_NSIS
    extracted_files = extractor.unpack(memoryview(pe.__data__))
    ```

ef command cannot open relative file path (version 0.6.36)

Description

ef command of version 0.6.36 cannot open relative file path. Version 0.6.35 does not have this issue.

To Reproduce

$ cd /tmp
$ echo sample > sample.txt
$ ef sample.txt
(00:21:05) failure in ef: exception of type ValueError; 'sample.txt' is not in the subpath of '/tmp' OR one path is relative and the other is absolute.
$ ef ./sample.txt
(00:22:10) failure in ef: exception of type ValueError; 'sample.txt' is not in the subpath of '/tmp' OR one path is relative and the other is absolute.
$ ef ../tmp/sample.txt
(00:24:53) failure in ef: exception of type ValueError; '../tmp/sample.txt' is not in the subpath of '/tmp' OR one path is relative and the other is absolute.
$ ef /tmp/sample.txt
sample
$

Environment

• Operating System: Ubuntu 22.04.4 LTS (WSL2)
• Python Version: 3.10.12
• Refinery Version 0.6.36

• Operating System: Ubuntu 24.04 LTS (WSL2)
• Python Version: Python 3.12.3
• Refinery Version 0.6.36

Additional bug in ef for absolute paths

The fix in #46 doesn't address the use case where I'm encountering this same issue:

C:\Users\Donald\repositories\refinery>ef C:\Users\Donald\Desktop\install.log
(10:15:25) failure in ef: exception of type ValueError; 
'\\\\?\\C:\\Users\\Donald\\Desktop\\install.log' is not in the subpath of '\\\\?\\C:\\Users\\Donald\\repositories\\refinery' OR one path is relative and the other is absolute.

In this case, I'm trying to ef a file that's not in a subfolder relative to my CWD.

Windows 11
Python 3.11.9
Up-to-date including bugfix

Originally posted by @Donaldduck8 in #46 (comment)

Feature Request: Add Support for Converting from Base92

Consider adding support for converting from base92.

This addition would broaden its functionality and cater to users who may encounter data encoded in this format.

Regards, Moshe

a better solution for linting error test is required

In 04f31d9, a work-around was issued to prevent linter errors to be reported for annotations that are compliant with PEP-3107, but not with PEP-484. I am hoping that with PEP-563, there will be a better way to allow the argparse annotations to pass a syntax check.

The PowerShell Problem

In fcc36d8, I added some extremely weak PowerShell support, this was improved slightly in 504f015. The main issues are the following:

1. PowerShell/PowerShell#1908
2. PowerShell/PowerShell#559

Our current workaround is to:

1. wrap all output as hex with a magic prefix when running under PowerShell
2. unwrap all input when we are running under PowerShell and the magic prefix is detected
3. simply accept the fact that PowerShell has no streaming stdin/stdout

If either of the above two PowerShell issues are ever resolved in a way that supports the binary refinery design, the workaround should be removed because it introduces an unnecessary encoding step.