werwolv/patternlanguage

The Pattern Language used by the ImHex Hex Editor

Pattern Language

This repository contains the source code for the Pattern Language used by the ImHex Hex Editor.

Documentation

Examples

fn main() {
    std::print("Hello World");
}

enum Type : u16 {
    A = 0x50,
    B,
    C
};

struct MyStruct {
    Type type;
    u32 x, y, z;
    padding[10];
    double a;
};

MyStruct myStruct @ 0x100;

Standard Library

The Pattern Language comes with its own standard library which can be found in the ImHex Pattern database

patternlanguage's Issues

std::hash::crc32 documentation is out of date

The std::hash::crc32 documentation lists the function as having 4 parameters, but in c948558 it was changed to 3 (address and size were combined into pattern).

Null-Pointer found in AST with recursive using declaration

The following snippet will cause Invalid parser result: Null-Pointer found in AST.

using P<auto T> = P<T>;

The following snippet will cause ImHex to crash with SIGSEGV.

using P<T> = P<T>;

[BUG] Broken conversion from u128 to s128

There seems to be a problem casting an u128 to an s128 in the pattern language.

Minimal example:

#include <std/io.pat>

u128 u = -1;
s128 s = s128(u);
std::print("unsigned:           {:#b}", u);         // all 1s
std::print("unsigned -> signed: {:#b}", s);         // somehow 0b11 on the border of 64-64 bits (and also truncated to s64)
std::print("signed -> unsigned: {:#b}", u128(s));   // emphasise the broken border 

for and while loops increasing the number of times you have to call parent

fn Test()
{
    u128 var = parent.data;
    while(true)
    {
        var = parent.parent.data;
        for (u8 i = 0, i < 10, i = i + 1)
        {
            var = parent.parent.parent.parent.data;
            
            for (u8 j = 0, j < 10, j = j + 1)
            {
                var = parent.parent.parent.parent.parent.parent.data;
            }
            
        }
        break;
    }
};

struct TestStruct
{
    u128 data;
    Test();
};

As you can see here, to access a variable from the parent struct in a function you have to call +1 parent per while loop and +2 parent per for loop, idk if this is intended or not. I would have expected that you only have to call parent once no matter where you are in the function.

Conversion of big endian integer to float fails

The following pattern will fail the assert.
The pattern works when endian is little.

#pragma endian big

#include <std/sys.pat>

u32 integervar = 7200;
float floatvar = integervar;
std::assert(floatvar == 7200, "floatvar is wrong" );

String + struct = segfault

The pattern language (both ImHex and the browser playground) will segfault upon running the following code:

struct A {};
A a;
str b = "" + a;

I debugged this a little bit, and it looks like the issue is in Pattern::getValue(). The clone() call creates a unique_ptr, which is freed when the function is done, but its pointer is left in the returned value, which is no longer valid.

[Feature] Default template arguments

What feature would you like to see?

Overrideable default values for template arguments like in C++.

struct Array<T = u32, auto Size = 10> {
  T data[Size];
};

How will this feature be useful to you and others?

I have a large number of "optional" structs in my patterns

struct Optional<T, U> {
    U isSome;
    if (isSome != 0) {
        T data;
    }
};

99% of the time U is a u8, but sometimes it is a u32. It would be nice to only have to specify U in the rare case rather than every time.

struct Optional<T, U = u8> {
    U isSome;
    if (isSome != 0) {
        T data;
    }
};

Optional<SomeOtherStruct> common_optional @ 0x00;
Optional<SomeOtherStruct, u32> uncommon_optional @ 0x10;

Request Type

• I can provide a PoC for this feature or am willing to work on it myself and submit a PR

Additional context?

No response

[Bug] Including <std/mem.pat> makes pattern not work.

Operating System

Linux

What's the issue you encountered?

(hi, it's the guy with the weird system again!)

When I add #include <std/mem.pat> to the beginning of my pattern, it doesn't show any result in the Pattern Data section, however, if I remove it, it shows the result perfectly.

How can the issue be reproduced?

• Add this code to the pattern editor (example):

#include <std/mem.pat>

bitfield bit { state: 1; };

struct Block {
	char magic[4];
	bit someBits[16];
	char data[4];
};

Block block @ 0x00;

• Try to run it, no result
• Remove/comment out #include <std/mem.pat>
• Result appears in Pattern Data section

ImHex Version

01adc8a

ImHex Build Type

• Nightly or built from sources

Additional context?

• uname -a: Linux fedora 5.18.6-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jun 22 13:46:18 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• I'm pretty sure it would work on any file but here is the file I was editing when it happened: miidata02
• I'm using the AppImage version of ImHex which is known to have some issues.
• This issue is copied from the ImHex repo.

format/format_entries does not work with a template with non-type values (and an @)

Originally reported here: WerWolv/ImHex-Patterns#147 - but closed due to wrong repository.

Consider the following file:

10 00 00 00 18 00 00 00 04 00 00 00 FF FF FF FF FF FF FF FF FF FF FF FF 74 65 73 74 31 32 33 34 35 6C 6F 6C 31 33 33 37 08 00 00 00 04 00 00 00 0C 00 00 00 05 00 00 00 11 00 00 00 03 00 00 00 14 00 00 00 04 00 00 00

test.zip

With the following pattern:

fn entryName(auto e) {    
    return e.name;
};

struct Entry<auto offset> {
    u32 nameOffset;
    u32 nameSize;
    char name[nameSize] @ offset + nameOffset;
}/*[[format("entryName")]]*/;

struct Header {
    u32 blockOffset;
    u32 nameBlockSize;
    u32 numberOfElements;
    
    Entry<blockOffset> entries[numberOfElements] 
        @ blockOffset + nameBlockSize
        //[[format_entries("entryName")]]
        ;
};

Header header @ 0x0;

Basically, this file requires the entry to know where the lookup table for the strings is located. I've used a template for that. The original issue was about not being to use a template in a format-function, but auto solves that. However, this particular file and pattern causes an error[E0011] when you try to use either of the format attributes.

I'm pretty sure it's related to the @ sign throwing things off.

Can't print big endian time, but works in little endian

expected result

in big endian mode std::time::format should print the datetime

current result

When printing a timestamp in big endian mode the printed value is "Invalid".

See the following pattern:

#pragma endian big

#include <std/string.pat>
#include <std/time.pat>

u32 datetime = 0xD6;

std::print("{}", std::time::format(std::time::to_utc(datetime)));

note Change big to little and it works.

[Bug] Unexpected expression closing nested template type instantiation

The pattern

struct A<T> {
    T a;
};

struct B<T> {
    T b;
};

A<B<u8>> ab;

yeilds

error[P0002]: Unexpected expression
  --> <Source Code>:9:5
9 | A<B<u8>> ab;
        ^
        Expected '>' to close template list, got Operator (>>).

putting a space between the greater-than signs fixes this but looks weird. This is a really minor bug. Even if there was just an error message explaining to add a space that would be an improvement.

Tested using PatternLanguage 57e89f6 (compiled from source, MSYS2) and ImHex 1.27.1 (Windows MSI Installer).

[Bug] Strange behavior on local variables in [[static]] structs

Consider the input

00 00

As expected, the pattern

#include <std/sys.pat>

struct A {
    u8 a = 1;
    u8 b;
};

A a[2] @ 0x0;
std::assert(a[0].a == 1, "a[0].a != 1");
std::assert(a[1].a == 1, "a[1].a != 1");

evaluates fine and the assertions pass. However, if we make the struct A [[static]].

#include <std/sys.pat>

struct A {
    u8 a = 1;
    u8 b; // If removed then the assertions pass
} [[static]];

A a[2] @ 0x0;
std::assert(a[0].a == 1, "a[0].a != 1");
std::assert(a[1].a == 1, "a[1].a != 1");

Then the first assertion passes but the second one fails. In fact, only the first member of an array of [[static]] structs will have the correct local variable values. All other members in the array will have 0 values for the local variables in my experience. If this is an intended optimization then perhaps the documentation could be updated to reflect this, since right now it only mentions layout in regards to [[static]].

Tested using PatternLanguage 57e89f6 (compiled from source, MSYS2) and ImHex 1.27.1 (Windows MSI Installer).

Crash when calculating addressof(self)

I have a recursive data structure which contains (among other things) a field for the size of the embedded array; reading that array consists of reading elements from the start address of the array until the index pointer matches the data size. Here’s a minimized example of what seemed like the natural way to achieve this:

struct Element {
	u8 child_len;
	Element children[while($ != addressof(children) + child_len)];
};
Element root @0;

However, with the example data 05 01 00 02 00 00, this reliably crashes in ImHex 0.33.1, and also seems to crash (at least it produces no data) on the current version of pl.werwolv.net.

Workaround: use addressof+sizeof the previous field, but this looks clunky and is hard to keep track of when there are several fields in play:

struct Element {
	u8 child_len;
	Element children[while($ != addressof(child_len) + sizeof(child_len) + child_len)];
};

[Suggestion] Attribute to make local variables within structs

In functions (and the global scope), u32 var = 123; is equivalent to u32 var; var = 123;. In pattern structs, u32 var = 123; creates a local variable (not part of the pattern data) that can be freely modified, whereas u32 var; var = 123; is simply forbidden (as u32 var; is parsed as part of the pattern data, and therefore must not be modified) - there is a functional difference between initialised and uninitialised variables/patterns.

However, arrays can't be initialised like that; neither u32 var[3] = {10,20,30};, u32 var[3] = [10,20,30];, u32 var[3] = 10,20,30;, nor u32 var[3] = 10; are allowed. This means that it's impossible to declare local array variables (not part of the pattern data) within pattern structs. I would've suggested "let us initialise arrays", but the same applies to structs (e.g. struct MyStruct { u32 value; }; MyStruct var = {0}; doesn't work), and I can't think of a good way to initialise structs while declaring them.
Currently, to get around that, you have to first make a section, then place an array or struct in that section, and then fill out the array/struct values. Doing this can be inconvenient, fills the Sections tab with many different sections if you e.g. create a section like this in a struct and use that struct 20 times, and makes the [[export]] attribute not display it in the Pattern Data tab (as you instead have to open and look within the respective section's own patterns).

Therefore, I suggest an attribute or a keyword to make a local variable even when it's not initialised. For example, u32 var[3] [[local]]; would create a local array with 3 zeroes, and struct MyStruct { u32 value; }; MyStruct var [[local]]; would create a local struct with value set to zero.
(For structs, care should be taken to ignore @ for its members, except when it's placed in a section created within that struct or a sub-struct (or just except when it's in any section at all?). Structs that do something like struct MyStruct { u32 value1; if (value1 > 10) u32 value2; } won't work as expected as value1 would always be zero at that time, but it's up to pattern makers to not use local structs that require being placed in memory.)

TL;DR: An attribute/keyword to force a variable to be local would be nice, even if it's not initialised (which is currently impossible for arrays/structs). All values that would've been read from (non-section) memory should probably default to 0.

Bitfield order defaults to MostToLeastSignificant in big endian

The default bits order for a bitfield (without a bitfield_order attribute) is LeastToMostSignificant, but when switching to big endian with #pragma endian big, that default seems to change to MostToLeastSignificant.

Using this bitfield as an example:

bitfield MyBitfield {
  x : 1;
};

For a file containing a single byte set to 0x01, without any #pragma directive, x has a value of 1. With #pragma endian big, x has a value of 0 (and when I change the content of the file to 0x80, then x becomes 1).

Because endianness is about the order of bytes and not about the order of bits, I think keeping the same default for big endian and for little endian would make more sense. LeastToMostSignificant already handles endianness properly (or at least, like I would expect it to). For example, with a file containing 00 00 00 01, and with the following pattern:

#pragma endian big
#include <std/core.pat>

bitfield MyBitfield {
  x : 1;
} [[bitfield_order(std::core::BitfieldOrder::LeastToMostSignificant, 32)]];

, x properly shows the value 1.

s64 wrong value evaluation

A value of 0x FFFF FFFF FFFF FFFF
parsed with
s64 test @0x0;
is detected as value 0x FFFF FFFE FFFF FFFF.
Probably a sign mistake?

ImHex Hex Editor v1.25.0

Edit: I hope this is the correct repository

Edit2: Great work, btw! ImHex with its pattern language makes it so much easier to reverse engineer file formats.

`std::hash::crc32` cannot calculate CRC32 of global variables

Apologies if I missed something obvious but it seems like there is no way to calculate CRC32 with std::hash::crc32 if the data is saved in a global variable, since global variables do not have memory addresses. Having this would be required for recalculating and verifying typical binary headers with CRC32 embedded into it, because I would have to make a copy of the header into a global variable first, change the existing checksum field to all zeroes, and then recalculate the checksum.

[Bug] Local variable assignment crashing

The pattern

struct A {
    u32 a = 0; // If a is a regular member then no crash
};

A a;
a.a = 0; // This line causes the crash

causes a crash if run immediately after opening ImHex or in plcli.

Alternatively,

#include <std/sys.pat>

struct A {
    u32 a = 0;
};

A a;
std::assert(a.a == 0, "a.a != 0");

yeilds

error[E0011]: Memory error.
Tried accessing out of bounds heap cell 1. This is a bug.

which appears to be related.

Tested using PatternLanguage 57e89f6 (compiled from source, MSYS2) and ImHex 1.27.1 (Windows MSI Installer).

Using template with defined enum data type causes pattern to lose field

bug_case.zip
I've attached a zip file containing a test file and pattern for reproduction.

Demo screenshots:

The test_data field gets dropped from the parsed data if it is used via a template variable, if the template variable is removed and replaced by parent.parent.head.test_data, it works as expected, also, if the test_enum type is replaced with u8 it also works as expected.

Overrun of data causes unassigned variable to be provided to functions

Could also be broken with all similar functions but logging what I observed:

struct Option<T> {
    bool is_some;
    T value;
} [[format("format_option")]];

fn format_option(Option<auto> option) {
    std::print("{}", option.is_some);
    if (option.is_some) {
        return std::format("Some({})", option.value);
    } else {
        return "None";
    }
};

I can see in pattern data that is_some is true but inside format_option it is false (in the console). Value is also not populated.

Enum with negative number (with signed integer) not work as expected

When I tried the following pattern language on web:

enum temp : s8 {
    unknown = -1,
    fe = 0xfe,
    one = 1,
    two = 2,
    zero = 0,
};

temp foo[5] @ 0x0;
s8 raw[5] @ 0x0;

with the following hex:

ff fe 00 01 02 03

I expected the first "ff" recognized as unknown, but recognized as ???.

actual

{
    "foo": [
        "temp::???",
        "temp::fe",
        "temp::zero",
        "temp::one",
        "temp::two"
    ],
    "raw": [
        -1,
        -2,
        0,
        1,
        2
    ]
}

expected

{
    "foo": [
        "temp::unknown",
        "temp::fe",  # or "temp:???"
        "temp::zero",
        "temp::one",
        "temp::two"
    ],
    "raw": [
        -1,
        -2,
        0,
        1,
        2
    ]
}

Defines and attributes don't go together.

Consider the following minimal example:

#define BASE_OFFSET = 48;

u8 componentBody[100] @ BASE_OFFSET [[color("0000FF")]];

BASE_OFFSET will throw an error like "Invalid function statement." - Also happens when done in structs.

If you remove the color-attribute, it works again.

u32 BASE_OFFSET = 48;

u8 componentBody[100] @ BASE_OFFSET [[color("0000FF")]];

Works just fine.

Not sure if intended.

[Improvement] Displaying string from upper structure when hidden

Well, this issue seems to belong to the ImHex repository, but I trace the actual implementation to pl::Pattern::getChildren here.

Motivation / Expected behaviour

In many situations, people want to display formatted info about structs instead of their inner structure (or maybe they want both):

struct MAC {
	u8 octets[6] [[hidden]];
} [[format("mac_formatter")]];

fn mac_formatter(MAC mac) {
	return std::format("{0:02X}:{1:02X}:{2:02X}:{3:02X}:{4:02X}:{5:02X}",
		mac.octets[0],
		mac.octets[1],
		mac.octets[2],
		mac.octets[3],
		mac.octets[4],
		mac.octets[5]);
};

MAC mac @ 0;

To avoid too much duplicate information, we can default structs to [[hidden]] and let the user explicitly set [[hidden(false)]].

Current behaviour

Struct info is never displayed in the hex editor. It seems patterns default to include themselves as their children. However, structs and some other complex structures do not:

PatternLanguage/lib/include/pl/patterns/pattern.hpp

Lines 129 to 131 in 1b0de5e

	[[nodiscard]] virtual std::vector<std::pair<u64, Pattern*>> getChildren() {
	return { { this->getOffset(), this } };
	}

PatternLanguage/lib/include/pl/patterns/pattern_struct.hpp

Lines 39 to 48 in 1b0de5e

	[[nodiscard]] std::vector<std::pair<u64, Pattern*>> getChildren() override {
	std::vector<std::pair<u64, Pattern*>> result;
	for (const auto &member : this->m_members) {
	auto children = member->getChildren();
	std::copy(children.begin(), children.end(), std::back_inserter(result));
	}
	return result;
	}

So, structs will never appear at all in the final flattened patterns or get displayed in the hex editor:

PatternLanguage/lib/source/pl/pattern_language.cpp

Lines 239 to 247 in 1b0de5e

	void PatternLanguage::flattenPatterns() {
	for (const auto &pattern : this->m_patterns) {
	auto children = pattern->getChildren();
	for (const auto &[address, child]: children) {
	this->m_flattenedPatterns[address].push_back(child);
	}
	}
	}

P.S.

After adding struct itself to its children, things seem to be working somehow, despite that the displayed size of the struct appears to have a maximum of 16 bytes, which might belong to another issue, I suppose?

struct A {
  u8 u[0x30];
};

Inconsistent requirement for scope resolution qualification of functions vs custom types

A user-defined type declared inside a namespace can be used inside the namespace without full qualification (aka explicit full scope resolution in name) but functions seem to require the use of their full names regardless of location of usage.

Consider the following declarations:

namespace ns {
    struct S {
        u8 c = 3;
    };

    fn fun(auto arg) {
        return arg+1;
    };
}

Using them the following code runs correctly:

import std.io as io;
namespace ns {

    S s;
    io::print("{}",ns::fun(s.c));
}

but dropping the full name from fun causes an error indicating that the function could not be found as shown in the next image. Either both S and fun should need their full name or neither should.

Not possible to run via the cli interface

A valid command to the imhex pattern language cli results in the following message:

$ imhex --pl run -i ../examples/diadisc.mov -p small.hexpat -v
The following argument was not expected: imhex
Run with --help for more information.

When an invalid command is run, e.g. missing input file there is an actual error message:

$ imhex --pl run -p small.hexpat
--input is required
Run with --help for more information.

Implement the ability to conditionally assign attributes

It would be great if there's a way to conditionally assign attributes to a variable. This would be pretty nice for verifying checksums included in some file formats, where the pattern language can verify the checksum for the data chunks if the algorithm is known, and dynamically rename the variable to something like CRC(verified) and CRC(unverified) depending on the result. Currently there does not seem to be a way to implement this if the variable is a part of a struct.

lua54.hexpat segfaults in `Parser::parseStatements()` after fixing function parameter named `const`

While trying to fix up the lua54.hexpat file for bitfields, I ran into this issue loading VLC/lua/extensions/VLSub.luac from my VLC installation.

On this line, statement is null.

Struct member access doesn't work on the left-hand side of +=, nor the left-hand side within patterns

Using the version of Pattern Language in ImHex v1.33.2, if abc is a struct, then abc.var is a little picky on the left-hand side of assignments in the global scope, and very picky on the left-hand side of assignments within structs (regardless of whether abc is a global struct or a member of the struct currently being parsed).
(This might explain half of WerWolv/ImHex#1599 ?)

Note the //This does not work comments in...

import std.io;

struct GlobalStruct {
    u8 var = 10;
};
GlobalStruct abc;

abc.var = 15; //This works
//abc.var += 5; //This does not work
abc.var = abc.var + 5; //This works
std::print("{}",abc.var+2); //This works
u8 global_var1 = abc.var + 5 [[export]]; //This works
u8 global_var2 = 0 [[comment(abc.var+3),export]]; //This works

struct MyStruct {
    //abc.var = 50; //This does not work
    //abc.var += 5; //This does not work
    //abc.var = abc.var + 5; //This does not work
    std::print("{}",abc.var+2); //This works
    u8 struct_var1 = abc.var + 5 [[export]]; //This works
    u8 struct_var2 [[comment(abc.var+3),export]]; //This works

    struct_var1 = 70; //This works
    //this.struct_var1 = 75; //This does not work
    u8 struct_var3 = this.struct_var1 + 4 [[export]]; //This works, meaning that "this.struct_var1" above isn't incorrect
};

MyStruct data @ $;

[Improvement] Half-Float (16 bit float) data type

Some file formats, often related to game engines and applications related to computer graphics, use 16 bit floating point types (see uses section).

It is a standardized format and IMO support for decoding this type in the core language would make sense.

Issue with Non-Type Template Parameters

struct SerializedArray<T, auto count>
{
    u32 version;
    T values[count] [[inline]];
};

struct SerializedClass<T>
{
    u32 version;
    T class [[inline]];
};


struct Test
{
    u32 var;
};

struct TestList
{
    u32 count;
    SerializedArray<Test, count> items;
};

struct TestMatrix
{
    u32 count;
    SerializedArray<TestList, count> items;
};

SerializedClass<TestMatrix> testMatrix @0;

When u have this nested usage of the template struct SerializedArray and have the Non-Type Template Parameter the same name as the variable u pass into it both times it will take the first one. So in this case the length of items in TestList is count of the TestMatrix. Only when renaming the non-type template param or the variable u pass into it changes, then it will have the correct count. Here is a file that you can test the pattern on and play with the names of the different counts to see how it gets fixed
test.zip

[Bug] Cannot `match` with ranged enum values.

Attempting to use a match operator with a enum with a range value compiles, but seems to not include the resolving variable in the pattern data result.

The value (7) is in the range, but when it's matched to, it fails to return any result. Specifying the value in the enum directly (like the None type is with a value of 3) will compile, so will inverting the statement 7 ... 0 (despite it being invalid in the Value output), but I assuming that's only because it's checking the first value instead of the whole range.

#include <std/io.pat>

bitfield BGR5A1 {
b: 5;
g: 5;
r: 5;
a: 1;
} [[static, sealed, color(std::format("{0:02X}{1:02X}{2:02X}FF", r * 8, g * 8, b * 8))]];


enum TextureFormat : u32 {
	Uncompressed = 0 ... 7,
	None = 3,
	Compressed = 8 ... 15
};

struct Texture {
	TextureFormat format @ 8 [[no_unique_address]];
    match (format) {
        (TextureFormat::Compressed): BGR5A1 texture[0]; //unimplemented
        (TextureFormat::Uncompressed): BGR5A1 texture[4000];
    }
};

Texture texture @ 0;

Interpreter: ImHex (Win64, Portable) 1.30.1

I've been trying to write a pattern to parse the PS2 icon format. I've gotten to the part where I've got the shapes, just no texture to apply onto the shape.

Example of an uncompressed icon is the Super Mario 64 PS2 homebrew port, which uses a textured cube for it's display model.