gennyble/hext

A Binary File Markup Language

hext

Do you find yourself exploring file formats, such as GIF, and finding yourself annoyed with hex editors? Do you find yourself wanting to craft 802.11 packets, but you know you'll never be able to keep all the fields straight in your head? Write those packets in hext! hext (*.hxt) is a file format for writing small (or large, if you dare) binary files with comments. You can switch between hex, binary, and decimal and even stick some string literals in there.

The CLI tool is called hxt and it can be found here. To install it, run

cargo install hxt

This is what a small GIF looks like:

~little-endian msb0
"GIF89a"    # File header and version string

# Logical Screen Descriptor
u16=4 u16=4 # Canvas width/height
.1          # Global Color Table Flag
.000        # Color Resolution
.0          # Reserved in 87a
.000        # Number of colors. 2^(this_value + 1) colors
00          # Background Color Index
00          # Pixel Aspect Ratio

# Global Color Table
=0   =0   =0   # Black
=255 =255 =255 # White 

# Image Descriptor
","         # Image Separator
u16=0 u16=0 # Image offset left/top
u16=4 u16=4 # Image width/height
.1          # Local Color Table Flag
.0          # Interlaced Flag
.000        # Reserved in 87a
.000        # Size of local color table

# Local Color Table
=0   =0   =0    # Black
=128 =0   =255  # Purple

# Image Data
02               # LZW Minimum Code Size
05 841D817A50 00 # Data sub-block

# Graphic Control Extension
"!" F9   # Extension Introducer and Graphic Control Label
=4       # Extension data block size
	.000 # Reserved
	.010 # Disposal Method (010 is restore to background color)
	.0   # User Input Flag
	.0   # Transparent color flag

	u16=50 # Delay (1/100ths of a second)
	00     # Transparent Color Index
00 # Block terminator

# 2nd Image Descriptor
"," u16=0 u16=0 u16=4 u16=4 .00000000

# 2nd Image's Data (same as the first)
02 05 841D817A50 00

";" # GIF Terminator

I really do mean small. Here it is:

hext's Issues

Listen to negative kind

Currently, we allow specifying a kind of negative in the file header. Currently it can be one's compliment, two's compliment, or sign-magnitude. We do not, however, do anything with this information. All signed values are output as two's compliment.

Hxt: CLI option for inserting some hext into a file at an offset

Or maybe a subcommand. Akin to hxt insert_into <filename> at <offset> from <hxt_name>.

Or maybe just a regular subcommand kind of thing: hxt insert_into <file> <offset> <hxt>

Allow both?

1.0.0 Tracking Issue

• Pay attention to the kind of negative (#6)
• Documentation on the format (#8)
• Documentation on the library
• Auto checksums and length filling would make hext inredibly more helpful (#2)
• #7

Hext Language Documentation

A nice way to say:

The first line that isn't a comment of blank, must be the file header. The header is structured as such:

~<property> <property> ...

You must specify your endianess and bit order. You can set those with little-endian, big-endian, msb0, lsb0. You can specify the endianess before the bit-order, or the other way round.

Hex is the default and must appear in octets. FF is valid, but FFA is not. You can group this all as one, FFOA, or spread it out, FF OA.

Binary is prefixed by a period, like so (ascii A), ..01000001. It can also be spread out .0100 .0010. By default you must have your binary be octets. You cannot, for example, have .0100 F, that's not allowed. You can put padbits in the header to pad out any octets you do not fill. It will pad according to your bitness property.

Decimal is prefixed with an =. You can precede the equals with an integer type and size, i for signed and u for unsigned. Integers can be 8, 16, 32, or 64 bits wide. If you do not specify a width you may not use i or u to specify the type, signed or unsigned. In that case, you'd just prefix all your signed integers with their sign (even the positives). The smallest integer for the decimal value you specified will be chosen. For example, in little-endian, =255 would output FF and =513 would be 01 02. If you write u16=255 you would get FF 00

Signed numbers are currently stored as two's compliment only. In the future you'll be able to specify the kind of negative you want in the header. Available options are:
ones-compliment, twos-compliment, sign-magnitude

Checmsum and Length Autofilling

A lot of Network protocols require a checksum, but who wants to do that manually. There should/could be an option to specify which byte (or maybe line? Some kind of marker) to insert/replace a checksum of a specific kind.

implicitly sized decimal

Currently you may do =63 and or =601 and it'll use the smallest integer that it can that's still in octets. So 63'd be a u8 and 601 a u16. In talking about hext with someone during an interview, they raised a good point: that is not what you may expect for a binary format. Should we make it so using decimal without a size is a syntax error? Set a default size, but explicitly in the header? Should it error if it overflows?

issue with `cargo install`

Rust Toolchain 1.65

~ ❯ cargo install hxt
    Updating crates.io index
  Installing hxt v0.1.1
error: failed to compile `hxt v0.1.1`, intermediate artifacts can be found at `/tmp/cargo-installmjgMXZ`

Caused by:
  failed to select a version for the requirement `funty = "~1.2"`
  candidate versions found which didn't match: 2.0.0, 1.1.0, 1.0.1, ...
  location searched: crates.io index
  required by package `bitvec v0.22.3`
      ... which satisfies dependency `bitvec = "^0.22.3"` of package `hext v0.4.0`
      ... which satisfies dependency `hext = "^0.4"` of package `hxt v0.1.1`

String style switching

Recentlyish I've wanted to write a string and have it length-prefixed, but cstrings still exist, yeah?
~string_style length-prefixed
How I believe Rust holds strings. Some int size (configurable?) and then the bytes of the string. Although Rust uses char I think, which is distinctly not a byte here so maybe a bad example.

~string_style null-terminated
Classic cstring style. The entire string and then one 0x00 to end it

~string_style raw (no-style?)
Just the bytes here. This is how it currently works.

Hext TextMate Grammar File

While not a very complicated format, it would still be nice to have some color in hext files. Here is some information on language servers: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide

Compilation Hints

When going form hext to binary, there are some things that are too important to leave as command line flags. Stuff like the bitness (#3) and maybe even, if it's implemented, the checksum location (#2).

Here's how I think it should be done here. Not many hints, and they should be short. Here are all the ones I can think might be needed:

~padding msb
~padding lsb
~padding off
~checksum from 0x00 to 0x10 at 0x20
~checksum from 0x00 to 0x10 replace ZZ

Where ~padding off is the default value.

Allow changing bitness

The ability to change from Msb to Lsb.

Support all possible integer sizes

u24? Heck yeah. u40? No, me 40.

It would be neat to provide support for all possible sizes of integer that make sense. By "make sense" I of course mean factors of 8, but I can be convinced that u13 or u7 needs support. Unsure how that would work though.

Make Hext a Reader/Writer

Binary Fields

Binary fields are implemented, but I'd like a way to throw an error if a line ends and the octet is incomplete, or better yet, if hex starts again and there is not a complete octet.