python-intelhex/intelhex

Python IntelHex library

Python IntelHex library

Introduction

The Intel HEX file format is widely used in microprocessors and microcontrollers area (embedded systems etc) as the de facto standard for representation of code to be programmed into microelectronic devices.

This work implements an intelhex Python library to read, write, create from scratch and manipulate data from Intel HEX file format.

The distribution package also includes several convenience Python scripts, including "classic" hex2bin and bin2hex converters and more, those based on the library itself. Check the docs to know more.

License

The code is distributed under BSD license, see LICENSE.txt.

In short: you can use IntelHex library in your project without any restrictions.

Supported Python versions

IntelHex library supports Python 3 (3.5 or later) only. The 2.2.1 release was the last one which has been checked against Python 2.7 and Python 3 until 3.5.

Install

Install using pip (recommended, no separate download required):

pip install intelhex

Download

• https://pypi.org/project/IntelHex/
• https://github.com/python-intelhex/intelhex/releases

Source code, bug reports, patches

IntelHex on GitHub:

https://github.com/python-intelhex/intelhex

User manual

User manual for IntelHex is available in the sources docs/manual/ directory. You can browse User Manual online:

https://readthedocs.org/projects/python-intelhex/

Changelog

See NEWS.rst

intelhex's Issues

Support for arbitrary word length and word address

When I use this package to generate memory initializing file for alera_mf MEM 1 port with 16 bit word length, the saved hex file is working. I create and use mif file format. It is working. When I convert mif file to hex file using Quartus II (13.1), the different are: 1. byte_count on each line is 2 and 2. the address are word address not byte address. Although I can add byte_count=2 when writing to hex file, the address there is no way to change the addresses.

ih.write_hex_file(f_hex, byte_count=2)

Use intelhex package:

:0200000080007E
:02000200800676
:02000400800C6E

Convert using Quartus II

:0200000080007E
:02000100800677
:02000200800C70

To solve the problem, I think creating a new class support any word length, and option for word address when writing hex file.

intelhex hangs on using `in` operator

In python, using the in operator is used to test for membership, thus it would be natural to expect that the following expression is valid to test if the given address is present in the hex. However it hangs of an unknown reason.

     is_present = 0x1000 in ih

This has a trivial fix by implementing __contains__. I have implemented a fix and a test for it in my fork: https://github.com/sveinse/intelhex. Please give notice if a PR is wanted for this. https://github.com/sveinse/intelhex

master...sveinse:intelhex:master

The binary limitation of bin2hex is 4Kbytes.

Any binary file over 4K will only be transforms the first 4Kbytes. The rest are discard.

does it support extended segment address write?

I saw in loadhex() is supported the record type == 2 (IHEX_EXTENDED_SEGMENT_ADDRESS_RECORD), but in write_hex_file(), there is not supported this record type?

Thanks.

Older PyPI releases link to malware?

Hi, I help support the PyPI server software by answering bug reports. We've recently been told that some of the older releases of this project (1.5 and below, I think) have a "Home Page" URL specified that points to a compromised server that is deploying malware.

Could you please check that out? If true, since it's not possible to edit older releases anymore, then if the server cannot be fixed, the only option may be to remove those older releases.

Thank you.

larger size after merge hex file

I use script hexmerge.py to merge two hex file, but after merging, I find the merged file size is larger than other tool.
For example, I have 3 files, 924k(0x8000000~0x8003AF0), 460k(0x8009000~0x80039210), 512k(0x8080000~0x80FFB30), first I use intelhex to merge, the merged file is 2.0M, than I use another tool named srecord, the merged file is 1.7M, but both of them work well.
I notice, the merged file used by intelhex like this:
:10000000288000205102000859020008ED1800085D
and srecord like this:
:20000000288000205102000859020008ED1800085D0200085F020008610200080000000012
I guess maybe there some format that intelhex used make it difference?

Documentation for hex2bin function is not generated.

The readthedocs site doesn't appear to have any documentation for the hex2bin function (only the script which can be used from outside of Python is documented). Search didn't help (See Related issues).

Related issues:

• #62

Support for comment character

Altough it is not normally supported by the .hex format, many vendors embed comments in their .hex file, which make them impossible to parse by IntelHex unless pre-cleaned programmatically.

It would be nice to be able to specify an optional comment character (;, #, --) that would make the parser ignore any line starting with it.

Or maybe ignore everything not starting with a record delimiter.

pip install mode on scripts

installing w/ pip (7.1.2, setuptools 18.4) on python 2.7.10, the scripts were installed mode 0644 (i.e., not set executable).

python3 gets wrong data type

>>> sys.version_info
sys.version_info(major=3, minor=5, micro=3, releaselevel='final', serial=0)
>>> import intelhex
>>> myhex = intelhex.IntelHex('/home/albert/albert.kurucz/MultiRadio/trunk RF200_AES128_SnapV2.8.2.hex')
>>> myhex.segments()
[(0, 420), (752, 761), (3072, 3360), (65536, 122354), (128256, 128562), (128682, 128690), (128698, 128934)]
>>> data=myhex.gets(0, 420)
>>> type(data)
<class 'str'>

with python3 expected data type returned by gets is <class 'bytes'> not <class 'str'>

intelhex==2.1

hexdiff.py not working on Ubuntu 19.10

hexdiff.py version 2.2.1

adams@ubuntu:/mnt/f$ pip install intelhex
Requirement already satisfied: intelhex in /home/adams/.local/lib/python2.7/site-packages (2.2.1)

adams@ubuntu:/mnt/f$ python hexdiff.py device1.raw device2.raw 
Traceback (most recent call last):
  File "hexdiff.py", line 90, in <module>
    sys.exit(main())
  File "hexdiff.py", line 84, in main
    ih1 = IntelHex(fname1)
  File "/home/adams/.local/lib/python2.7/site-packages/intelhex/__init__.py", line 90, in __init__
    self.loadhex(source)
  File "/home/adams/.local/lib/python2.7/site-packages/intelhex/__init__.py", line 212, in loadhex
    decode(s, line)
  File "/home/adams/.local/lib/python2.7/site-packages/intelhex/__init__.py", line 123, in _decode_record
    raise HexRecordError(line=line)
intelhex.HexRecordError: Hex file contains invalid record at line 1

adams@ubuntu:/mnt/f$ python
python              python2.7           python2-config      python3.7           python3-futurize    python3-pasteurize  python-config       
python2             python2.7-config    python3             python3.7m          python3m            python3-wsdump 

adams@ubuntu:/mnt/f$ python3.7 hexdiff.py device1.raw device2.raw
Traceback (most recent call last):
  File "hexdiff.py", line 90, in <module>
    sys.exit(main())
  File "hexdiff.py", line 83, in main
    from intelhex import IntelHex, diff_dumps
ModuleNotFoundError: No module named 'intelhex'

hex2bin does not work on Linux

From pull request #7 (comment)

Example of hex file:

:01000000bd42
:00000001FF

On windows, both hex2dump and hex2bin work.
On Linux, only hex2dump (the character is not display correctly but I don't
care)

Python versions: On windows python 3.4.3 and on linux 3.4.1.

Swapping of bytes

Hello,

Your library has been very helpful for reading .hex files.

Context: I'm using the IntelHex library to read from a .hex file, which I then use to flash a microcontroller.

As you can appreciate, there are thousands of bytes contained in the .hex, and so I'm attempting to read and send the maximum number of bytes in one message at a time, which is 4095.

My code looks something like this:

startAddress = 0x00840000
maxFrameSize = 4095
data = []

"""  Send 100 blocks of data from hex file """
for blocks in range(100):

    """ Increment sequence number for every data block sent """
    for sequenceNumber in range(0x01, 0xFF):

          """ Read data from .hex and append to array """
           for i in range(maxFrameSize):
              
               data.append(int(IH[startAddress + I]))

           """ Send """
           client.transfer_data(sequence_number= sequenceNumber, data= bytes(data))
           
           """ Clear array """             
           data.clear()
           
           """ Update Start address for next block """
           startAddress += maxFrameSize

The issue I've encountered is that some bytes are swapped according to my trace, preventing me from finishing the data transfer.

The image is a diff containing my trace on the left, and a trace generated from another vendor-supplied microcontroller flashing tool, with the differences highlighted.

I hope I've provided enough information.

Any help would be appreciated!

CS:IP entry point calculation seems to be incorrect

According to: https://en.wikibooks.org/wiki/X86_Assembly/16,_32,_and_64_Bits

... to create a 20 bit address, it would be done by taking the 16-bit value of a segment register and put it on the address bus,
but shifted 4 times to the left (thus effectively multiplying the register by 16), and then by adding the offset from another
register untouched to the value on the bus, thus creating a full a 20-bit address.

If CS = 258C and IP = 001216, then CS:IP will point to a 20 bit address equivalent to "CS × 16 + IP" which will be
258C × 1016 + 001216 = 258C0 + 001216 = 258D2 (Remember: 16 decimal = 1016).

However the implementation is: https://github.com/python-intelhex/intelhex/blob/master/intelhex/scripts/hexinfo.py

        if keys == ['CS','IP']:
            entry = ih.start_addr['CS'] * 65536 + ih.start_addr['IP']

Is it not an implementation mistake?

better pip install support: extras

I'd like to have some extras for pip install, e.g. optional tests, benchmark and scripts.

See https://www.python.org/dev/peps/pep-0508/#id21

Write hex file with chooseable byte count

At the moment the only possible data field byte count is 16 when using write_hex_file.

Intelhex wiki: The maximum byte count is 255 (0xFF). 16 (0x10) and 32 (0x20) are commonly used byte counts.

Should be able to choose variable byte_count ranging from 1 to 255 when writing the hex file.

Search doesn't appear to work on the readthedocs site.

On the readthedocs site, I can click on hex2bin in the scripts category, but searching for the term does nothing. On either Firefox or Chrome-based Edge browsers, I get "Searching..." with blinking dots, but it never goes anywhere.

A [0:0] slice will return the entire contents, when it should return a empty object

Consider the following.

ih = IntelHex({0:0, 1:1, 2:2})
# The bug
print(ih[0:0].tobinarray()) # array('B', [0, 1, 2])
# Correct for other indices
print(ih[1:1].tobinarray()) # array('B', [])

# stdlib __getitem__ for reference
li = [1, 2, 3]
print(li[0:0]) # []

Looking at the source for _getitem_() this behavior happens because the endaddr 0 evaluates the same as None.

The fix should simply be:

# Current
start = addr.start or addresses[0]
stop = addr.stop or (addresses[-1]+1)
# Fixed
start = addr.start if addr.start is not None else addresses[0]
stop = addr.stop if addr.stop is not None else (addresses[-1]+1)

[lp #372625] tobinarray end parameter is inconsistent with standard python range behaviour

https://bugs.launchpad.net/intelhex/+bug/372625
Old bug from LP. Moving it here.

The most important bits:

Standard range function has 3 parameters: start, stop [,step]
here: stop is exclusive

My tobinXXX group of methods first parameters are: start, end
here: end is inclusive

I have 2 choices:

1. remove end and add stop
2. rename end to end_inclusive, and add stop additionaly.

Require API break or semi-break.

End line problem with AVRStudio

If I use the lib to generate hex file on the OS Win everything is ok
If I use the lib to generate hex file on the Unix OS and after this I try to upload this file to chip on Win OS a I will get a error.

Problem in the end of line symbol. Because Unix use only LF, but windows software expect a LFCR.

Array attribute

python 9 is not supporting the tostring() attribute of array

Merge fails on two IntelHex16bit files...

I am reading a PIC16 flash image from the target (oh instance IntelHex16bit), and merging it with a new image (nh instance IntelHex16bit) and get the following error:
nh.merge(oh[base:base+size], overlap='replace')
File "C:\Users\md84189\AppData\Local\Programs\Python\Python39\lib\site-packages\intelhex_init_.py", line 964, in getitem init.py", line 964, in getitem
addr1 = addr16 * 2
TypeError: unsupported operand type(s) for *: 'slice' and 'int'

If both instances are IntelHex or if nh instance is IntelHex and oh remains IntelHex16bit, the operation completes.

Note: Using Python 3.9.0, intelhex-2.3.0

list(ih) causes the Python interpreter to hang up

from intelhex import IntelHex
ih = IntelHex()
ih[0] = 0x55
list(ih)

xrange fails to overflow?

Running the tests gives me

======================================================================            
FAIL: test_xrange_longint (intelhex.test.TestXrangeLongInt)                       
----------------------------------------------------------------------            
Traceback (most recent call last):       
  File "/tmp/nix-build-python2.7-intelhex-2.1.drv-0/intelhex-2.1/intelhex/test.py", line 1601, in test_xrange_longint
    self.assertRaises(OverflowError, xrange, 2684625744, 2684625747)              
AssertionError: OverflowError not raised 

----------------------------------------------------------------------

trying something similar in a python interpreter gives the same results:

$ python                                                                       
Python 2.7.13 (default, Dec 17 2016, 20:05:07)                                                                
[GCC 5.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> xrange(2684625744, 2684625747)                                                                            
xrange(2684625744, 2684625747)                                                                                

while something like this does in fact throw an OverflowError:

>>> xrange(2684625744, 2**100)                                                                                
Traceback (most recent call last):                                                                            
  File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to C long

TypeError while loading from TemporaryFile (Python 3)

I tried to load an HEX-Image from a TemporaryFile() with python 3.

Unfortunately the behaviour of TemporaryFile() differ from a normal file object.
Whereas file object read 'str' type, a TemporaryFile() read and writes 'bytes' type.
You can see the conversion in the code snippet below.

fobj = open('testimage.hex','r')
data = fobj.read()

ftmp = TemporaryFile()
ftmp.write(bytes(data,encoding='UTF-8'))
ftmp.seek(0)

ih.loadfile(ftmp, 'hex')

fobj.close()
ftmp.close()

Executing the code throws following exception:
decode(s, line)
File "D:\Programme\WinPython-64bit-3.4.3.3\python-3.4.3.amd64\lib\site-packages\intelhex_init_.py", line 111, in _decode_record
s = s.rstrip('\r\n')
TypeError: 'str' does not support the buffer interface

In ..._init_.py", line 111, in _decode_record, I inserted

        if isinstance(s, bytes):
            s = s.decode()
        s = s.rstrip('\r\n')

before the s = s.rstrip('\r\n') to convert it to str.

Well, it solves my problem in python 3. I don't know if it is compatible with python 2.
What do you think? Is it applicable in future releases.

padding byte not retained in slices

Hello,

In the current version (2.3.0), the padding byte resets to 0xFF when a slice is taken:

In [14]: ih.padding
Out[14]: 0

In [15]: ih[:4].padding
Out[15]: 255

It would make sense to copy the padding byte to the new instance.

tostring bug when using Python 3.9

intelhex/intelhex/compat.py

Line 60 in 9769d3f

array_tobytes = getattr(array.array, "tobytes", array.array.tostring)

tostring has to become tobytes for this to work in Python 3.9, likely in versions before this as well.

Check two IntelHex objects equality

Hi,
I am looking for effective way of hex file data comparison, mainly equality check.
Is there already some effective way of comparison?

Did you consider overriding of __eq__ ?

Starting Address record line is always the first line, but in original file it was last line

I am testing this module for some production/testing firmware automation and I am super pleased!
I am noticing that when I encode a hex file to a dict with this module, then fetch and re-save to a hex file (the data is stored in a database as a JSON record) one line gets swapped in the file thus causing my md5 checksums to not match. While I am fairly certain the structure of an intel hex file prevents this from being an issue, it would be nice to under stand how this happens.

Encode to database:

file_name = file_path.split('/')[-1]
with open(file_path, 'rb') as file:
    md5 = hashlib.md5(
        file.read()
    ).hexdigest()
intel_hex = IntelHex()
intel_hex.loadhex(file_path)
if image_type.lower() in ['mfg', 'manufacturing']:
    self.manufacturing_fw = intel_hex.todict()
    self.manufacturing_fw_name = file_name
    self.manufacturing_md5 = md5

Decode From database:

if image_type.lower() in ['mfg', 'manufacturing']:
    output_file_name = f'{out_path}/{self.part_number}-{self.manufacturing_fw_name}'
    firmware_data = self.manufacturing_fw

data = {}
for key, value in firmware_data.items():
    if key != 'start_addr':
        key = int(key)
    data[key] = value

intel_hex = IntelHex()
intel_hex.fromdict(data)
intel_hex.write_hex_file(output_file_name)

I need to add these lines upon export to re-arrange the lines to get the exported file to match the original exactly.
Not efficient or the minimum solution, but it works with everything I have tested so far.

file = open(output_file_name, 'r')
lines = file.readlines()
file.close()
output = open(output_file_name, 'w')
total_lines = len(lines)
last_line = lines[-1]
lines.append(lines[-1])
lines[0], lines[total_lines-1] = lines[total_lines-1], lines[0]
lines.append(last_line)
for line in lines[1:-1]:
    output.write(line)
output.close()

I appreciate any help with this!

hex2bin error

Hey. I tried use hex2bin function im my code then tried your script and all the time I get this error:
Traceback (most recent call last): File "hex2.py", line 139, in <module> sys.exit(hex2bin(fin, fout, start, end, size, pad)) File "/usr/local/lib/python3.4/dist-packages/intelhex/__init__.py", line 1001, in hex2bin h.tobinfile(fout, start, end) File "/usr/local/lib/python3.4/dist-packages/intelhex/__init__.py", line 412, in tobinfile fobj.write(self._tobinstr_really(start, end, pad, size)) TypeError: must be str, not bytes
Script I use as python3 hex2bin Blinky.hex

release 2.2

There are some changes waiting to be released, and some pull requests waiting for me as well.

Can you add hex file support for TI C28x DSP?

The ram and rom width of TI C28x DSP such as TMS320F28335 is 16 bits, and data word is big endian. I use command below to generate hex file from ELF file.

hex2000.exe --memwidth=16 --romwidth=16 --intel -o dat.hex firmware.out

Here is a segment of hex file of C28x. Word on address 0x3F2132 is 0x835E and word on address 0x3F2142 is 0x0000 and word on address 0x3F2152 is 0xAAAB.

:02000004003FBB
:20213200835E3F6C09083F71FA0B3F7453F83F7814BE3F7B3AAC3F7DC46D3F7EB10F3F7FEC
:2021420000003F8000004300F98341A20FDB3D49007F00000000BF00AAABBE2AAAAB3E2A74
:0C215200AAAB3D2A88893C088889BC089B
:00000001FF

When I use IntexHex16bit to parse the hex file, it raised AddressOverlapError. Can you add support for C28x? Thank you.

hexinfo.py script error

When trying to run hexinfo.py on an intelhex image, I get the following:

File "c:\projects\test\venv\Scripts\hexinfo.py", line 63, in summarize_yaml                                                                                             
    keys.sort()                                                                               
AttributeError: 'dict_keys' object has no attribute 'sort'                                    

> pip show intelhex -v
Name: intelhex
Version: 2.1
Summary: Python Intel Hex library
Home-page: https://pypi.python.org/pypi/IntelHex
Author: Alexander Belchenko
Author-email: [email protected]
License: BSD
Location: c:\projects\test\venv\lib\site-packages
Requires:
Metadata-Version: 2.0
Installer: pip
Classifiers:
  Programming Language :: Python
  Programming Language :: Python :: 2
  Programming Language :: Python :: 3
  Classifier: Development Status :: 5 - Production/Stable
  Classifier: Environment :: Console
  Classifier: Intended Audience :: Developers
  Classifier: Intended Audience :: Telecommunications Industry
  Classifier: License :: OSI Approved :: BSD License
  Classifier: Operating System :: OS Independent
  Classifier: Programming Language :: Python
  Classifier: Topic :: Scientific/Engineering
  Classifier: Topic :: Software Development :: Embedded Systems
  Classifier: Topic :: Utilities
Entry-points:


> python -V
Python 3.5.2 :: Anaconda 4.1.1 (64-bit)

Is it possible to get rid of those padding byte in list(ih)?

from intelhex import IntelHex
ih = IntelHex('pimp_my_xor.hex')
ih.minaddr()
256
ihx = ih[0x100:0x103]
ihx.tobinstr()
b'\x89\xc2\x8d'
list(ihx)
[255, 255, 255, 255, 255, 255, 255, ..., 255, 255, 137, 194, 141]

bin2hex.py adds extra random byte

I am using bin2hex.py:

bin2hex.py test out.ihex; cat out.ihex
:0200000061326B
:00000001FF

Note '0x6B'.

Input is two chars in a binary file:

Is this expected behaviour?

bin2hex.py should have option to save in the existing or newer file

Option to insert 'Extended Segment Address' record type instead of 'Extended Linear Address'

Hello,
my issue might be considered more as improvement. So I don't know if this is the right place.
I've observed that when You want to use write_hex_file, then You cannot decide which type of Extend Address record type will be used in hex file:

• Extended Segment Address: record type 0x02
  or
• Extended Linear Address: record type 0x04

Always second one will be used (0x04) and it cannot be changed as far I can see.
The code responsible
while cur_addr <= maxaddr:
if need_offset_record:
bin = array('B', asbytes('\0'*7))
bin[0] = 2 # reclen
bin[1] = 0 # offset msb
bin[2] = 0 # offset lsb
bin[3] = 4 # rectyp
I know that having that record type (0x04) shouldn't be a problem for most cases, but I have to deal with some legacy software. I run into problems when this record is not 0x02.
Is there any chance to add an option for 'write_to_hex' to have possibility to have functionality tp force using Extended Segment Address record types in output file.
In particular scenario when You initialize IntelHex with file that contains this kind of records, writing this with 'write_hex_file' (without changing any byte of payload) will result with differences comparing to original.

I can try to create a pull request with changes by myself, but I'm not very experienced Python programmer. I do Embedded (mostly C), but I starting to use Python more and more, especially for tooling.