henrikstengaard/hst-imager

Hst Imager is an imaging tool to read and write disk images to and from physical drives

Hst Imager

Hst Imager is an imaging tool to read and write disk images to and from physical drives. It supports reading from .rar, .zip, .gz, .xz compressed image files when writing to physical drive and writing to .zip, .gz compressed image files when reading from physical drive.

This tool can be used to create new blank images or create images of physical drives like hard disks, SSD, CF- and MicroSD-cards for backup and/or modification and then write them to physical drives.

Warning Hst Imager has been tested extensively regarding it's raw disk access. However it's highly recommended to make a backup of your physical drive or image file, so your working with a copy in case Hst Imager might corrupt it. YOU HAVE BEEN WARNED NOW!

Warning Hst Imager filters out fixed disks, so only USB attached physical drives are accessible. Be very sure to select the correct physical drive. Otherwise Hst Imager might destroy your disk and it's file system. Raw disk access requires administrator privileges, so you need to run as administrator or with sudo.

Supported operating systems

Hst Imager supports following operating systems:

• Windows
• macOS
• Linux

Versions

Hst Imager comes as a gui and a console version.

Gui

See Hst Imager Gui page for installation and usage of Hst Imager Gui.

Console

See Hst Imager Console page for installation and usage of Hst Imager Console.

Features

Hst Imager comes with following features:

• List physical drives.
• Read information from physical drive or image file (supports writing .rar, .zip, .gz, .xz compressed image files).
• Read physical drive to image file (supports writing to .zip, .gz, compressed image files).
• Write image file to physical drive (supports reading from .rar, .zip, .gz, .xz compressed image files).
• Convert image file between .img/.hdf and .vhd (supports reading from .rar, .zip, .gz, .xz compressed image files).
• Create blank .img/.hdf and .vhd image file.
• Optimize image file size.

Hst Imager Console version comes with following additional features:

• File system:
  • Supports local files and directories, image files, physical drives, ISO9660 .iso, Zip archive .zip, Lha archive .lha or Amiga Disk File .adf as source.
  • Supports local files and directories, image files, physical drives or Amiga Disk File .adf as destination.
  • List files and subdirectories in a file system.
  • Copy files from source to destination file system.
  • Extract files from source to destination file system.
• Amiga Disk File:
  • Create ADF disk image file.
• Master Boot Record:
  • Read Master Boot Record information.
  • Initialize Master Boot Record.
  • Add partition to Master Boot Record.
  • Delete partition from Master Boot Record.
  • Format partition in Master Boot Record.
• Rigid Disk Block;
  • Read Rigid Disk Block information.
  • Initialize Rigid Disk Block.
  • Add file system to Rigid Disk Block.
  • Delete file system from Rigid Disk Block.
  • Export file system from Rigid Disk Block to file.
  • Import file systems from Rigid Disk Block or ADF file.
  • Update file system in Rigid Disk Block.
  • Add partition to Rigid Disk Block.
  • Copy partition from one Rigid Disk Block to another.
  • Delete partition from Rigid Disk Block.
  • Export partition from Rigid Disk Block to hard file.
  • Format partition in Rigid Disk Block.
  • Import partition from hard file to Rigid Disk Block.
  • Kill and restore partition in Rigid Disk Block.
  • Update partition in Rigid Disk Block.

Read and write to and from physical drives requires administrative rights on Windows, macOS and Linux.

Img file format

Img file format is a raw dump of hard disks, SSD, CF- and MicroSD-cards and consists of a sector-by-sector binary copy of the source.

Creating an .img image file from a 64GB CF-card using Hst Imager will require 64GB of free disk space on the specified destination path.

Vhd file format

Vhd file format is a virtual hard disk drive with fixed and dynamic sizes.

Fixed sized vhd file pre-allocates the requested size when created same way as .img file format.

Dynamic sized vhd file only allocates storage to store actual data. Unused or zero filled parts of vhd file are not allocated resulting in smaller image files compared to img image files.

Creating a dynamic sized vhd image file from a 64GB CF-card using Hst Imager will only require free disk space on the specified destination path matching disk space used on source physical drive. Zero filled (unused) sectors are skipped, when creating a vhd image resulting in a much smaller image file.

Amiga support

Hst Imager supports Amiga Rigid Disk Block (RDSK, partition table used by Amiga computers) and can initialize new Rigid Disk Block and modify existing Rigid Disk Block.

Reading an Amiga hard drive to an image files is very useful with Amiga emulators to make changes much faster than real hardware and afterwards write the modified image file back to a hard drive.

Amiga emulators with vhd support

Following Amiga emulators support .vhd image files:

• WinUAE 4.9.0: https://www.winuae.net/
• FS-UAE v3.1.66: https://fs-uae.net/

FS-UAE might require following custom option to force RDB mode by manually changing FS-UAE configuration file (replace 0 with other hard drive number if needed):

hard_drive_0_type = rdb

hst-imager's Issues

Proposed feature #2 - Change the Mask

Rarely used features, however if feasible would it be possible adding to the "rdb part add" the mask option?

I have spoiled it in the code with the default value 2147483646 (7ffffffe) which is used in most cases for hard disk images.

The usage that comes to my mind is for images meant for CF cards, where the 0xfffffe mask is used instead of the 0x7ffffffe

Thank you for considering this.

Maybe a little bug - Boot Priority cannot be negative

Hi Henrik,

My tests are continuing with maybe the most difficult image: The A3000 Dual Boot for KS 1.4 one, which requires 4 partitions (two DOS1 and two PFS3)

I am having some other issue with the format that goes a little mad with either two FS in the RDB or too many partitions to format (but it is too early to raise an issue here, need more testing over the weekend with a fresh mind)

For now, I believe I have spoiled a little tiny bug.

The boot-priority in the real amiga can be a negative number (IIRC the range is from -127 to 127), but here it returns some bogus number when set to -1

I could have simply used the boot-priorities from 0 to 4 (5 is normally the floppy) to make it work, but for the sake of completeness of Hst-Imager I wanted to report to you in case you would feel to fix that :-)

This is the full list of commands:

hst.imager.exe blank WB3000.hdf 32GB
hst.imager.exe rdb init WB3000.hdf --size 99%
hst.imager.exe rdb fs add WB3000.hdf pfs3aio PFS3
hst.imager.exe rdb fs add WB3000.hdf FFS_v401 DOS1
hst.imager.exe rdb part add WB3000.hdf WB_1.3 DOS1 1GB --bootable --max-transfer 130560 --buffers 80 --boot-priority -1
hst.imager.exe rdb part add WB3000.hdf WB_2.x DOS1 15MB --bootable --max-transfer 130560 --buffers 80 --boot-priority 1
hst.imager.exe rdb part add WB3000.hdf DH0 PFS3 500MB --max-transfer 130560 --buffers 80
hst.imager.exe rdb part add WB3000.hdf DH1 PFS3 * --max-transfer 130560 --buffers 80
hst.imager.exe rdb part format WB3000.hdf 1 WB13
hst.imager.exe rdb part format WB3000.hdf 2 WB2x
hst.imager.exe rdb part format WB3000.hdf 3 Workbench
hst.imager.exe rdb part format WB3000.hdf 4 Data
hst.imager.exe rdb info WB3000.hdf

JavaScript Error with the GUI version likely due to OS language

Hello Henrik,

First of all, thank you so much for this great utility. I have been using the console version, which works just fine in creating PFS3 images while formatting them. To the best of my knowledge, it is the only utility capable to do so and my powershell scripts has been updated accordingly.

I believe there is a little glitch in the GUI portable (and zipped) versions. On my Windows 10 22H2 x64, the portable version returns a Javascript exeption "uncaught exception: ENOENT..".

I guess the issue is that the error says that it expects "C:\Users\roc\AppData...." but I realised my OS in in Italian with something like "C:\Utenti\roc\AppData". I tried all the three portable versions and launched them either elevated or with standard user.

Would it be possible to invoke something like %userprofile% which maps directly regardless of the language or to use the portable folder where it actually runs?

Regardless, If you would be open to some additional little features to Hst-imager, I could suggest a few more in case some of them would be feasible and felt useful for all.

Again. Small glitch, gret util ;-) Tak.

Proposed feature #1 - Fill the partition with a percentage of the disk space available

The * option is really useful, because it fills the remainder of an image with the final paritition. For instance, when the DH0 is 100MB the DH1 takes the rest of the space, regardless how large the image is.

Maybe I am little "old-style" but as a matter of precaution, I tend to use the 99% of the remainder of the disk to avoid "final cylinder" issues.

If you feel appropriate, would it be possible adding the possibility to fill with a given percentage (say 99%) instead of the 100% of the space?

Thank you for your consideration.

Excellent program, immediately replaced anything else I had - small request for SATA

Hi!

thanks, this is a great piece of software!

I sometimes clone SATA SSDs but to transfer the hybrid GPT system I need to do a sector-by-sector clone. Using USB3 2.5" case that gives a speed of 160-17-mb/s but maybe connecting the destination disk to a SATA port would speed up the cloning procedure. Can SATA drives be detected as destination somehow???

br,
george

Possible format issue with two FSes in the RDB

Hi Henrik,

Normally one FileSystem in the RDB is more than enough, however there is another corner-case where this is not sufficient.

The A3000 Dual Boot (1.3 plus 3.2.x) with the KS 1,4, requires the first two partitions to be DOS1 (WB_1.3 and WB_2.x) with the FFS up to the 40.1. The second is the one where the superkickstart is.

The last two 3.2.x partitions can be PFS3.

Unfortunately with two FSes in the RDB, there are no errors during the creation of the image and the formatting of partitions, but (unless I am doing something wonky ) the last two are not mountable/formatted.

Would it be possible allowing two FS in the RDB?

Thank you in advance for your help!

These are the commands. I tried both PFS3 and PDS3.

hst.imager.exe blank WB3000.hdf 32GB
hst.imager.exe rdb init WB3000.hdf --size 99%
hst.imager.exe rdb fs add WB3000.hdf pfs3aio PFS3
hst.imager.exe rdb fs add WB3000.hdf FFS_v401 DOS1
hst.imager.exe rdb part add WB3000.hdf WB_1.3 DOS1 1GB --max-transfer 0x0001fe00 --mask 0x7ffffffe --buffers 80 --bootable --boot-priority 0
hst.imager.exe rdb part add WB3000.hdf WB_2.x DOS1 0.015GB --max-transfer 0x0001fe00 --mask 0x7ffffffe --buffers 80 --bootable --boot-priority 1
hst.imager.exe rdb part add WB3000.hdf DH0 PFS3 1GB --max-transfer 0x0001fe00 --mask 0x7ffffffe --buffers 80
hst.imager.exe rdb part add WB3000.hdf DH1 PFS3 * --max-transfer 0x0001fe00 --mask 0x7ffffffe --buffers 80
hst.imager.exe rdb part format WB3000.hdf 1 WB13
hst.imager.exe rdb part format WB3000.hdf 2 WB2x
hst.imager.exe rdb part format WB3000.hdf 3 Workbench
hst.imager.exe rdb part format WB3000.hdf 4 Data
hst.imager.exe rdb info WB3000.hdf

List directory content and copy files to and from images

My hst amiga library used for accessing Rigid Disk Blocks and formatting Fast File System and PFS3 partitions has recently been expanded with support for reading and writing files to and from both Fast File System and PFS3.

The idea is to add basic file system commands like dir, copy, rename, delete and move. Copy will support source set to a physical drive or image file and destination to a folder like d:\copy-to-amiga and vice-versa.

Example of listing partitions in Rigid Disk Block:

hst.imager.exe fs dir D:\test.vhd rdb

[20:32:59 INF] Hst Imager v0.1.0 (12/08/2022 15:31:48)
[20:32:59 INF] Henrik Noerfjand Stengaard
[20:32:59 INF] [CMD] fs dir D:\test.vhd rdb

Entries:

Name |  Size | Date | Attributes
-----|-------|------|-----------
DH0  | <DIR> |      |
DH1  | <DIR> |      |
DH2  | <DIR> |      |

3 directories, 0 files
[20:33:00 INF] Done

Example of listing Super Frog WHDLoad contents in Rigid Disk Block DH1 partition:

hst.imager.exe fs dir D:\test.vhd rdb/dh1/whdload/games/s/superfrog

[16:33:20 INF] Hst Imager v0.1.0 (12/08/2022 15:31:48)
[16:33:20 INF] Henrik Noerfjand Stengaard
[16:33:20 INF] [CMD] fs dir D:\test.vhd rdb/dh1/whdload/games/s/superfrog

Entries:

Name                 |      Size | Date                | Attributes
---------------------|-----------|---------------------|-----------
Disk.1               |    960 KB | 2006-09-08 06:47:30 | ----RWED
Disk.2               |    960 KB | 2006-09-08 06:48:00 | ----RWED
Disk.3               |    960 KB | 2006-09-08 06:48:30 | ----RWED
igame.iff            |   30.3 KB | 2021-09-29 09:30:28 | ----RWED
Manual               |   13.7 KB | 2001-05-03 09:22:50 | ----RWED
Manual.info          |      2 KB | 2021-07-29 04:08:26 | H---RWED
ReadMe               |      7 KB | 2021-07-29 04:07:18 | ----RWED
ReadMe.info          |      2 KB | 2021-07-29 04:08:26 | H---RWED
Superfrog.highs      | 160 bytes | 2001-05-03 09:51:30 | ----RWED
Superfrog.info       |   18.3 KB | 2021-07-29 04:08:30 | ----RW-D
Superfrog.slave      |    6.6 KB | 2021-07-29 04:07:20 | ----RWED
SuperfrogIntro.info  |   19.6 KB | 2021-07-29 04:08:36 | ----RW-D
SuperfrogIntro.slave |    3.4 KB | 2021-07-29 04:07:28 | ----RWED

0 directories, 13 files
[16:33:21 INF] Done

Low: DOS7 format

Although I am interested in PFS3 images, I have tested the DOS7 with the FFS 47.4

It works per-se, however it takes approx. 20 minutes to format a 31 GB Data partition.

Not an issue for me, but just wanted to inform:

[14:45:35 INF] Hst Imager v0.9.254 (12/02/2022 18:38:53)
[14:45:35 INF] Henrik Noerfjand Stengaard
[14:45:35 INF] [CMD] rdb part format WB3x.hdf 2 Data
[14:45:36 INF] Formatting partition in Rigid Disk Block at 'WB3x.hdf'
[14:45:36 INF] - Name 'DH1'
[14:45:36 INF] - DOS type '0x444F5307' (DOS\7)
[14:45:36 INF] - Volume name 'Data'
[15:05:57 INF] Done

These are the commands:

hst.imager.exe blank WB3x.hdf 32GB
hst.imager.exe rdb init WB3x.hdf --size 99%
hst.imager.exe rdb fs add WB3x.hdf FFS_v474 DOS7
hst.imager.exe rdb part add WB3x.hdf DH0 DOS7 1GB --max-transfer 0x0001fe00 --mask 0x7ffffffe --buffers 80 --bootable --boot-priority 0
hst.imager.exe rdb part add WB3x.hdf DH1 DOS7 * --max-transfer 0x0001fe00 --mask 0x7ffffffe --buffers 80
hst.imager.exe rdb part format WB3x.hdf 1 Workbench
hst.imager.exe rdb part format WB3x.hdf 2 Data
hst.imager.exe rdb info WB3x.hdf

Windows version: Files with leading/trailing spaces or dots

Hello,

First of all, thank you so much for adding the copy feature. It is quick and efficient and finally I am able to retire the "copy with WinUAE" that I have been using so far to create the PFS/3 images.

As a filesystem, NTFS manages difficult amiga files such as those with a leading/trailing spaces or trailing dots, but unfortunately the Windows explorer doesn't handle them. As a workaround, personally I either use the the syntax \?"Drive:\DirPath\FileName1.") or WSL to rename or copy them.

As you know, there are some of these difficult files within the WHDLoad folders - e.g. [17:16:57 INF] WHDLoad\Games\A\Astate\data\dat\PIOCHE. (2.1 KB) [17:16:57 ERR] Failed to execute command 'Hst.Imager.Core.Commands.FsCopyCommand'

Currently the windows hst.imager is impacted by this Windows-Explorer's constrain, thus it reports an error while stopping the copy of the files.

Would it be possible to either somehow handle these files or, as an alternative approach, hst.imager could continue despite skipping the offending files?

Luckily, there is already a workaround :-) It is enough to use the Linux flavour of hst.imager within Windows Subsystem for Linux so it works like a champ.

So, if this would not be feasible, no issues in using the Linux's one but having in Windows would be juicy ;-)

Thank you in advance for your consideration and keep up the good work!

Ability to detect virtual Amiga drive inside MBR 0x76 partition

SD cards partitioned for use with the pistorm accelerator boards have a partition with ID 0x76 that are recognized by the Amiga as a disk.

The process for manually partitioning a drive in this manner is described here:

https://github.com/michalsc/Emu68/blob/master/docs/tutorials/SD_Preparation.md

HST-Imager can be used to create a RDB based disk image, which then can be made pistorm usable by running it through this tool:

https://github.com/PiStorm/hdf2emu68

However, in the resulting image file the RDB disk is now "hidden" inside the 0x76 partition and HST-Imager no longer recognizes it.

It would be nice if HST-Imager could recognize the "disk/partitions inside the partition", so that the process of maintaning a pistorm compatible image could be done with one (less) tool.

This way of having a virtual Amiga drive within a DOS partition seems to have been established with Amithlon, and is also recognized by WinUAE.

Proposed feature #3 - non-RDB images

This is difficult and likely not feasible/appropriate.

The UAE4All (at least the PS-Vita version) unfortunately works only with non-RDB images. You have created a HstWB version for it, so you know this well.

Now, Hst-image is meant for mastering RDB images, while a non-RDB seems really out of scope.

Nevertheless, if this could create non-RDB images (I guess the only option is FFS here) it would close another corner-case.

Once again, thank you for your consideration.

Use file attributes in .uaem files when copying from/to host filesystem

How do you feel about adding the option to use the attributes files written by (FS-)UAE when copying to/from the host computer filesystem? I often use UAE with a "work" volume mounted to a plain directory, to which UAE will then write out the Amiga fs attributes to a sidecar .uaem file. I'm sure you're already familiar with these, but for completeness sake they are plain text files on the following format:

hsparwed 2023-12-16 22:41:53.12

Further documentation here.

For reference, a similar feature was added to amitools here.

It would be awesome if I could copy a file tree from my UAE "file volume" to a real Amiga volume with hst.imager with all the attributes intact! I believe that fs copy and fs extract are the commands that would be affected by this feature request.

Also, thanks for a great tool that I'm getting good use of already!