vortigont/esp32-flashz

zlib compressed OTA firmware update for ESP32. Implements on-the-fly OTA images decompression on upload/flashing

ESP32-FlashZ

an arduino library that provides zlib compressed OTA update feature for esp32

EXAMPLES | CHANGELOG | | |

Unlike esp8266, esp32's bootloader does not (yet) support compressed image updates. But esp32 has a miniz lib decompressor in ROM, so it can be used to decompress zlib packed stream on the fly during OTA update and write decompressed data to SPI flash. This could be done both for firmware and filesystem image.

I wrote this lib just for fun to get the idea of miniz functions hidden in esp32 ROM. FlashZ lib is integrated into EmbUI framework as OTA update handler. The code inspired by esptool that does the same with it's serial flasher stub.

Features

• low code overhead, deco algo is already present in ROM
• both firmware and filesystem compressed images upload supported
• compatible with ESP32 WebServer, autodetect compressed/non-compressed images
• compatible with AsyncWebServer, autodetect compressed/non-compressed images
• stream decompression, i.e. via http client (fetch and flash compressed image from remote URL)
• PlatformIO integration via post_flash.py script that automates on-the-fly compression and OTA upload for your project
• on-the-fly compressed upload example via pako js lib (tnx to @playmiel for contribution)

Requirements

• 32k of heap memory during decompression for dict data
• derives from Arduino's UpdaterClass to perform flash operation on inflated data
• firmware images must be compressed in zlib stream (not a gzip format file!). There are many ways to do this, i.e.
  • use pigz tool with -z flag
  • use python's zlib module (check post_flash.py script for Platformio)
  • use perl's Compress::Zlib module
  • use pako js lib

Check examples to get more idea on how to intergate this lib into platformio projects

Compatibility

Platform	Firmware	Filesystem
ESP32	✔️	✔️
ESP32-S2	✔️	✔️
ESP32-S3	✔️	✔️
ESP32-c3	✔️	✔️

Comparison

some basic tests

OTA update	Origin	zlib
esp32, fw, ~1MiB	9.8 s	10.1 s
esp32, fs, ~1.5MiB, 90% sparse	9 s	6 s
esp32-c3, fw, ~1MiB	9.2 s	9.1 s
esp32-c3, fs, ~1.5MiB, 90% sparse	4.1 s	2.5 s
esp32-s2, fw, ~1MiB	7.5 s	7.5 s
esp32-s2, fs, ~1.5MiB, 90% sparse	4.7 s	1.8 s

Using FlashZ lib

FlashZ Lib consists of a low level FlashZ singleton that derives from a built-in Arduino's UpdateClass class and provides additional methods to handle zlib compressed data. It tries to maintain same API as Update.

FlashZ::beginz suposed to be called instead of UpdateClass::begin call to intialize update procedure. It allocates zlib Inflator structures and dictionary memory. For uncompressed data FlashZ::begin could be used as usual with UpdaterClass.

FlashZ::writez is called to inflate and flash zlib compressed block of data. bool final flag is used to signal last piece of data input.

FlashZ::writezStream can read a standart Stream class objects, decompress and write decompressed stream to flash. NOTE: total stream length must be known in advance, so that Inflator insures a proper end of stream is reached on decompression.

FlashZ::abortz or FlashZ::endz must be called to end the update and release dynamically allocated Inflator memory.

To stich FlashZ with networking and OTA updates here is a FlashZhttp class. This is not a complete OTA updater solution but more of a reference implementation example. Any real-life projects could easily implement something similar with more features, bells and whistles. FlashZhttp class integrates WebServer or AsyncWebServer file upload feature with FlashZ low level methods. Also it can initiate streamed download via http client from a remote URL (only plain http). FlashZhttp methods includes some heuristic in attempt to autodetect file image format and type, so that it can handle both compressed and uncompressed images transparently. But for compressed file it can't autodetect between firmware and FS image, so it need some metadata to differetiate. This is implemented via additional POST data fields.

Build-time options

By default AsyncWebServer support is not build into lib, do not want to intorduce dependency for external lib. To get AsyncWebServer support, FlashZ lib must be build with FZ_WITH_ASYNCSRV flag. This could be done via PlatformIO build_flags. AsyncWebServer and ESP32 WebServer support options are mutually exclusive due to some definitions clashing. See EXAMPLES projects for further details.

By default FlashZhttp class within the lib includes support for HTTP Client to be able to download and flash an image from a remote URL. It uses Arduino's HTTPClient.h lib for that, which ALWAYS includes SSL support, even if client code does not meant to be using it. SSL support makes a huge impact on resulting firmware image size, it grows in about 100-120k. If you do not need HTTP Client support for the sake of reducing resulting image size you can define FZ_NOHTTPCLIENT build flag to completely disable HTTP Client and allow linker exclude SSL-related code from the resulting firmware image. This is here untill I find a better way to workaround it, maybe a flag to exclude FlashZhttp completely?

Also you should always specify NO_GLOBAL_UPDATE build flag for your project to prevent Arduino's UpdateClass creating it's instance by default. FlashZ uses it's own instance of a derived class and default one just wastes your memory (about 180 bytes). See arduino-esp32/pull#8500

On-the-fly compression of uploaded images via pako js lib

An example of implementing firmware updating web page with embedded js code that compresses raw uploaded images on-the-fly could be found in asyncserver-flashz-pakojs.

Integration with PlatformIO

It is pretty easy to integrate PlatformIO with HTTP OTA update via post build scripting. Python's zlib module could be used to compress firmware image after building and http-client module to upload a compressed image to ESP32 board Over-the-Air. See a reference implementation in post_flashz.py example. It relies on FlashZhttp class methods to process POST form data but could be adjusted easily. Additional platformio.ini variables are used to set remote address of a board. Uploading compressed firmware/FS is done automagicaly via simple pio run -t upload. (MCU must be connected to network and reachable).

Using CLI tools for updates

I'm a linux user and prefer to use cli tools for automating tasks rather than web browser. So here are some oneliners for ESP32-FlashZ updating

• compress and upload firmware image to ESP

pigz -9kzc .pio/build/esp32c3/firmware.bin | curl -v http://$ESPHOST/update -F file=@-

• compress and upload File System image to ESP

pigz -9kzc .pio/build/esp32-s2/littlefs.bin | curl -v http://$ESPHOST/update -F "img=fs" -F file=@-

• trigger esp32's firmware self-update from a remote host

curl http://$ESPHOST/update -F "img=fw" -F "url=http://$REMOTE/download/firmware.bin.zz"

• trigger esp32's FS self-update from a remote host

curl http://$ESPHOST/update -F "img=fs" -F "url=http://$REMOTE/download/littlefs.bin.zz"

License

Since I get the idea from a esptool code, this lib inherits esptool's GNU General Public License v2.0

esp32-flashz's Issues

Error: Not Enough Space!

Hello,
I have an error when running the update and I don't know how to solve it.

My uncompressed code version is able to update a 1048 KB BIN, but when I try the esp32-flashz version with the same BIN compressed to 633 KB I always get the following error when I am about 60% of the way through the download:

[139062][E][flashz.cpp:294] flash_cb(): [FLASHZ] ERROR, flashed 0 of 32768 bytes chunk, err: Not Enough Space!
[139062][E][flashz.cpp:263] writez(): [FLASHZ] Inflate ERROR: -1

I'm using ESP32 and I'm writing with FlashZ::getInstance().writez(const uint8_t *data, size_t len, bool final)

Can you help me?

AsyncTCP watchdog reset on highly compressed images

If high compress ratio sparse file system images are uploaded via AsyncServer, watchdog may trigger CPU reset. Looks like writing decompressed image takes longer time than AyncTCP's timeout for packet processing.

Similar implementation for ESP-IDF available?

I am looking for a library exactly like this, but designed for the ESP-IDF.

Any hints?

Or is it worth to port it back to the ESP-IDF?

inflate failure - MZ_ERR: -3, inflate status: -2

[190037][I][flashz.cpp:298] flash_cb(): [FLASHZ] flashed 32768 bytes
[190099][D][flashz.cpp:171] inflate_block_to_cb(): [FLASHZ] CB - idx:1671168, head:1065363604, dbgn:0, dend:0, ddatalen:32768, avin:1085, tin:1018615, tout:1671168, fin:0
[190359][I][flashz.cpp:298] flash_cb(): [FLASHZ] flashed 32768 bytes
[190375][W][flashz.cpp:144] inflate_block_to_cb(): [FLASHZ] inflate failure - MZ_ERR: -3, inflate status: -2
[190376][E][flashz.cpp:263] writez(): [FLASHZ] Inflate ERROR: -3
[190380][W][flashz-http.cpp:163] file_upload(): [FZ-HTTP] OTA failed in progress: No Error
I don't know why I have this error always in the same place with a compressed file sent to asyncwebserver either with the command

pigz -9kzc firmware.bin | curl -v http://192.168.0.165/update -F file=@-

or with my code :

function sendFile(file, url) {
  var reader = new FileReader();
  reader.onload = function(e) {
    var data = new Uint8Array(e.target.result);
    // Compress only if the file is firmware and not already compressed
// Compress only if the file is firmware and not already compressed
if (url.includes("firmware") && !file.name.endsWith('.zz')) {
  var compressedData = pako.deflate(data);
  var decompressedData = pako.inflate(compressedData);

  // Les données décompressées doivent être identiques aux données originales
  console.log('Les données sont identiques:', data.toString() === decompressedData.toString());
  
  data = compressedData; // Utiliser deflate pour la compression zlib
}

    var blob = new Blob([data], {type: 'application/zlib'}); 
    var formData = new FormData();
    formData.append('file', blob, file.name.endsWith('.zz') ? file.name : file.name + '.zz'); // Changez l'extension du fichier pour .zz
    formData.append("img", url.includes("spiffs") ? "fs" : "firmware");

    fetch("/update", { 
      method: "POST",
      body: formData
    })
    .then((response) => {
      if (!response.ok) {
        throw new Error("Erreur réseau lors de l'envoi du fichier.");
      }
      return response.text();
    })
    .then((data) => {
      console.log(data);
      document.getElementById("message_" + (url.includes("spiffs") ? "spiffs" : "firmware")).innerText = data;
    })
    .catch((error) => {
      console.error("Erreur lors de l'envoi du fichier :", error);
      document.getElementById("message_" + (url.includes("spiffs") ? "spiffs" : "firmware")).innerText =
        "Erreur lors de l'envoi du fichier "+ (url.includes("spiffs") ? "spiffs" : "firmware");
    });
  };
  reader.readAsArrayBuffer(file);
}

-Werror=unused-variable -Wespressif=lets-break-everyones-library

Hi!

got this error treated as a warning while running a compilation CI test with compilation warnings set to all.

  /home/runner/Arduino/libraries/esp32-flashz/src/flashz.cpp: In member function 'size_t FlashZ::writezStream(Stream&, size_t)':
  /home/runner/Arduino/libraries/esp32-flashz/src/flashz.cpp:307:9: error: unused variable 'err' [-Werror=unused-variable]
       int err = deco.inflate_stream_to_cb(data, len, [this](size_t i, const uint8_t* d, size_t s, bool f) -> int { return flash_cb(i, d, s, f); });
           ^~~
  cc1plus: some warnings being treated as errors

there are several ways this can be fixed:

• by setting compiler warnings to default (which I obviously did in my CI script)
• by ignoring the warning, either before I include esp32-flashz, or by modifying esp32-flashz.cpp:

#pragma GCC diagnostic ignored "-Wunused-variable"

• by modifying esp32-flashz.cpp and actually using err in the unit:

if( err ) {
   // do something
}

Error beginz

size_t totalSize = atoi(sizeFromString);
I replace Update.begin(totalSize) by Flashz::beginz(totalSize)
But when i try to compile:
cannot call member function 'bool FlashZ::beginz(size_t, int, int, uint8_t, const char*)' without object

And the same with:
cannot call member function 'size_t FlashZ::writez(const uint8_t*, size_t, bool)' without object