Changes to the target description format.

[probe-rs]

Changes to the target description format.

[Tiwalun]

I quite like that I can just use a YAML file for custom targets right now. When building a custom flash loader for RISCV, it was very convenient to just edit a text file, without having to compile it afterwards. I think using WASM will make this quite a bit more complicated, especially for the simple cases, where no target specific code is required except for the flash loader itself.

For targets with specific code required, I quite like having some kind of limited scripting available,
like e.g. the Segger script files. They offer the ability to override some functions, like for target reset, with custom functions. I think a full scripting language or even Rust is not really required, usually the ability to write / read some memory locations is enough.

With the changes regarding the core list and memory map I agree, to support multiple cores this sounds reasonable.

[Disasm]

A few thoughts previously posted on Matrix:

This approach comes with a downside of course: The user cannot hotload any target descriptions.

If Rust is going to replace Yaml, then why not to leave Yaml->Rust compatibility layer for loading yaml files at runtime? It will not be as powerful as pure rust, but anyway.
In fact, this yaml->rust thing is the only one I don't understant, but you clearly have more context than me :)

Other things look good, even from my riscv perspective.

Also I know it's not the case and probably will never be the case, but if you have an FPGA softcore, then your core configuration may be changed from build to build and that's where yaml description may help as it can be autogenerated along the core.

And one thing to add: some targets are more than just a core. For example, you may have a board with a FE310 chip an it can have different flash chips on different boards, also QSPI or generic SPI connection. These may require either board-specific flash loaders or somewhat generic "SPI flash" loader code that knows nothing about the core.

[cbiffle]

Yeah, I'd like to register a strong preference for being able to tweak and iterate on target definitions without rebuilding my tools/patching dependencies. I really like how data-driven the current probe-rs implementation is, without having a module per potential target processor -- where eventually target-processor-specific hacks get introduced. The current YAML model succeeds at this, even if YAML itself is not my favorite. (As long as it's a serde-compatible format, it almost doesn't matter anymore -- we're using TOML in our system builder but we could easily switch to JSON, etc.)

Format-wise, being able to use actual ELF binaries instead of a base64 string and a YAML-encoded symbol table would also be nice for iterating. A target description could potentially be a directory containing a control file and a collection of ELF binaries for verbs.

Remember that "the flash" is often not a thing. More complex systems (as @Disasm noted) will have several banks of internal flash, an external memory controller or two, and possibly the ability to boot from eMMC. The Keil CMSIS pack files include named flasher programs for each potential target (QSPI, internal, etc.), and including these as string-keyed named memories is probably the right approach, since vendors (or board makers!) can add new ones.

Otherwise the changes you propose look fine.

[mvirkkunen]

rhai might also be a good candidate for the scripting language. It looks like Rust, it's written in Rust, and it has exactly one mandatory dependency (smallvec).

[Yatekii]

I think I need to clarify how I intended to replace the YAML format with Rust/WASM :)

By no means I intend to recompile probe-rs or tools that use probe-rs on each ever so slight target config change :D That would be absolutely terrible to use.

What I rather suggest is that we load a WASM module (one for each target) during runtime, which contains certain symbols for things like the memory map, the algorithm blob, etc. Just like the YAML files.
But WASM would enable us to define a few handler signatures, let's say for unlocking a chip, which the implementor of the target definition could define additionally. We could then load those too if present and execute them during runtime!
This would only ever require recompilation of the WASM blob.

Format-wise, being able to use actual ELF binaries instead of a base64 string and a YAML-encoded symbol table would also be nice for iterating. A target description could potentially be a directory containing a control file and a collection of ELF binaries for verbs.

Actually, why I chose it to be base64 is just because it was easier to handle in the beginning. It sounds like a good thing to have separate files for the blobs! So let's add that to the feature list!

I am aware that there is several differently named algorithm blobs for different usages. The problem here is that those are all assigned to a specific region of memory. so there is actually ELF blobs where you might need 3 different algorithms (say, normal flash, option bytes, and some more config). How do you enable the user to select ALL of the algorithms they need and not just one? ;) Because not each target has the same amount and not every algorithm works for all memory regions :) At the moment we discover automatically which algorithm we need by the section. And we patched the target files to only contain one algorithm per region. I think this is not an issue within Keil because you have to select the algorithm manually I think (no guarantees).

[Yatekii]

I would rather not create a proprietary script dialect as this is time consuming, an entirely new project and not appealing to many (including me) I guess, as they have to learn some new language.

If we add some scripting I would prefer to use an existing language (without any real preference which one).

Other than that, I can understand that you all don't want to run an additional compile pass, even tho I don't see any trouble with it :)

[Tiwalun]

There are some existing projects for scripting languages listed on Awesome Rust: https://github.com/rust-unofficial/awesome-rust#scripting. I was thinking of one of these, not implementing everything from scratch 😉

[Yatekii]

The main question here is how we make sure the language is actually maintained into the future and not just an experiment :)

mun-lang looks nice here, but might not be the most mature.

[benmkw]

is RustObjectNotation something that might fit here? https://github.com/ron-rs/ron
I've not used it yet but seems like rust+plaintext could be a sweet spot.

If no computation is necessary I think it would be best to keep it simple and have a config file format instead of a scripting language.

[Yatekii]

So I have made the changes I think are required for starters in #367.

The thing that is missing still is the addition of debug sequences and a mechanism for detecting targets.

Optimally we make those changes in multiple PRs. So this one is a start in that direction.

In this same PR I would also like to change the structure of the target files as follows:

Before we basically had

• family1.yaml
• family2.yaml
• ...

Now I would like to propose this structure

• st
  • sequences.lua
  • f0
    • targets.yaml
    • sequences.lua
    • algo1.elf
    • algo2.elf
  • f1
    • ...
• nordic
  • ...

You can see the .lua ending. @mvirkkunen and I have located Lua as a promising candidate for sequences.
I had a feeling that most of the people involved didn't like WASM as a format for this. And I don't see Python, JS, Ruby, TCL or any weird custom language as actually possible candiates.
Lua comes without many dependencies, is meant for embedding, has nice Rust bindings with a proper sandbox (still exploitable, but our application is not security cirtical in that regard), has somewhat sane syntax and data types (it has actual integers and not just floats).
@mvirkkunen has made a first try which resulted in this: https://gist.github.com/mvirkkunen/5c5850c640d0c7a5628ec7cd4cb46884.

The idea is then to have per target sequences and per manufacturer sequences. The per manufacturer sequences are for auto-detection of a target for example.

If you have any inputs, ideas, objections or anything else, please state them here so we can finally get this going to fix some of our longest standing issues :)

[Yatekii]

I'll close this as most changes have been made and if we need further changes we can do this with the relevant issue/PR.