azriel91/peace

Zero Stress Automation

🕊️ peace – zero stress automation

peace is a framework to build empathetic and forgiving software automation.

See:

• peace.mk for the project vision.
• Background for the motivation to create this framework.
• Operations UX for a book about the dimensions considered during peace's design and development.

Guiding Principles

• A joy to use.
• Ergonomic API and guidance to do the right thing.
• Understandable output.

Features

Symbol	Meaning
🟢	Works well
🟡	Partial support
⚫	Planned
🔵	Compatible by design
🟣	Works, "fun idea"

• 🟢 Idempotent: Multiple invocations result in the goal outcome.
• 🟢 Clean: Every item creation is paired with how it is cleaned up.
• 🟢 Understandable: Progress is shown at an understandable level of detail.
• 🔵 Understandable: Error reporting is compatible with miette.
• 🟡 Interruptible: Execution can be interrupted.
• 🟢 Resumable: Automation resumes where it was interrupted.
• 🟢 Diffable: States and diffs are serialized as YAML.
• 🟢 Efficient: Tasks are concurrently executed via fn_graph.
• 🟢 Namespaced: Profile directories isolate environments from each other.
• 🟢 Type Safe: Items and parameters are defined in code, not configuration.

Roadmap

• 🟢 Define items to manage with automation.
• 🟢 Define dependencies between items.
• 🟢 Define "apply" logic.
• 🟢 Define "clean up" logic.
• 🟢 Discover current and goal states.
• 🟢 Define diff calculation between states.
• 🟢 Store and recall parameters across commands.
• 🟢 Diff states between multiple profiles.
• 🟢 Type-safe referential parameters -- specify usage of values generated during automation as parameters to subsequent items.
• 🟡 Feature-gated incremental functionality.
• 🟡 Off-the-shelf support for common items.
• 🟡 Dry run.
• 🟣 WASM support.
• ⚫ Cancel-safe interruption via tokio-graceful-shutdown.
• ⚫ Secure-by-design Support: Encrypted value storage, decrypted per execution / time based agent.
• ⚫ Tutorial for writing a software lifecycle management tool.
• ⚫ Built-in application execution methods -- CLI, web service.
• ⚫ peace binary for configuration based workflows.
• ⚫ Web based UI with interactive graph.
• ⚫ Agent mode to run peace on servers (Web API invocation).

Further ideas:

• Back up current state.
• Restore previous state.
• Telemetry / metrics logging for analysis.

Examples

Examples are run using --package instead of --example, as each example is organized as its own crate.

cargo run --package $example_name --all-features

# e.g.
cargo build --package download --all-features
cargo run -q --package download --all-features -- init https://ifconfig.me ip.json

for cmd in status goal diff ensure ensure diff clean clean diff
do
    printf "=== ${cmd} ===\n"
    cargo run -q --package download --all-features -- --format text $cmd
    printf '\n'
done

# Look at metadata that Peace has saved
find .peace -type f -exec bash -c 'echo \# {}; cat {}; echo' \;

# Clean up the metadata directory
rm -rf .peace

WASM

The download example can be built as a web assembly application using wasm-pack:

cd examples/download
wasm-pack build --target web

In the examples/download directory, start an HTTP server, and open http://localhost:8000/:

python3 -m http.server 8000 # or
simple-http-server --nocache --port 8000 -i

License

Licensed under either of

• Apache License, Version 2.0, (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

peace's Issues

Ensure command

Implement the ensure command.

This does not include implementing reading previous States from disk and comparing with freshly fetched States as a guard.

Remove `StatesCurrentDiscoverCmd` and `StatesDesiredDiscoverCmd`

In #107, the StatesDiscoverCmd was changed to be able to discover current or desired states, or both, and present progress and collect errors. This makes StatesCurrentDiscoverCmd and StatesDesiredDiscoverCmd redundant.

Tasks:

• Remove StatesCurrentDiscoverCmd and StatesDesiredDiscoverCmd.
• Add tests for StatesDiscoverCmd.

Automation Software Developer perspective

Adjust the framework and prepare a presentation that caters for the automation software developer perspective.

Keep in mind it should still accommodate the following perspectives:

• User (of the automation software)
• Item implementor
• Framework maintainer
• Business

In the future, we can add presentation material for the other perspectives as well.

Tasks

• Rename ItemSpec to Item.

• Presentation material should demonstrate the contrast between YAML and code.

  • Discovering ordering between items.
  • Discovery through code completion instead of web searching.
  • Compile time checking instead of "try and see", rollback.
  • Not limited to cloud provider resources.

`StateDiscover` command

Note: This is different from git fetch, which fetches tracked states from remotes. This fetches states for each item.

Retrieving current and desired states can be slow. We should fetch it on request and read from the last fetched state.

Tasks:

• Create a .peace directory to store Peace automation data.

• Current States should be stored in the .peace directory.

• StatesDesired should be stored in the .peace directory.

• Create StatesDiscoverCmd to call both StateCurrentFnSpec and StateDesiredFnSpec.

• Update StatesCurrentCmd to read from .peace directory and output current state.

• Update StatesDesiredCmd to read from .peace directory and output state desired.

• Display StatesCurrent / StatesDesired in a format suitable for the caller.

  Deferred to #28.

.peace directory considerations

• What will be stored in the .peace directory when designing the structure.

  • Significant command executions

    StateDiscover, EnsureOpSpec, and CleanOpSpec reports (record / alter state).

    Retention strategies: unlimited, rollover every n executions.

  • States -- current and desired.

• Searchability: Should be easy to text search via grep.

• Sortability: Easy to see information from file names

• Version controllability:

  • Should be easy to understand diff.
  • What should be version controlled by VCS, vs what should be flattened?

Possible structure:

workspace
|- .peace
    |- profile1 / main / default
    |   |- .history
    |   |   |- 00000000_2022-08-21T20_48_02_init.yaml
    |   |   |- 00000001_2022-08-21T20_48_07_fetch.yaml
    |   |   |- 00000002_2022-08-21T20_50_32_ensure_dry.yaml
    |   |   |- 00000003_2022-08-21T20_50_43_ensure.yaml
    |   |   |- 00000004_2022-08-22T08_16_09_clean_dry.yaml
    |   |   |- 00000005_2022-08-22T08_16_29_clean.yaml
    |   |
    |   |- .meta.yaml  # Store the last fetched time so we can inform the user.
    |   |              # Should time be stored per item spec, or per invocation?
    |   |
    |   |- init.yaml  # Parameters used to initialize this profile
    |   |             # We write to this so that each time the user re-`init`s,
    |   |             # if they version control it, they can rediscover the
    |   |             # states from previous inits, and clean them up.
    |   |
    |   |- states.yaml
    |   |- states_desired.yaml
    |
    |- profile2
        |- .history
        |   |- 00000000_2022-08-21T20_48_02_init.yaml
        |   |- 00000001_2022-08-21T20_48_07_fetch.yaml
        |
        |- .meta.yaml
        |- init.yaml
        |- states.yaml
        |- states_desired.yaml

Consolidate `*Cmd` logic

Currently:

• EnsureCmd and CleanCmd delegate to ApplyCmd.
• ApplyCmd and StatesDiscoverCmd both collect outcomes to surface errors as a whole.

What we would like for all commands:

• Ability to use other commands' logic, but all of the *Cmd invocations share one progress output.
• Output type is generic
• Collect errors from each item spec, and return all values and errors as a whole.

Use cases:

• EnsureCmd:

  • Forward iteration through each item.

    Calls state_current, apply_check, apply_exec in sequence.

• CleanCmd

  • Forward iteration through each item.

    Calls try_state_current.

  • Reverse iteration through each item.

    Calls apply_check, apply_exec in sequence.

• StatesDiscoverCmd:

  • Forward iteration through each item.

    Calls try_state_current / try_state_desired.

User Friendly Progress Output

User Friendly Progress Output

Progress Rendering

During command execution, to provide understandable output, we should have:

• Progress bars
• Elapsed time
• Estimated time remaining
• Enough information -- not too little, not too much.
• Color/Colour indicating importance of information, as well as status (queued/pending, success, in progress, failure, error)

Progress Data Model

Because peace is intended to support:

• interactive command line
• build logging
• machine readable output
• web request output

Then:

• The progress information needs to be serializable.
• It should contain enough information to be rendered.
• On a web page, detail may be collapsible, so including more information may be okay to avoid additional round trip time.

Bonus

For a CLI process / web request that is executed separately to the process that is running the command execution, the progress information needs to be retrievable independently. So:

• Progress information needs to be retrievable by a separate process -- perhaps written to / read from disk, or exposed on a well-known port.

Available Libraries

The libraries should ideally support async Rust with tokio, since that's what peace uses.

• prodash: Good for CLI usage and build logging, and uses tokio.

• indicatif: Good for CLI usage, spawns its own threads. This is used in choochoo / ops_ux.

  This is compatible with tokio and would compile to WASM once console-rs/indicatif#504 is merged and released.

  Also, the CLICOLOR_FORCE=1 variable must be set if running the indicatif tests.

• zzz: Minimal progress bar, tokio ready.

• spinners: Nice looking spinners.

`States*DiscoverCmd` should be able to show progress bars

StatesCurrent and StatesDesired discovery can take a while to execute.

• It should be possible to show progress bars when these commands are executed.
• It should be possible to not show progress bars as well, because some discovery may be fast (e.g. if it is all local).

Because ApplyOpSpec does state current and state desired discovery, this will result in the progress bar needing to be filled from 0 to full, three times.

This issue does not cover layered progress bar styling where we have a "state current progress layer", "state desired progress layer", and "apply layer".

Item Spec initialization parameters

ItemSpecs may need to be initialized with parameters provided by the end user.

Thoughts:

1. Most likely the parameters should be flow specific, as we may not want to mandate the user to provide parameters for flows that they are not using. However, if one item spec is used in multiple flows, it also means re-entering those parameters each time a new flow is used. Need to work out a good trade off.

2. Actually it's better to request that all initialization parameters are specified up front and together. If we separate them by flow, then one may expect that publishing an artifact to artifact_repository_url will automatically allow it to be used in a subsequent flow that downloads the artifact. If the artifact_repository_urls are not consistent between flows, then it's a bad user experience / a surprise.

3. But then again, if we have a dev environment flow, and later a production environment flow, not all users will care about the initialization parameters of all flows.

4. So, we probably need both workspace specific, and flow specific parameters.

What we need:

• Parameters common to different flows. Defined by Workspace type parameter.
• Parameters specific to certain flows. Defined by ItemSpec associated type.

API Improvements / Code Maintenance

• Change WebStorage to take in &Paths instead of &strs.
• resources_type_state module should be resources::ts.

Provide types that will be inserted into Resources, instead of having a W<'op, Option<T>> and R<'op, Option<T>>
This requires &mut Resources, and while the graph is processing, we only have access to &Resources.

Rough idea:

use resman::Resource;

/// Something included inside a `Data` implementation.
#[derive(Debug)]
pub struct Insert<T> {
    tx: Sender<Box<dyn Resource>>,
}

impl<'borrow, T> Data<'borrow> for Insert<'borrow, T>
where
    T: Debug + Send + Sync + 'static,
{
    fn borrow(_resources: &'borrow Resources) -> Self {
        // no-op, as the ItemSpec will instantiate T.
    }
}

// T must be set before `Insert` is dropped?

impl<'borrow, T> Drop for Insert<'borrow, T>
where
    T: Debug + Send + Sync + 'static,
{
    fn drop() -> Self {
        self.tx.send(t);
    }
}

// Somewhere else in the framework.
// This still can't be inserted during a graph iteration.
let r = rx.receive().await;
resources.insert(t);

Refine `FullSpec` to suit framework use cases

To implement additional functionality in a sensible model, we need to refine the traits.

git status is closer to a diff command with summarized output. When managing an item, it is often useful to view the current state without comparison to any expected state. For the peace framework, separate commands to query current and desired states is a suitable divergence from the git command line interface -- close git parallels are git show $remote/main and git diff HEAD..$remote/main.

• Rename StatusFnSpec to StateNowFnSpec, StatusDesiredFnSpec to StateDesiredFnSpec.
• Add DiffCmd to return the difference between StateNow and StateDesired.
• Change EnsureOpSpec::check to also take in the logical state diff.

Rationale:

For a FullSpec's check, it may need Resources to be altered by a predecessor's exec_dry / exec.

• StatusCommand: For every FullSpec, run status, status_desired, and diff, then output diff.

  However, for FullSpec successors, if a predecessor has Some(Diff), it means the successor may not be able to run status, as it may require the output of the predecessor to exist, before it can run. e.g. a server must exist before you can check if the file exists on the server.

• DiffCommand: For every FullSpec, run status, status_desired, and diff, then output diff.

• EnsureCommand:

  For every FullSpec, run status, status_desired, check, and exec_dry/exec, before progressing onto the next dependent FullSpec.

  The functions must be run in heterogeneous instead of homogeneous batches because status from FullSpec2 may report a more detailed status -- FullSpec1::exec may have inserted a Resource that FullSpec2::status uses and detects a more fine-grained status.

Status command

StatusCommand runs the same functions run as DiffCommand, just less verbose output.

ServerLaunch:
  status: server doesn't exist
  status_desired: server exists
  diff: server is missing

AppTransfer:
  status: since server doesn't exist, cannot detect status, but can infer file doesn't exist
  status_desired: file is on server
  diff: file is not on server

AppUnpack:
  status: since file path doesn't exist, cannot detect status, but can infer app needs to be unpacked
  status_desired: file is unpacked
  diff: file is not unpacked

AppConfigure:
  status: since app installation directory doesn't exist, cannot detect status, but can infer app needs to be configured
  status_desired: app is configured
  diff: app is not configured

Ensure command

ServerLaunch:
  status: discovers server doesn't exist
  status_desired: server exists
  diff: server is missing
  check: server is missing, need to launch
  exec_dry: inserts example ip 1.2.3.4 into Resources
  exec: launches server, and inserts ip 1.2.3.4

AppTransfer:
  status: server exists, log in to check file state
  status_desired: file should be transferred to server
  diff: file is not on server
  check: file is not on server, need to transfer
  exec_dry: inserts AppPath("/example/path") into Resources
  exec: ~

AppUnpack:
  check: ~
  exec: ~

AppConfigure:
  check: ~
  exec: ~

Design `peace::fmt::Presentable` / `peace::fmt::Presenter`

For human readable output, it is desirable for consumers as well as the framework to be able to:

• output arbitrary types
• have them formatted by the output implementation
• without needing the output implementation to know the specific type that is being output.

The OutputWrite trait was written to handle different output formats – human-readable vs CI logging vs structured text – but not how to present the output.

Peace should provide two traits:

• OutputWrite: Maps between the information and the output format, e.g. human readable vs parseable.

  Examples:

  • CLI output: Writes progress as log messages (CI), or progress bar (interactive), or nothing (when piped).
  • Web output: Write JSON.
  • Native application: Update a UI component.

• OutputFormatter: Maps between the information and the display format, e.g. write this as an ID, or short text, or long text.

  Examples:

  • CLI output: Colour text with ANSI colour codes.
  • Web output: Create and style web elements.
  • Native application: Create and style UI components.

Current State

The OutputWrite trait has methods for:

• writing progress: setting up state, updating progress, and ending.
• writing states (current, desired, diff, etc.)
• writing error

These methods are specific to States, and if we add methods per type, it doesn't allow any arbitrary type to be formatted and written.

Desired State

To be usable with arbitrary information, OutputWrite should have methods to output different kinds of information. These information kinds are based on the purpose of the information, not on how they should be grouped or presented.

Information Kinds

• Progress: Information about the execution of automation.

• Outcome: Information that the automation is purposed to produce.

• Notes: Meta information about the outcome or progress -- informatives, warnings.

  These can be used to refine the automation.

For each information kind, OutputWrite should be able to:

• Write one or many of that information kind
• Reason over the parameters of that information, and potentially pass it to a formatter.

For structured output, all information should be serializable.

Presentation / Formatting

For human readable output to be understandable, the OutputWrite implementation should improve clarity by adding styling or reducing information overload. For this to work with arbitrary types, the OutputWrite needs additional hints to determine how to format the information.

Examples:

• An object may be presented as a list, and the type needs to define which fields that list is built from.
• When presenting a list of named items, the type needs to define both the name and the description, which allows the names to be styled differently to the descriptions.
• When presenting a large object, the density of information can be reduced through collapsible sections, and more detail displayed when the sections are expanded.

Implementation

To achieve this, we can:

• Define a peace::fmt::Presentable trait, analogous to std::fmt::Display

• Define a peace::fmt::Presenter struct, analogous to std::fmt::Formatter

• Presenter has methods to format:

  • short text descriptions
  • long text descriptions (e.g. always split at \ns)
  • names, e.g. always bold
  • lists of Presentables
  • groups, e.g. always collapsible, or presenter may choose to not display if level of detail is too high

• Implementors will impl Presentable for MyType. This can be made easier with a derive macro.

• Update OutputWrite to take in &dyn Presentable instead of concrete types, and the OutputWrite implementation can decide whether or not to delegate to Presenter for presentation information. e.g. a serializing output write may not need to.

Note: Structured output that is read by humans (e.g. prettified YAML or JSON) is not a peace::fmt::Presentable concern, but an OutputWrite parameter, as it is a standard format serialization parameter, not formatting hints that the output endpoint needs.

Instead of using &strs for what is presented, we could add type safety to:

• Enforce certain constraints, e.g. short descriptions must be one line, less than 200 characters
• For human readable output, instead of std::fmt::Display, types implement peace::fmt::Presentable trait where a peace::fmt::Presenter is passed in.

The type safety can be opt-in, e.g. allow &strs, but if using the type-safe wrappers, you get compilation errors when the constraints are not met.

Recursion

If the Presentable trait is recursive like Debug, then we need to make sure implementors understand that a "name" will always be styled as a name, unless one creates a wrapping type that does not delegate to the name's underlying Presentable implementation (just like Debug).

`StatesCurrentDiscoverCmd` and `StatesDesiredDiscoverCmd` should include an entry for each item spec, even if the state cannot be looked up

Steps to reproduce

1. Run StatesDiscoverCmd for a Flow, where an item spec's current / desired state cannot be looked up due to a predecessor state not being available.

Expected outcome

The number of states presented matches the number of item specs in the graph, and hence is always consistent.

Actual outcome

Item specs' whose state cannot be discovered are not listed:

<!-- 4 current states -->
1. `iam_policy`: does not exist
2. `iam_role`: does not exist
3. `web_app_download`: `azriel91/web_app/0.1.1/web_app.tar` non-existent, ETag not yet determined
4. `web_app_extract`: 0 files

<!-- 3 desired states -->
1. `iam_policy`: /dev policy should exist
2. `iam_role`: /dev role should exist
3. `web_app_download`: `azriel91/web_app/0.1.1/web_app.tar` containing 6492160 bytes, ETag: "0x8DAEA1931AF89EE"

The web_app_extract desired state is not listed as it cannot determine how many files are in the tar file which has not been downloaded.

Improve CLI progress bar user interface

Add the following to CLI progress bar display for better visuals:

• emojis to indicate status
• messages to show high level detail
• elapsed / remaining duration

Status command

What: Be able to define a graph of FullSpecs, and run $app status to see the state of each full spec's item.

Consider:

• Should the state be serialized and stored somewhere? Fetching the state from a remote managed item may not be cheap.
• If so, track the time that we last fetched the state.

Code Maintenance / Simplification

Some outstanding code clean up:

• Collapse StateTypeRegs into StateTypeReg.

  Previously the serialized types for current state and desired states were different -- current state used State<Logical, Physical>, and desired state used just Logical. This was consolidated in #69 and #70.

• (maybe) Move peace_rt_model_core::params::* to peace_cmd::params

  rt_model_core seems like a catch all crate. Currently params reside in there because they were initially created there, and WorkspaceInitializer from rt_model_native and rt_model_web have functions to serialize them. Perhaps we don't need the parameters for serialization to be strongly typed.

• (maybe) Differentiate between StateCurrentDiffs and StateSavedDiffs.

  StateDiffs can be computed between StatesDesired - StatesCurrent, or StatesDesired - StatesSaved.

  If #86 is implemented, perhaps we don't have to do this.

Update dependencies and use workspace dependencies

Maintenance ticket to:

• Update dependencies to their latest versions. This also includes updating dependencies of smaller libraries peace depends on.
• Move all dependencies to the workspace Cargo.toml. This doesn't include examples' dependencies, so that it is easier for automation tool developers to see what versions of libraries are being used.

Check that `StatesSaved` matches `StatesCurrent` before running `EnsureCmd::exec` / `CleanCmd::exec`

Similar to git push, an apply command should not alter state if the user made the decision based on stale saved state.

Perhaps do this after #67, so it only has to do be done once.

Use single `State` associated type in item spec

Originally, logical and physical states were separated on the basis that implementors must distinguish the two concepts, and diff only happens on the logical state.

However, it is becoming apparent that:

• The separation of logical and physical state does exist, but certain things only have a logical state, and requiring State<Logical, Physical> everywhere is unnecessary complexity.
• Consistency of a single state type per item spec has the advantage of:
  • Not needing to go through the state.logical layer in each *Spec function
  • Serialization is consistent (always the same type)
  • Serialization is simplified (no additional layer as well)
• Implementors may still use State<Logical, Physical> when it makes sense for the item. We should make it easily discoverable in the documentation.

This should be done early since the effort that grows is large per new functionality added to the framework.

Workflow Task Graph: `WorkSpec` and related constraints

Provide traits / types that can be collected into a workflow task graph.

Whether or not to track operation support through an enum.

If an operation is not supported, should we track it via an enum, and require implementors to choose a variant? Maybe not -- it adds complexity to the API that may not be necessary in the common case.

If we had one, it may look like the following snippet:

#[derive(Debug)]
pub enum OpOutputMaybe {
    /// This operation is not supported.
    NotSupported {
        /// Message to display when this operation is run.
        ///
        /// This may be a link to an issue to explain why this is not supported.
        suggestion: Option<String>,
    },
    /// Nothing needs to be done for this to work.
    Unnecessary,
    /// This logic has not been written yet.
    ImplPending {
        /// Message to display when this operation is run.
        ///
        /// This may be a link to an issue to implement this operation.
        suggestion: Option<String>,
    },
}

Experimental WASM support

In order to have interactive examples on a website, we should compile to a WASM application.

Consolidate Item Spec functions into single trait

What

Currently ItemSpec has a number of associated types to split up the logic into different trait implementations.

Consolidating this into the top level ItemSpec trait should reduce the cognitive load on maintainers for:

• Keeping the data associated types in sync
• Reducing the number of traits to learn and track.

Details

Current traits

pub trait ItemSpec: DynClone {
    // Data types
    type Error: std::error::Error;
    type State: Clone + fmt::Display + Serialize + DeserializeOwned;
    type StateDiff: Clone + fmt::Display + Serialize + DeserializeOwned;

    // Logic types
    type StateCurrentFnSpec: TryFnSpec<Error = Self::Error, Output = Self::State>;
    type StateDesiredFnSpec: TryFnSpec<Error = Self::Error, Output = Self::State>;
    type StateDiffFnSpec: StateDiffFnSpec<Error = Self::Error, State = Self::State, StateDiff = Self::StateDiff>;
    type EnsureOpSpec: EnsureOpSpec<Error = Self::Error, State = Self::State, StateDiff = Self::StateDiff>;
    type CleanOpSpec: CleanOpSpec<Error = Self::Error, State = Self::State>;
}

pub trait TryFnSpec {
    type Output;
    type Data<'op>: Data<'op>
    where
        Self: 'op;
    type Error: std::error::Error;

    async fn try_exec(data: Self::Data<'_>) -> Result<Option<Self::Output>, Self::Error>;
    async fn exec(data: Self::Data<'_>) -> Result<Self::Output, Self::Error>;
}

pub trait StateDiffFnSpec {
    type State: Clone + Serialize + DeserializeOwned;
    type StateDiff;
    type Data<'op>: Data<'op>
    where
        Self: 'op;
    type Error: std::error::Error;

    async fn exec(
        data: Self::Data<'_>,
        state_current: &Self::State,
        state_desired: &Self::State,
    ) -> Result<Self::StateDiff, Self::Error>;
}

pub trait EnsureOpSpec {
    type Error: std::error::Error;
    type State: Clone + Serialize + DeserializeOwned;
    type StateDiff: Clone + Serialize + DeserializeOwned;
    type Data<'op>: Data<'op>
    where
        Self: 'op;

    async fn check(
        data: Self::Data<'_>,
        state_current: &Self::State,
        state_desired: &Self::State,
        diff: &Self::StateDiff,
    ) -> Result<OpCheckStatus, Self::Error>;

    async fn exec_dry(
        ctx: OpCtx<'_>,
        data: Self::Data<'_>,
        state_current: &Self::State,
        state_desired: &Self::State,
        diff: &Self::StateDiff,
    ) -> Result<Self::State, Self::Error>;

    async fn exec(
        ctx: OpCtx<'_>,
        data: Self::Data<'_>,
        state_current: &Self::State,
        state_desired: &Self::State,
        diff: &Self::StateDiff,
    ) -> Result<Self::State, Self::Error>;
}

pub trait CleanOpSpec {
    type Error: std::error::Error;
    type State: Clone + Serialize + DeserializeOwned;
    type Data<'op>: Data<'op>
    where
        Self: 'op;

    async fn check(data: Self::Data<'_>, state: &Self::State)
    -> Result<OpCheckStatus, Self::Error>;
    async fn exec_dry(data: Self::Data<'_>, state: &Self::State) -> Result<(), Self::Error>;
    async fn exec(data: Self::Data<'_>, state: &Self::State) -> Result<(), Self::Error>;
}

Consolidated trait

pub trait ItemSpec: DynClone {
    // Data types
    type Error: std::error::Error;
    type State: Clone + fmt::Display + Serialize + DeserializeOwned;
    type StateDiff: Clone + fmt::Display + Serialize + DeserializeOwned;

    async fn state_current_try(data: Self::Data<'_>) -> Result<Option<Self::Output>, Self::Error>;
    async fn state_current(data: Self::Data<'_>) -> Result<Self::Output, Self::Error>;
    async fn state_desired_try(data: Self::Data<'_>) -> Result<Option<Self::Output>, Self::Error>;
    async fn state_desired(data: Self::Data<'_>) -> Result<Self::Output, Self::Error>;
    async fn state_diff(
        data: Self::Data<'_>,
        state_current: &Self::State,
        state_desired: &Self::State,
    ) -> Result<Self::StateDiff, Self::Error>;

    async fn ensure_check(
        data: Self::Data<'_>,
        state_current: &Self::State,
        state_desired: &Self::State,
        diff: &Self::StateDiff,
    ) -> Result<OpCheckStatus, Self::Error>;

    async fn ensure_exec_dry(
        ctx: OpCtx<'_>,
        data: Self::Data<'_>,
        state_current: &Self::State,
        state_desired: &Self::State,
        diff: &Self::StateDiff,
    ) -> Result<Self::State, Self::Error>;

    async fn ensure_exec(
        ctx: OpCtx<'_>,
        data: Self::Data<'_>,
        state_current: &Self::State,
        state_desired: &Self::State,
        diff: &Self::StateDiff,
    ) -> Result<Self::State, Self::Error>;

    async fn clean_check(data: Self::Data<'_>, state: &Self::State)
    -> Result<OpCheckStatus, Self::Error>;
    async fn clean_exec_dry(data: Self::Data<'_>, state: &Self::State) -> Result<(), Self::Error>;
    async fn clean_exec(data: Self::Data<'_>, state: &Self::State) -> Result<(), Self::Error>;
}

See also #67.

Cross-profile / cross-flow commands

What

In #80, in order to list profiles and a profile parameter, non-trivial code was written in the app_cycle example.

In addition, a lot of type parameters were introduced to the CmdContext type as the keys of the *Params type registries. This makes passing a CmdContext as a parameter noisy – as there are 6 type parameters and one lifetime parameter.

Improvement

Make it easy to do the following:

• Workspace-wide aka cross-profile commands.

  • List all profiles – iterate over directories.
  • List all profiles and a particular profile parameter – load profile params.
  • List execution history for all profiles (for write-flows).
  • List execution history for specific profiles.

• Cross-profile-flow commands – still a workspace-wide command.

  • List all profiles' flow parameter for a particular flow.

• Profile-wide aka cross-flow commands.

  Not sure if there are any.

• WorkspaceParams, ProfileParams, and FlowParams' types must be specified, if they are to be deserialized.

• Notably, ProfileParams and FlowParams may be different for different profiles and flows.

  If they are different, then it makes it impossible to deserialize them for a given CmdContext. We could constrain the params types to be a superset of all profile/flow params, which essentially is making them the same umbrella type.

  This should be feasible for ProfileParams, as profiles are intended to be logically separate copies of the same managed items. Production profiles may require more parameters, but the parameter type can be the same.

  However, FlowParams being different per flow is a fair assumption. This means cross profile inspections of the same flow is achievable -- the same FlowParams type and ItemSpecGraph can prepare the TypeRegistries to deserialize the FlowParamsFile, StatesSavedFile, and StatesDesiredFile.

• A Profile is needed when there are ProfileParams to store, as it is used to calculate the ProfileDir to store the params.

• A FlowId is needed when there are FlowParams to store, or an item spec graph to execute, as it is used calculate the FlowDir to store the params, or read or write States.

• You should be able to list profiles, read profile params, and list flows, without needing to have either a profile or a flow.

• For States from all flows to be deserializable, there must be a type registry with all item specs' State registered. This is a maintenance cost for implementors, but unavoidable if that kind of functionality is desired.

Tasks

• Extract ParamsTypeRegs to store TypeReg<*ParamsK, BoxDt>.

• Change CmdContext to have ItemSpecGraph iff a FlowId is given. i.e. A FlowId is strictly linked to an ItemSpecGraph.

• Change CmdContext to not require a Profile, but a Profile is necessary if a flow command is to be executed, i.e. when an ItemSpecGraph is provided.

  This means either:

  • Certain *Cmds will only accept CmdContext when CmdContext has a Profile specified. Can use type state on the CmdContext type.
  • *Cmds take in profile in their exec function.

  Other commands that are cross-profile will be one of:

  • Within the Cmd, an iteration over profiles, then execute the logic.
  • An iteration over profiles, and execute the Cmd for that profile.

`ApplyCmd` should only execute if `StatesPrevious` and `StatesCurrent` are in sync

Before ApplyCmd runs ApplyOpSpec::exec, it should compare StatesCurrent and StatesPrevious, and if they aren't in sync, notify the user to discover states first.

This is similar to git push failing if the local branch head does not match the remote branch head.

Parameter Limit Checking

Any input / data that is reasoned upon should be sanitized, so that non-sensible values are not operated on.

The peace framework should guide implementors to define data types that have safety limits, and make it easy to do so. The framework can then use those limits to warn users when limits have been crossed.

• Limit categories / thresholds:

  • invalid (e.g. -1 users)
  • safe / common values
  • questionable / "trust me, I know what I'm doing"
  • technically unfeasible (e.g. 5 TB RAM)

• Warn users when using questionable values, unless they pass in a flag to silence the warning.

• See flux

Feature gated so that framework implementors may opt-in to implementing this.

Rename crate to `peace`

zzzz is slow to type, and peace is representative of the state we aim to produce through the framework.

Luckily the crate name isn't taken.

Split `ItemSpecParams` from `FlowParams`.

#94 is the work to allow users to specify values or references to values in item spec parameters.

As a stepping stone, we should split flow params and item spec params, so that:

• Item spec parameters are treated as a first class concept, so it is easier to see what users need to provide as input to an item spec.
• Peace can serialize the parameters, and provide a better diff experience.
• Users do not need to register the parameters as flow params, for them to be serialized.

Warn developers when two cmd params of the same type are inserted

Since Resources is a type map, when a cmd param (workspace_param, profile_param, flow_param) is inserted, params of the same type will overwrite a previous entry. This may or may not be desired, so warning every time is noise.

Desired overwrites:

• Saved params overwritten by newly provided params
• Profile / Flow param being more specific than Workspace param.

Undesired overwrites:

• Profile / Flow param using the same type as a Workspace param.

Maybe there is no software solution.

Found during writing tests that assert on workspace params / flow params insertion.

---

Current insertions are in CmdCtxBuilder::*_params_insert. Search for resources.insert_raw.

This may also apply to peace_code_gen::cmd::impl_build.

Replace `EnsureOpSpec` and `CleanOpSpec` with `ApplyOpSpec`

The ensure and clean operations are the same "apply" logic, with the target state either being the desired state, or the cleaned state.

Peace is intended to be able to support a set of target states such as "an empty set of servers", e.g. reusing servers for testing installation of software. This means arbitrary target states for each item -- "desired state for items a and b, but cleaned state for item c".

This may include some support to model "defined state" and "clean state" as top level concepts which are currently captured as separate commands.

• As part of this, move peace_core::progress to peace_cfg::resources.

Use `etag` in FileDownload item spec.

Use etag to determine if a file needs to be redownloaded.

This will enable the app_cycle example to have idempotent downloading.

Return consumer provided error type `E` in fallible functions

We should decide whether the user's error is the top level error type, which means we add the E: From<FrameworkError> types for each framework error.

or,

We only return framework error types, and framework errors implement From<E>.

Either option still needs E to implement some diagnostic trait that we can output nicely.

I think the first option is cleaner, because if E is a single type, then consumers receive back their own type whenever something errs, which, when the framework is stable, should be one of:

• The consumer's logic threw an error, so we want to propagate that back to them
• Incorrect usage of framework logic (we should try and reduce this at compile time by using types + type states to enforce proper usage)

It also doesn't make sense for framework errors to implement From<E> where E is never reachable from that particular framework function call.

Profiles / Environments / `remote`s

For the purpose of the peace framework, we should decide on the term "profile" or "environment", instead of "remote".

For many script-like automations like "download this file and run it", if we want to know the state of our downloaded file, we don't normally care about different profiles / environments. In that scenario we only care about:

• Here's my URL
• Here's the destination path
• Logic:
  • Is the file already downloaded?
  • Do the download.

There's no environment to connect to, so a git-fetch-like operation doesn't make sense -- in other words, the local computer is the default environment. Meaning if we want environments to be a general supported concept, it should be defaulted for workflows that don't need environments.

However, one may later decide "actually I want to run this for a different set of parameters", which is conceptually a different profile / environment.

Web Application Lifecycle Example

Create an example that shows the lifecycle of application development:

Deployment / Upgrade flow:

1. Build application.
2. Deploy n servers.
3. Upload configuration.
4. Clean servers.

Switch to new `CmdCtx`

Update the existing *Cmds to use peace_cmd::ctx::CmdCtx instead of peace_rt_model::cmd::CmdContext.

• Update *Cmds to use CmdCtx

• Update examples to use CmdCtx

• Delete peace_rt_model::cmd::CmdContext

• Move peace_rt_model_core::cmd_context_params::* to peace_rt_model_core::params

• ParamsTypeRegs should be in the scope, so can can remove the PKeys type parameter in CmdCtx.

• In SingleProfileSingleFlow, Flow should be borrowed so graph doesn't need to be cloned

• Improve CmdCtx API so resources can be reused, and output can be released.

  The goal isn't necessarily to extract output or resources, but to:

  • Reuse the cmd_ctx for subsequent Cmd invocations.
  • Assert certain state between Cmds.

  Perhaps this is better implemented as traits on Resources' type states.

  See URLO thread.

  Other options:

  • *Cmds return the data (e.g. StatesSaved) in the cmd return value.

    In this option, *Cmds don't mutate CmdCtx type state. *Cmd consumers are responsible for turning the CmdCtx into the higher TS by calling CmdCtx::resources_update.

  • Resources<HigherTS> can be turned back into Resources<LowerTS>, and optionally extract the resource that made it into the HigherTS.

  ℹ️ Info: Went with taking &mut cmd_ctx in *Cmds.

Web interface experiment

Experiment with a web interface. For now, the experiment doesn't have to be fully WASM compiled, but can be web requests to a backend server.

I really like leptos and dioxus, and the review (youtube) that Greg (creator of leptos) did on the different frameworks.

dioxus' syntax is a bit lighter; both have custom tools for building:

• dioxus: dioxus-cli
• leptos: cargo-leptos
• both use trunk to package for the web.

`DiffCmd` should take in states to diff.

Instead of DiffCmd always diffing StatesSaved and StatesDesired, it should take in the States to diff.

This will enable the following distinctly useful cases:

• diffing between state saved and state desired
• diffing between state current and state desired
• diffing between state currents of multiple profiles

Referential item spec parameter values

What

Item spec parameter values are currently specified before a flow's execution.

Users should be able to specify that item spec parameter values should be looked up from values generated during execution.

Notes

An idea on how to implement this is as follows:

1. Have the item spec params take in T.

   Derive a builder for the item spec params struct:

   • Consumers: When creating a flow and inserting flow params, call:

     ItemSpecParams::builder()
         .with_x("value")
         .with_x_ref::<T>()
         .with_x_from::<T, U>(|t| t.u())
         .build()

   • Implementors: For each field have a with_x, with_x_ref::<T>(), and with_x_from::<T, U>(|t| -> U {})

     #[derive(Params)]
     pub struct Params {
         /// ID of something generated.
         id_to_attach: String,
     }

   • Peace:

     1. Hold &mut Resources and insert States into it as they are finished.

     2. Or should we use the existing Resources? -- likely, some things write to W<'op, T>

     3. Whenever we run an any function exec (state current / ensure op spec exec / etcetera all apply), we take ItemSpec::Params, and run ItemSpecParamsBuilder::build_from(resources), which will use the static value / fetched and mapped values to return ItemSpecParams.

     4. Somehow that gets injected into the exec function -- not sure if we put a separate parameter whether that's nice or not, or if we can merge the ItemSpecParams type into the Data.

        Maybe instead of W<'_, Params>, we have a P<'_, Params'>, whose implementation does step 4.

     Still need to work out how to insert state into Resources. Do we silently include a W<'_, State> alongside every item spec's data writes? Maybe.

     params.fields()
         .iter()
         .map(|field_rt| field_rt.value(resources));
     
     impl FieldRt for Field<T, U> {
         fn fetch(resources: &Resources) -> Ref<'_, T> {
             match field {
                 FieldParam::Value(value) => value,
                 FieldParam::Ref => resources.borrow::<T>(),
                 FieldParam::From => {
                     let t = resources.borrow::<T>();
                     U::from(t)
                 }
             }
         }
     }

2. Then the item spec's data accessed includes R<'op, T> for data dependency calculation.

   1. Peace will take the ItemSpecParamsBuilder, figure out which fields are refs / froms.
   2. Peace will fetch the T refs from Resources, and turn the builder into ItemSpecParams and insert write to it in Resources.

3. Item spec functions will be given the params with the values read from Resources.

   ItemSpecs' data must have a Params<'_, ItemSpec::Params> field in it to access the params.

---

Design Thoughts

Is this too complicated, and people would just want to write a script to pass information between steps?

A process needs:

• Values / references for parameter values from the user
• Values generated by functions
• Transforming those values / references into flat values
• A way to pass the values between functions based on what was decided.

User

• Uses ParamsBuilder, defines param values, or refs / froms
• Uses ItemSpec, defines item spec IDs and params builder values.

Item Spec Implementor

• Define Params -- what a user needs to pass in.
• Define runtime Data, e.g. sdk::Client -- other logic / data used during execution.
• Functions take in Params, Data, ctx, and state

fn state_clean              (params, data) -> Self::State;
fn state_current    (fn_ctx, params, data) -> Self::State;
fn try_state_current(fn_ctx, params, data) -> Self::State;
fn state_desired    (fn_ctx, params, data) -> Self::State;
fn try_state_desired(fn_ctx, params, data) -> Self::State;
fn state_diff               (params, data, state_a, state_b) -> Self::State;
fn check            (fn_ctx, params, data, state_current, state_desired, state_diff) -> Self::State;
fn apply_dry        (fn_ctx, params, data, state_current, state_desired, state_diff) -> Self::State;
fn apply            (fn_ctx, params, data, state_current, state_desired, state_diff) -> Self::State;

The function signature is quite long.

Grouping data and params in another layer may be a hassle, but may be a sensible decision if there will be a fn_ctx object provided to every function call.

Maybe the type for the params argument should be a generated type, with Option for each field -- wherever there is a ref or from value, the referenced value may not necessarily be available. Or maybe it should all be Some, and Peace should return an error / panic when a value cannot be resolved, except for try_state_current and try_state_desired, it may legitimately not be present.

Experiment:

• state_clean: Path to file(s) to clean up may be provided by predecessor.

  match params.file_paths() {
      Param::Value(file_paths) => {}
      Param::NotAvailable => {}
  }

  Does the State's presentation need to have access to params, to output value? Perhaps not -- the params could have changed between creation of the item, and the presentation. So if the state should show a file path, then it should store it in the clean state.

• state_current: MD5 for remote file may not be available, if host IP is not available.

  match params.host_ip() {
      Param::Value(ip_address) => {}
      Param::NotAvailable => {}
  }

• state_desired: MD5 for file to upload may not be available, if file has not been downloaded.

  match params.file_path() {
      Param::Value(file_path) => {}
      Param::NotAvailable => {}
  }

If we group the fn_ctx, params, and data into the FnCtx, whereby the original fn_ctx may be empty, then the function signatures look like:

fn state_clean      (fn_ctx) -> Self::State;
fn state_current    (fn_ctx) -> Self::State;
fn try_state_current(fn_ctx) -> Self::State;
fn state_desired    (fn_ctx) -> Self::State;
fn try_state_desired(fn_ctx) -> Self::State;
fn state_diff       (fn_ctx, state_a, state_b) -> Self::State;
fn check            (fn_ctx, state_current, state_desired, state_diff) -> Self::State;
fn apply_dry        (fn_ctx, state_current, state_desired, state_diff) -> Self::State;
fn apply            (fn_ctx, state_current, state_desired, state_diff) -> Self::State;

Which looks cleaner; we have to make FnCtx generic, which adds a bit of mental load. Also, in code it doesn't just work as one FnCtx<'_> type -- *Cmds need to pass in a type erased FnCtx for the progress sender (and later on more contextual information) -- i.e. no generics for ItemSpec::Params or ItemSpec::Data can be on the type, so a subset of FnCtx could be passed in from *Cmd, and the fuller FnCtx with ItemSpec::Params and Data augmented inside ItemSpecWrapper. This creates a web of types.

Perhaps the flat 6 parameters is more easily grasped, even if messy.

Framework

• From ItemSpec::Params, define ParamsBuilder
• Maybe the type for the params argument should be a generated type, with Option for each field -- wherever there is a ref or from value, the referenced value may not necessarily be available. Or maybe it should all be Some, and Peace should return an error / panic when a value cannot be resolved, except for try_state_current and try_state_desired, it may legitimately not be present.

Update `states_saved.yaml` with physical states generated during `EnsureCmd::exec`

Currently (0.0.5) EnsureCmd discovers and saves states after executing EnsureOpSpec::exec.

An approach more resilient to interruptions is, instead of:

1. Ensure items 1, 2, 3
2. Discover item states 1, 2, 3

Switch this to:

1. Ensure item 1, discover state 1
2. Ensure item 2, discover state 2
3. Ensure item 3, discover state 3

This means if there is an interruption, or an item spec's ensure execution fails, the state(s) of previous items is already in memory, and can be saved. Also, if the connection to the internet breaks, then it may not be possible to run StatesCurrentDiscoverCmd to discover it again.

An implication of this is, StateCurrentFnSpec::Data should be a subset of EnsureOpSpec::Data (the framework could enforce it to be the same type).

Replace `check` function with separate operation to fetch desired state.

The status operation retrieves the current state.

If we have a corresponding operation to retrieve the desired state, then the check function can simply be a diff between the two statuses, and pulled into the framework code.

Perhaps the check function should still be able to be implemented, in case consumers want to override the default diff-as-work required behaviour.

`DiffCmd` common use-case functions

In #101 the DiffCmd::exec was updated to take in flow, output, states_a, states_b, as the states that are compared may not necessarily be current vs desired, but current states between multiple profiles.

This issue is to re-add functions so implementors can call DiffCmd with cmd_ctx, for the state diffs that they want, without having to run States*ReadCmd / States*DiscoverCmd themselves.

Recursive `Params` resolution

Follow up / improvement to the design of referential item spec parameters (#94).

Every Params is resolved through a ParamsSpec, not just its fields, which
is closer to our current ValueSpec:

Something like this, maybe generated for each type:

enum ValueSpec<T> {
    /// Loads a stored value spec.
    ///
    /// The value used is determined by the value spec that was
    /// last stored in the `params_specs_file`. This means it
    /// could be loaded as a `Value(T)` during context `build()`.
    ///
    /// This variant may be provided when defining a command context
    /// builder. However, this variant is never serialized, but
    /// whichever value was *first* stored is re-loaded then
    /// re-serialized.
    ///
    /// If no value spec was previously serialized, then the command
    /// context build will return an error.
    Stored,
    /// Uses the provided value.
    ///
    /// The value used is whatever is passed in to the command context
    /// builder.
    Value(T),
    /// Uses a value loaded from `resources` at runtime.
    ///
    /// The value may have been provided by workspace params, or
    /// inserted by a predecessor at runtime.
    From,
    /// Uses a mapped value loaded from `resources` at runtime.
    ///
    /// The value may have been provided by workspace params, or
    /// inserted by a predecessor at runtime, and is mapped by the
    /// given function.
    ///
    /// This is serialized as `FromMap` with a string value. For
    /// deserialization, there is no actual backing function, so
    /// the user must provide the `FromMap` in subsequent command
    /// context builds.
    FromMap(Box<dyn MappingFn>),
    /// Resolves this value through `ValueSpec`s for each of its fields.
    ///
    /// Rather than searching for
    FieldWise {
        // The top level spec is expected to be on the heap,
        // so we don't box these fields.
        field_1: ValueSpec<Field1>,
        field_2: ValueSpec<Field2>,
    },
}

trait ValueSpecRt {
    ///
}

impl<T> ValueSpecRt for ValueSpec<T> {
    //
}

Params Resolution

1. from_map may need to return Optional if the borrowed resource has a None value.

2. Resolved values make sense when the flow is executed forward, but for clean up, we may not have inserted Current<IS::State> of a predecessor, when cleaning up a successor. This means trying to resolve a value may not make sense for Apply?

   or perhaps we should run try_state_current for all states first.

3. from_map should allow borrowing multiple values from Resources, because users may combine outputs from different item specs to compute the value.

Remember:

• #122
• In ItemSpecWrapper::apply_check(), a params partial not being turned into a Param may still need apply to be run -- for CleanCmd, just because the params cannot be fully resolved doesn't mean you don't need to clean.

Multiple Init Params Support

Currently WorkspaceInit, ProfileInit, and FlowInit are single types.

Implementors may wish to take in init parameters that are relevant to different ItemSpecs, and so each init parameter needs to be able to be mapped to multiple different data types and inserted as resources.

Use cases:

• Parameters from the user during CLI usage -- these should be serialized and stored, and deserialized for subsequent commands.
• Tests may want to insert params for each item spec -- not necessarily to be serialized, but doesn't hurt if they are.

Options:

• Store a TypeMap per workspace / profile / flow, then move the resources across to the CmdContextBuilder resources map.

  We need to be able to deserialize parameters from the *_init.yaml files, so taking in arbitrary with_*_init(..) doesn't work.

• Still take a single type per workspace / profile / flow, but also take a mapper that maps from that init param type to different data types and inserts them into the resources map.

• Also have with_resource(..) methods in CmdContextBuilder for things that don't need to be serialized/deserialized -- saves people from needing to access Resources after building the context and then insert.

Document Purpose and Concepts

Document the purpose of the peace framework, and technical concepts used in the solution design.

User friendly output on the command line - first cut

The CliOutput currently serializes StatesCurrent, StatesDesired, and StateDiffs when output to the command line. The serialized output can be very verbose and difficult to take in at a glance, especially for flows with large numbers of ItemSpecs.

Improvements that can be made:

• Single line output per ItemSpec state. Possibly a TypeMap that constrains values to implement Display (#37).
• Colourized output. Maybe use console -- this is what indicatif uses, and we may use that for writing progress (#38).
• CliOutputFormatter to toggle between single line vs serialized form (#39).
• Confirm compatibility with miette::Diagnostic (#40).

---

Random snippet for writing each item spec's state on a single line.

let stdout = &mut self.stdout;
stream::iter(states_current.iter())
    .map(Result::<_, E>::Ok)
    .try_fold(
        stdout,
        |stdout, (item_spec_id, _item_spec_state)| async move {
            stdout
                .write_all(item_spec_id.as_bytes())
                .await
                .map_err(Error::StdoutWrite)?;
            stdout.write_all(b": ").await.map_err(Error::StdoutWrite)?;
            // stdout.write_all(/* state */).await?;
            Ok(stdout)
        },
    )
    .await?;

Clean Command

Provide the CleanCmd, and update example to use it.

Shell Command Item Specs

Shell commands are quick ways to experiment setting up an item spec.

If users may define an item spec's functions using shell commands, then they may be able to prototype a flow quicker, and translate the item spec into a proper Rust item spec later on.

There are different ways shell commands can be managed that are desired by consumers:

• Run the command if the last arbitrary recorded state is different to an arbitrary test condition.
• Run the command if its last execution is earlier than some source file modification time.

These would be different item specs, depending on when the ensure exec is intended to be run.

For this issue, just implement the first, as the second technically could be implemented by an implementor in the shell commands.

User friendly output on the command line - second cut

Part two of use friendly CLI output (part one -- #28).

• Refactor Error to consolidate all State*Deserialize and Serialize errors into a single variant.

• Specify diagnostic source code, spans, and help messages for each variant.

• Update *Cmds and tests to write errors to OutputWrite -- maybe not? Depends if the Cmd is meant to be the one writing.

• Update InMemoryTextOutput for the web to display miette reports.

• Update book documentation on how to use miette.

  The download example uses a workaround to show inner diagnostics to get around the pass-though not working issue:

  • zkat/miette#170
  • zkat/miette#172

Naming and API Refinements: Current / Desired (Goal) States and Commands

In 0.0.10 and earlier:

• StatesCurrent is stored as states_saved.yaml, and StatesDesired is stored as states_desired.yaml.
• It is a surprise / hard to discover when one wants to read the previously stored StatesCurrent, and has to remember to use StatesSavedReadCmd instead of StatesCurrent..Cmd.
• It is additional effort when one has StatesCurrent, and needs to pass in StatesSaved to EnsureCmd -- they have to map the data type (related: #59).
• "desired" doesn't capture the meaning that that is the state when the automation ensures.

To address these:

• "desired" should be renamed to "goal".

  • StatesDesired*Cmd renamed to StatesGoal*Cmd
  • states_desired.yaml renamed to states_goal.yaml
  • Update docs.

  Name bikeshedding:

  • leave as is
  • built / created (some things aren't built / created, and a bit obscure)
  • complete / finished this? (may confuse user that it is "done")
  • done (overloaded / hard to search / confusing)
  • formed (overloaded term)
  • forged (obscure)
  • goal this? (semantic overlap with "target"?)
  • manufactured (too long)

• Rename StatesSaved* to StatesCurrentStored.

• Add StatesGoalStored* to distinguish between StatesGoal and what was saved.

• Make sure it is easy to go from States*Stored to States*.

---

Out of scope:

• Rename DiffCmd to StatesDiffCmd (?) -- what about diffing parameters or specs.

• Store state for each item, even if it was not discovered.

  • Current states.
  • Goal states.
  • State diffs.

  Fixes the ordering in stored files.

• StatesDiffCmd API to make it obvious whether we are diffing:

  Single profile, single flow:

  • Discovered current and discovered goal states.
  • Discovered current and discovered goal states.
  • Saved current and discovered goal states.
  • Discovered current and saved goal states.

  Multi profile, single flow:

  • Current states of both profiles.
  • Goal states of both profiles.

  Arbitrary two states.

In `CmdCtxBuilder::build()`, error on mapping fn not provided

• *Spec::Stored should use the passed in *Spec::* value.
• *Spec::{Value, InMemory, MappingFn} should not be overwritten.
• ParamsSpec::Fieldwise should perform a fieldwise merge.

Consider changing `StatesDesired` to use `State<Logical, Physical>`

Currently:

• StatesDesired is conceptually a Map<ItemSpecId, StateLogical>.
• StatesCurrent is conceptually a Map<ItemSpecId, State<StateLogical, StatePhysical>>.

The original rationale for this design is to model logical state as something a consumer / user can define or declare, whereas physical state is computed and cannot be known until actual execution.

The downsides to this approach are:

• Not simple to visually compare both states_current.yaml and states_desired.yaml.

• The shape of state comparisons cannot be treated uniformly.

  Difference between states.get::<State<StateLogical, StatePhysical>, _> and states_desired.get::<StateLogical, _>:

  let states = resources.borrow::<StatesCurrent>();
  let state = states.get::<State<StateLogical, StatePhysical>, _>(&item_spec_id);
  let states_desired = resources.borrow::<StatesDesired>();
  let state_desired = states_desired.get::<StateLogical, _>(&item_spec_id);

This has caused at least one mix up where I tried to retrieve a StateLogical from StatesPrevious, where I should've been requesting a State<StateLogical, StatePhysical>.

---

Part of fixing this:

• Update docs to indicate that StatesSaved is strictly speaking the re-read values from states_current.yaml which may have gone out of date
• Ensure that StatesCurrent is present iff the current states were discovered in the exact current execution. Change StatesCurrentReadCmd to StatesPreviousReadCmd.
• StatesDeserializer should be refactored to deserialize StatesSaved and StatesDesired.
• peace::rt_model_*::Error should use a common variant for States*Deserialize and States*Serialize.

Resilience when discovering current and desired states

Sometimes it is not possible to discover StateCurrent or StateDesired. Examples:

• Current state: Unable to discover the content hash of a file on a server, if the server doesn't exist -- i.e. would be launched by a predecessor item spec.
• Desired state: Unable to discover the content hash of a file locally, if the file needs to be created -- e.g. executable needs to be compiled from source, or archive file needs to be created.

This ticket is to change peace to do one of the following:

• StateCurrentFnSpec and StateDesiredFnSpec should only be run for item specs whose predecessor's EnsureOpSpec::check returns OpCheckStatus::ExecNotRequired, or
• Change the signature of StateCurrentFnSpec::exec and StateDesiredFnSpec::exec for implementors to return whether StateCurrent could/could not be discovered (more effort for implementors, but potential for useful information to be returned).

Background: State Discovery Constraints

(taken from logical_state.md)

In an item spec's parameters, there must be the following categories of information:

• src: information of what the item should be, or where to look up that information.

  Thus, src is a reference to where to look up state_desired.

• dest: reference to where the actual item should be.

  dest is a reference to where to push state_current.

Both src and dest may reference resources that are ensured by predecessor item specs. Meaning sometimes state_desired and state_current cannot be discovered because they rely on the predecessors' completions.

Examples

• A list of files in a zip file cannot be read, if the zip file is not downloaded.
• A file on a server cannot be read, if the server doesn't exist.
• A server cannot have a domain name assigned to it, if the server doesn't exist.

Implications

• If dest is not available, then state_current may simply be "does not exist".

• If src is not available, and we want to show state_desired that is not just "we can't look it up", then src must be defined in terms of something readable during discovery.

• If that is not possible, or is too expensive, then one or more of the following has to be chosen:

  1. StateDesiredFnSpecs have to always cater for src not being available.

     It incurs mental effort to always cater for src not being available – i.e. implementing an item spec would need knowledge beyond itself.

  2. the peace framework defaults to not running state_current_fn_spec for items that have a logical dependency on things that EnsureOpSpec::check returns ExecRequired

     For this to work, when StateCurrentFnSpec::exec is requested, peace will:

     1. For each non-parent item, run StateCurrentFnSpec, StateDesiredFnSpec, StateDiffFnSpec, and EnsureOpSpec::check.
     2. If EnsureOpSpec::check returns OpCheckStatus::ExecNotRequired, then successor items can be processed as well.

  3. StateCurrentFnSpec could return Result<Option<Status>, E>:

     • Ok(None): State cannot be discovered, likely because predecessor hasn't run

     • Ok(Some(State<_, _>)): State cannot be discovered.

     • Err(E): An error happened when discovering state.

       May be difficult to distinguish some cases from Ok(None), e.g. failed to connect to server, is it because the server doesn't exist, or because the address is incorrect.

       Should we have two StateCurrentFnSpecs? Or pass in whether it's being called from Discover vs Ensure – i.e. some information that says "err when failing to connect because the predecessor has been ensured".

  Option 2 may be something we have to do anyway – we will not be able to provide current state to run EnsureOpSpec::exec for successors for the same reason.

  Option 3 may coexist with option 2.

  Note: State discovery may be expensive, so it is important to be able to show a saved copy of what is discovered.