ls1intum/aeolus

aeolus's Issues

Define JSON schema for AF and action.yaml

Results from weekly meeting on the 07.08.2023

What is defined here?

Preliminary language structure, we have actions (clone, script [build, ], etc.) that are part of a job. The actions are defined in so-called, for now, AeolusFiles and can include community-developed actions that are open-source and usable across the system.

What are the different files for?

AeolusFile

Purpose

Contains several actions that should be executed for an exercise

Defines

Actions that are executed at various points in the lifecycle of an exercise

Needs

• version: for API compatibility
• author: bragging rights/accountability and general contact person if something is not working as expected
• all actions need to be compatible with version specified in file

How is the lifecycle subdivided?

1. preRelease || prepare
2. workingTime
3. postDeadline
4. evaluation

Action

Defined/written by community. We provide the definition and guidelines, and some basic stuff (clone/compile actions) to build basic jobs.

Purpose

• appends stuff to the resulting job
• or does stuff that is needed for the resulting build plan

Defines:

• code that is executed upon generation of job
• code that is added to the job and executed in the CI system during a triggered build

Adheres to:

• schema that is set in Aeolus spec

Idea:

defines particular steps that are part of the final result or prepare something in the CI system.
Examples:

• In Jenkins, add strings to a given prefix and return it, e.g. ad definition of a step generating the code for the pipeline
• In Bamboo, curl Rest API of Bamboo to prepare CI system for eventual addition of the build plan
• In CLI, install packages and append your commands to a given prefix, generating the bash script

Potential problems (that I see right now):

• dependency management: if requirements.txt -> installation is not always fast (e.g. building of wheels takes time)

Investigate Bamboo maven plugin used in Artemis

Check if it's still not possible to generate build plans from yaml

Handle environment variables

essentially all CI solutions make use of env variables, figure out a way to map those using Aeolus.

Idea: Gather a list of all default environment variables for the target systems and map these to our new system

Research Bamboo REST-API

Check if PUT request is now possible, check if the web crawler for editing is obsolete.

AC:

• we know if one can now easily edit build plans using the REST API

Integrate Bamboo

AC:

• bamboo build plans can be created in Bamboo
• the build plans run as expected

Build scripts using a template engine

Motivation

Currently, the Bash scripts that are created from the Windfiles are ‘templated’ by manually adding them line-by-line to a result string. This has multiple disadvantages:

• It is repetitive: add_line() needs to be called for every line. Each time the indentation has to be specified.
• It is potentially error-prone and/or hard to understand:
  • The indentation has to be kept in mind for all commands.
  • Escaping in strings with special characters like \ and quotation marks.
  • It is hard to see the overall concept of a block of the script, since each line is always made up of some Python code + the actual line that will appear in the Bash script.

Proposed solution

A template engine like for example Jinja2 would allow to create the scripts as templates with placeholders that can be filled in. It also allows for repeated blocks and blocks that are only ‘rendered’ (i.e., added to the script) based on some condition.
Especially when combined with a TypedDict that holds the variables passed into the template, this allows for script templates that are easier to maintain.

Multiple smaller template files could be used and then be combined into the whole script.

More complex individual shell commands that are built from multiple conditional parameters could still be built in Python code and then passed in as a whole to the template to not have to deal with too many optional parameters in the template.

Example

Approximate translation of

Aeolus/cli/generators/cli.py

Line 58 in 789bcb0

def add_postfix(self) -> None:

into a template.

main() {
{% if needs_lifecycle_parameter %}
  local current_lifecycle="${1}"
{% endif %}

{% if needs_subshells %}
  if [[ "${1}" == "aeolus_sourcing" ]]; then
    # just source to use the methods in the subshell, no execution
    return 0
  fi
  local _script_name
  _script_name=$(realpath "${0}")
{% endif %}

{% if has_always_actions %}
  trap final_aeolus_post_action EXIT
{% endif %}

{% for function in functions %}
  {% if needs_subshells %}
  bash -c "source ${_script_name} aeolus_sourcing; {{ function.name }} {{ function.parameter }}"
  {% else %}
  {{ function.name }} {{ function.parameter }}
  {% endif %}

  {% if has_multiple_steps %}
  cd "{{ initial_directory_variable }}"
  {% endif %}
{% end %}
}

main "${@}"

Additional considerations

Jinja2 is just one example of a template engine I am somewhat familiar with. Other template engines might exist that are better suited for this concrete task.

The proposed refactoring is not essential to provide the core feature set of Aeolus. I would consider this as a possible nice-to-have to ensure long-term maintainability of the tool.