kazadhum/file_intro_plugin

file_intro_plugin's Issues

Further suggestions

Hello @miguelriemoliveira, @rarrais and @MisterOwlPT!

I've got a working plugin now, and I was wondering if you had any suggestions. I wrote the plugin to be used for anything, so I don't mention ATOM or calibration specifically. The way I have it now, the plugin assumes the results file is a .csv file (should I expand this to other file formats?). The Rigelfile looks like this.

As you can see, it receives the row and column of the value to be used to evaluate the results, as well as thresholds for the various levels of quality. Finally, it receives a required level of quality, or how good the results need to be to 'pass' the test.

What I wanted to ask is if this seems right to you and, furthermore, if you have any suggestions.

Result File

Hi @rarrais and @MisterOwlPT! I was wondering something that might come up as an issue later. So this plugin would use as input the result file from a calibration using ATOM in our containers. My question is, how do we get the csv file from inside the containers to use in this introspection plugin? Does Rigel include a tool to extract such a file?

Thanks in advance!

Plugin compliant with new features.

Hi @Kazadhum

I implemented core new features to the Rigel framework that will, among other things, support your work (Rigel now supports concurrent execution of multiple plugins and decouples introspection from infrastructure launching).

However, to take advantage of these new features you'll need to change some things in your plugin and the Rigefileyou are using. I leave here a detailed description of everything that changed for your reference (@rarrais included).

Please keep in mind the following:

• The new features are working but were not yet tested intensively. For this reason, I created a new branch flow. This way you'll always have the stable develop intact if something goes wrong and you need to revert. I suggest you do something similar with your work. Consider creating a branch for your updated plugin and associated Rigelfiles.

• Next week I'll be on holiday. I'll try to pay attention to GitHub issues and help you if required.

UPDATES TO RIGEL

NOTE: remember to use branch flow !

Rigel plugins

Changes to interface

The run function was renamed to start.

A new function process(self) -> None was introduced.

All other required functions remain the same.

During execution, the following plugin functions are now called in the following order:

• setup()
• start()
• process()
• stop()

Notice that process is always called after start.

Rigel plugins must be compliant with this new interface.

NOTE: Arguments to the plugin constructor also remain unchanged.

New function process()

Whenever applicable, this function acts as a complement to start and should contain logic that is either:

• user-dependent (built-in function input, keyboard events, ...)
• blocking or very long (while loops, time.sleep, ...)
• analysis of results

All other business logic should remain inside start.

This division allows for concurrent execution of plugins.
Depending on the chosen flow of execution the function process may or not be called.
Check the section on executors below for more information.

Data sharing mechanism

Plugins that are executed in sequence can now share data between themselves!

This is done via a class attribute shared_data of type Dict[str, Any]. This attribute is automatically passed between all plugins for them to store/read data according as required.

All plugins automatically have access to this attribute without the need for changes (you can use access it self.shared_data).

New plugin rigel.plugins.core.ComposePlugin

The existing plugin rigel.plugins.core.TestPlugin was broken into two plugins. One kept the original name and the other name was called _rigel.plugins.core.ComposePlugin_.

The new ComposePlugin is responsible only for launching a containerized ROS application (think of it as Docker Compose for ROS). After launching the containers it waits forever for user input (CTRL-C / CTRL-Z). Check the following Rigelfile excerpt:

# ...
plugin: "rigel.plugins.core.ComposePlugin"
with:
    components:
    -
        name: "simulation"
        image: "{{ AWS_ECR }}/simulation:latest"
        command: ["roslaunch", "tm_sim_gazebo", "run.launch", "gui:=true"]
        envs:
            DISPLAY: "{{ DISPLAY }}"
        volumes:
            - !!python/tuple ["/tmp/.X11-unix", "/tmp/.X11-unix"]
    -
        name: "robot"
        image: "{{ AWS_ECR }}/robot:latest"
        command: ["roslaunch", "robot_app", "run.launch", "use_rosbridge:=true"]
        envs:
            DISPLAY: "{{ DISPLAY }}"
        volumes:
            - !!python/tuple ["/tmp/.X11-unix", "/tmp/.X11-unix"]
        introspection: True  # <--- This component has a ROS bridge installed
# ...

If a component is declared with introspection: True the plugin will store the name of the container for that component in the shared plugin data (key simulation_hostname). This can be used by other plugins, if applicable. If required, check the plugin model.

The TestPlugin is now responsible only for the introspection of a containerized ROS system. It still requires ROS bridge. Check the following Rigelfile excerpt:

# ...
plugin: "rigel.plugins.core.TestPlugin"
with:
    requirements:
    - 'globally : some /OSPS/TM/HeartBeep [osps_msgs/TMHeartBeep] {publisherId = "friday"}'
    - "globally : some /OSPS/TM/TaskStatus [osps_msgs/TMTaskStatus] {statusCode = 5}"
# ...

If no field hostname or port were declared the plugin will automatically look for connection data inside the shared plugin data. If required check the plugin model.

Updated plugins

All plugins except rigel.plugins.aws.RoboMakerPlugin were updated, are compliant with the new protocol and are ready to use.

Executors, job sequences and execution flows

Until now Rigel was only capable of executing sequential job sequences (via the rigel run sequence command).

Now it supports the following execution flows:

• sequential
• concurrent
• parallel

Sequential job sequences

NOTE: this corresponds to the execution already supported by Rigel.

Consider the following Rigelfile excerpt where an example sequential job sequence example_sequence is declared:

# ...
sequences:
    example_sequence:
        stages:
            jobs: ["a", "b"]  # <-- One or more job ids
# ...

Assume that job a uses plugin PluginA and job b uses plugin PluginB.

Executing the command rigel run sequence example_sequence will trigger the execution of the following functions:

• PluginA.setup()
• PluginA.start()
• PluginA.process()
• PluginA.stop()

• PluginB.setup()
• PluginB.start()
• PluginB.process()
• PluginB.stop()

Summary: PluginA executes completely and then PluginB executes completely.

Concurrent job sequences

Consider the following Rigelfile excerpt where an example concurrent job sequence example_sequence is declared:

# ...
sequences:
    example_sequence:
        stages:
            jobs: ["a", "b"]
            dependencies: ["c", "d"]
# ...

Assume that job a uses plugin PluginA, job b uses plugin PluginB ... and vice-versa.

Executing the command rigel run sequence example_sequence will trigger the execution of the following functions:

• PluginC.setup()
• PluginC.start()
• PluginD.setup()
• PluginD.start()

• PluginA.setup()
• PluginA.start()
• PluginA.process()
• PluginA.stop()
• PluginB.setup()
• PluginB.start()
• PluginB.process()
• PluginB.stop()

• PluginC.stop()
• PluginD.stop()

Summary: PluginC and PluginD are partially executed (function process is never called). Before the stop function is called, PluginA and PluginB are completely executed and in sequence.

NOTE: Consider using this execution flow in your work. Use the new rigel.plugins.core.ComposePlugin to launch the containers you require and use your plugin to inspect their execution as required.

Parallel job sequences

Consider the following Rigelfile excerpt where an example parallel job sequence example_sequence is declared:

# ...
sequences:
    example_sequence:
        stages:
            -  # <-- Will be executed in a separate Thread_0
                jobs: ["a", "b"]

            -  # <-- Will be executed in a separate Thread_1
                jobs: ["c"]
                dependencies: ["d"]
# ...

Here, exceptionally, each stage declared within stages must consist of either a sequential or concurrent sequence of jobs.

Assume that job a uses plugin PluginA, job b uses plugin PluginB ... and vice-versa.

Executing the command rigel run sequence example_sequence will trigger the execution of the following functions:

Thread_0	Thread_1
PluginA.setup()	PluginD.setup()
PluginA.start()	PluginD.start()
PluginA.process()	PluginC.setup()
PluginA.stop()	PluginC.start()
PluginB.setup()	PluginC.process()
PluginB.start()	PluginC.stop()
PluginB.process()	PluginD.stop()
PluginB.stop()

Summary: plugins are called according to the type of subsequence. Each subsequence is executed on a different thread.

I hope you find this message useful 😄

Let me know if you have any questions or found any problems.

Happy Easter! 🐰

Rigelfile Simplification

The next thing to do here is to simplify the Rigelfile field corresponding to this plugin.

Specifically, we can "transfer" most - if not all - of the other fields to a dictionary for each column. That way, it is possible to do a verification and introspection on multiple axes.

Additionally, it would also be pertinent to add the functionality of checking for the existence - or lack thereof - of certain strings, so the plugin can also analyze non-numerical results.