file_intro_plugin's Issues
Result File
Hi @rarrais and @MisterOwlPT! I was wondering something that might come up as an issue later. So this plugin would use as input the result file from a calibration using ATOM in our containers. My question is, how do we get the csv
file from inside the containers to use in this introspection plugin? Does Rigel include a tool to extract such a file?
Thanks in advance!
Rigelfile Simplification
The next thing to do here is to simplify the Rigelfile field corresponding to this plugin.
Specifically, we can "transfer" most - if not all - of the other fields to a dictionary for each column. That way, it is possible to do a verification and introspection on multiple axes.
Additionally, it would also be pertinent to add the functionality of checking for the existence - or lack thereof - of certain strings, so the plugin can also analyze non-numerical results.
Plugin compliant with new features.
Hi @Kazadhum
I implemented core new features to the Rigel framework that will, among other things, support your work (Rigel now supports concurrent execution of multiple plugins and decouples introspection from infrastructure launching).
However, to take advantage of these new features you'll need to change some things in your plugin and the Rigefileyou are using. I leave here a detailed description of everything that changed for your reference (@rarrais included).
Please keep in mind the following:
-
The new features are working but were not yet tested intensively. For this reason, I created a new branch flow. This way you'll always have the stable develop intact if something goes wrong and you need to revert. I suggest you do something similar with your work. Consider creating a branch for your updated plugin and associated Rigelfiles.
-
Next week I'll be on holiday. I'll try to pay attention to GitHub issues and help you if required.
UPDATES TO RIGEL
NOTE: remember to use branch flow !
Rigel plugins
Changes to interface
The run
function was renamed to start
.
A new function process(self) -> None
was introduced.
All other required functions remain the same.
During execution, the following plugin functions are now called in the following order:
setup()
start()
process()
stop()
Notice that process
is always called after start
.
Rigel plugins must be compliant with this new interface.
NOTE: Arguments to the plugin constructor also remain unchanged.
New function process()
Whenever applicable, this function acts as a complement to start
and should contain logic that is either:
- user-dependent (built-in function
input
, keyboard events, ...) - blocking or very long (
while
loops,time.sleep
, ...) - analysis of results
All other business logic should remain inside start
.
This division allows for concurrent execution of plugins.
Depending on the chosen flow of execution the function process
may or not be called.
Check the section on executors below for more information.
Data sharing mechanism
Plugins that are executed in sequence can now share data between themselves!
This is done via a class attribute shared_data
of type Dict[str, Any]
. This attribute is automatically passed between all plugins for them to store/read data according as required.
All plugins automatically have access to this attribute without the need for changes (you can use access it self.shared_data
).
New plugin rigel.plugins.core.ComposePlugin
The existing plugin rigel.plugins.core.TestPlugin
was broken into two plugins. One kept the original name and the other name was called _rigel.plugins.core.ComposePlugin_
.
The new ComposePlugin
is responsible only for launching a containerized ROS application (think of it as Docker Compose for ROS). After launching the containers it waits forever for user input (CTRL-C / CTRL-Z). Check the following Rigelfile excerpt:
# ...
plugin: "rigel.plugins.core.ComposePlugin"
with:
components:
-
name: "simulation"
image: "{{ AWS_ECR }}/simulation:latest"
command: ["roslaunch", "tm_sim_gazebo", "run.launch", "gui:=true"]
envs:
DISPLAY: "{{ DISPLAY }}"
volumes:
- !!python/tuple ["/tmp/.X11-unix", "/tmp/.X11-unix"]
-
name: "robot"
image: "{{ AWS_ECR }}/robot:latest"
command: ["roslaunch", "robot_app", "run.launch", "use_rosbridge:=true"]
envs:
DISPLAY: "{{ DISPLAY }}"
volumes:
- !!python/tuple ["/tmp/.X11-unix", "/tmp/.X11-unix"]
introspection: True # <--- This component has a ROS bridge installed
# ...
If a component is declared with introspection: True
the plugin will store the name of the container for that component in the shared plugin data (key simulation_hostname
). This can be used by other plugins, if applicable. If required, check the plugin model.
The TestPlugin
is now responsible only for the introspection of a containerized ROS system. It still requires ROS bridge. Check the following Rigelfile excerpt:
# ...
plugin: "rigel.plugins.core.TestPlugin"
with:
requirements:
- 'globally : some /OSPS/TM/HeartBeep [osps_msgs/TMHeartBeep] {publisherId = "friday"}'
- "globally : some /OSPS/TM/TaskStatus [osps_msgs/TMTaskStatus] {statusCode = 5}"
# ...
If no field hostname
or port
were declared the plugin will automatically look for connection data inside the shared plugin data. If required check the plugin model.
Updated plugins
All plugins except rigel.plugins.aws.RoboMakerPlugin
were updated, are compliant with the new protocol and are ready to use.
Executors, job sequences and execution flows
Until now Rigel was only capable of executing sequential job sequences (via the rigel run sequence
command).
Now it supports the following execution flows:
- sequential
- concurrent
- parallel
Sequential job sequences
NOTE: this corresponds to the execution already supported by Rigel.
Consider the following Rigelfile excerpt where an example sequential job sequence example_sequence
is declared:
# ...
sequences:
example_sequence:
stages:
jobs: ["a", "b"] # <-- One or more job ids
# ...
Assume that job a
uses plugin PluginA
and job b
uses plugin PluginB
.
Executing the command rigel run sequence example_sequence
will trigger the execution of the following functions:
PluginA.setup()
PluginA.start()
PluginA.process()
PluginA.stop()
PluginB.setup()
PluginB.start()
PluginB.process()
PluginB.stop()
Summary: PluginA
executes completely and then PluginB
executes completely.
Concurrent job sequences
Consider the following Rigelfile excerpt where an example concurrent job sequence example_sequence
is declared:
# ...
sequences:
example_sequence:
stages:
jobs: ["a", "b"]
dependencies: ["c", "d"]
# ...
Assume that job a
uses plugin PluginA
, job b
uses plugin PluginB
... and vice-versa.
Executing the command rigel run sequence example_sequence
will trigger the execution of the following functions:
PluginC.setup()
PluginC.start()
PluginD.setup()
PluginD.start()
PluginA.setup()
PluginA.start()
PluginA.process()
PluginA.stop()
PluginB.setup()
PluginB.start()
PluginB.process()
PluginB.stop()
PluginC.stop()
PluginD.stop()
Summary: PluginC
and PluginD
are partially executed (function process
is never called). Before the stop
function is called, PluginA
and PluginB
are completely executed and in sequence.
NOTE: Consider using this execution flow in your work. Use the new
rigel.plugins.core.ComposePlugin
to launch the containers you require and use your plugin to inspect their execution as required.
Parallel job sequences
Consider the following Rigelfile excerpt where an example parallel job sequence example_sequence
is declared:
# ...
sequences:
example_sequence:
stages:
- # <-- Will be executed in a separate Thread_0
jobs: ["a", "b"]
- # <-- Will be executed in a separate Thread_1
jobs: ["c"]
dependencies: ["d"]
# ...
Here, exceptionally, each stage declared within stages
must consist of either a sequential or concurrent sequence of jobs.
Assume that job a
uses plugin PluginA
, job b
uses plugin PluginB
... and vice-versa.
Executing the command rigel run sequence example_sequence
will trigger the execution of the following functions:
Thread_0 | Thread_1 |
---|---|
PluginA.setup() |
PluginD.setup() |
PluginA.start() |
PluginD.start() |
PluginA.process() |
PluginC.setup() |
PluginA.stop() |
PluginC.start() |
PluginB.setup() |
PluginC.process() |
PluginB.start() |
PluginC.stop() |
PluginB.process() |
PluginD.stop() |
PluginB.stop() |
Summary: plugins are called according to the type of subsequence. Each subsequence is executed on a different thread.
I hope you find this message useful ๐
Let me know if you have any questions or found any problems.
Happy Easter! ๐ฐ
Further suggestions
Hello @miguelriemoliveira, @rarrais and @MisterOwlPT!
I've got a working plugin now, and I was wondering if you had any suggestions. I wrote the plugin to be used for anything, so I don't mention ATOM or calibration specifically. The way I have it now, the plugin assumes the results file is a .csv
file (should I expand this to other file formats?). The Rigelfile looks like this.
As you can see, it receives the row and column of the value to be used to evaluate the results, as well as thresholds for the various levels of quality. Finally, it receives a required level of quality, or how good the results need to be to 'pass' the test.
What I wanted to ask is if this seems right to you and, furthermore, if you have any suggestions.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.