mpiraux/mpf

Minimal Performance Framework

mpf: Minimal Performance Framework

mpf is a tool to write evaluation experiments, exploring variables of interest to understand the performance of computer systems. mpf can compile code, deploy software to a given computer cluster and orchestrate the evaluation to collect data for which graphs can be generated.

Features

• Experiments are written in IPython.
• Variables can be defined and combined to explore executions scenarios for a given experiment
• Code can be deployed and compiled
• A simple graph is generated from an experiment run

Defining experiments

A mpf experiment consists of an IPython script. It defines several sections, specifying the variables to explore, the roles taking part in the experiments and the specific code they execute. The following example demonstrates how mpf can run an iperf3 server and client and explore values of the -P parallel and -Z zerocopy parameter.

#!/usr/bin/env -S ipython --
import mpf

mpf.default_setup()
mpf.add_variable('parallel', range(1,9))
mpf.add_variable('zerocopy', {'': 'disabled', '-Z': 'enabled'})

@mpf.run(role='server')
def start_server():
    %ex iperf3 -D -s -1 > /dev/null

@mpf.run(role='client', delay=1)
def start_client(mpf_ctx, parallel, zerocopy):
    result = %ex iperf3 -f k -t 2 -P $parallel $zerocopy -c {mpf_ctx['roles']['server']['interfaces'][0]['ip']} | tail -n 3 | grep -ioE "[0-9.]+ [kmg]bits"
    return {'goodput': result[0]}

df = next(mpf.run_experiment())

The script defines several parts constituting the experiment. First, it defines two variables that will be explored in the experiment. mpf combines the values of each variable to derive the experiment runs. Second, mpf allows defining functions that will be executed on particular nodes of the cluster. In our example, the function start_server will be executed on the machine bearing the server role. The start_client function will be executed following a one second delay after the previous function completed. This function specifies three parameters. The mpf_ctx object describes the context of the experiment, e.g., the IP addresses of each node. parallel, zerocopy will receive values from the ranges defined above. The start_client collect the iperf3 goodput and returns it as part of the result dictionary.

After the experiment has ran, a pandas Dataframe is returned with the experiment variables composing its indexes and the results its columns. Each row corresponds to one run.

The script can be executed directly and receive a cluster YAML file defining the machines that will run the experiment.

$ tests/experiments/iperf.ipy -c tests/experiments/cluster.yaml

Requirements

Running experiments with mpf requires reasonably coherent Python and packages versions between the executor and the remote nodes. ipyparallel should be installed on the remote nodes.

mpf's Issues

Storage of results

I would prefer mpf to be explicit about the storage of collected results. For that purpose I would avoid a file format that cannot reasonably self-describe. I expect the magnitude of data collected through mpf to be relatively low in most cases so the description overhead will remain acceptable.
Also, each experiment should produce a separate set of results, probably in a separate file. The file should of course be machine readable so I would aim for YAML.

Python packaging

A generic issue is to make the code easily installable by pip, both for development and release. I think I need a refresher on that.

Deploying and compiling code

There is currently no way to instruct mpf how to deploy and compile the code we would like it to evaluate. I'm not sure how this should be abstracted. This probably requires to:

• Obtain the source code, either from a git repo or through rsync/scp. I tend to use the latter the most.
• Install dependencies to build the code. This depends on the Linux distribution. I feel that if we allow to install Debian/Ubuntu packages and Fedora/RHEL packages, we cover all current use cases.
• Build the code, which can require very different step (autoconf, cmake, ninja, etc..). So an open-format, e.g., bash commands would be quickly required
• Install the code somewhere practical for mpf, if needed

What is not needed for me at this stage:

• Figure out whether changes occured and this process should be triggered again. When using rsync and make, they can figure it out quite quickly
• Compile once and deploy everywhere. To me parallel compilation is the simplest way to go, at the cost of having to deploy the code everywhere its needed. I don't have a strong use case for that but you're mileage may vary.

Remote code execution

We have IPython functions to run the experiments but no way to run them on a remote machine.
IPython parallel support could be the way. Especially:

• https://ipyparallel.readthedocs.io/en/latest/tutorial/process.html#starting-a-cluster-with-ssh
• https://ipyparallel.readthedocs.io/en/latest/tutorial/direct.html

mpf should leverage ipyparallel and map roles to machine in the cluster. I'm unsure whether mpf should define the cluster itself. It will depend on the amount of boilerplate needed to get it to work. If it is too much then mpf will provide a simpler way.

Variable hierarchy

Some variables in an experiment are relevant to values of others. For instance, in my single-path & multi-path experiment, the path scheduler variable is not relevant to the single-path case. Yet when variant and path scheduler are two different variables, the single-path case will explore both values of the scheduler variable.

Continuous Integration

We need to also think out about CIs. I'm not so aware of cool Python-related CIs. Getting the package to "build" and executes the tests would be the minimum.

Resuming experiment

When an experiment fails, e.g. due to a full disk or unavailable SSH materials, there is no easy way to restart it.

Several steps for reproducibility are already in place: seeded variable values exploration, copy of experiment and cluster file to the experiment results directory.

mpf.run_experiment should be modified to yield partial DataFrame with intermediate results. Then, the last saved DataFrame could be passed again to mpf.run_experiment along with the experiment id to resume it.

Visualising data

Following my thoughts on #5, I'm more and more convinced that using pandas DataFrame could bring many benefits. This would make the integration of mpf as a true notebook more straightforward, giving to the user the data collected by experiments as DataFrames for them to filter, select or aggregate results in any desired way.

Also, I've been briefly looking at libraries that could be used to produce graphes based on DataFrames and found these two. The amount of configuration seems reasonable. If we were to design a smart guessing algo to produce plots, I would make it drive one of these, instead of reinventing everything atop matplotlib.

https://nbviewer.org/gist/santosjorge/f3b07b2be8094deea8c6
https://plotly.com/python/