carverdamien / scheddisplay Goto Github PK

License: GNU General Public License v3.0

Python 93.02% Shell 5.94% Dockerfile 0.26% JavaScript 0.78%

scheddisplay's Introduction

SchedDisplay

SchedDisplay is a visualization tool for SchedLog, a custom ring buffer collecting scheduling events in the Linux kernel.

(Note that SchedDisplay is currently being rewritten into TraceDisplay, a visualization tool for trace-cmd records.)

SchedDisplay in a few Steps

Build and launch the webserver with ./docker script.
Open the web app http://localhost:5006/v0.
Select a tarball file which contains the data recorded during an experiment and then click select. (see below for record/import)
Select a json file which contains the instructions on how to build the lines in the image from the recorded data and then click select. Uploading a local json file is also supported. (see below for writing custom instructions)
Select Figure to view the image.

There are additional tabs in the application. The Console tab reports progress and errors. The Stat tab computes statistics. The Var tab shows available events in the SchedLog kernel.

A jupyter environment is also available https://localhost (the ssl certificate is generated by jupyter-notebook.sh). The token can be obtain by running docker exec -ti NAME_OF_SCHEDDISPLAY_CONTAINER jupyter notebook list. Feel free to duplicate and modify ./examples/notebook/jupyter-example.ipynb.

The tarball record

You can try SchedDisplay with the trace provided in ./examples/trace/kbuild-sched-example.tar. To record your own trace, follow these instructions:

install the SchedLog kernel.
record your experiment with the sched_log script.
convert the per-cpu buffers into compressed numpy arrays with the report.py script.
create the tarball and store it in ./examples/trace

A tarball must at least contain the following files:

sched_log_traced_events.start (a copy of /proc/sched_log_traced_events) which contains the mapping EVENT_ID -> EVENT_NAME.
timestamp.npz
pid.npz
event.npz
cpu.npz
arg1.npz
arg0.npz

Check RecordSchedLog to discover how we automate our experiments.

The json instructions

The input field lists the data required to build the image. Some data are directly recorded through SchedLog (timestamp, cpu, event, pid, arg0, arg1). Other data must be computed.

The output field lists the data to store in the image. Some data are mandatory (x0,y0,x1,y1,c).

The c field lists instructions on how to build categories of lines. For example, the following category draws small vertical lines when the frequency measured at a scheduling tick is between 1.2 and 1.7 GHz on a given CPU.

{
        "label" : "[1.2, 1.7] GHz",
        "color" : "#8ded6d",
        "concatenate" : [[
                ["query","event==$TICK & arg1>1200000 & arg1<=1700000"],
                ["=","x0","timestamp"],
                ["=","x1","timestamp"],
                ["=","y0",["+","cpu",0.0]],
                ["=","y1",["+","cpu",0.5]]
        ]]
},

See examples/line/freq4_new.json for a full example.

scheddisplay's People

Contributors

Stargazers

Watchers

scheddisplay's Issues

Add more log between « lines built » and « plot »

Search Feature

User gives an automata to the tool and it will return sequences in the data that are accepted by the automata.

init comm.npz with content of ps.start

Add overview tab

Show global stats over time like total nr_running

Measure time between BLOCK and UNBLOCK

oltp 10-20 sec

Track filiation

@damien, j'ai joué un peu plus avec nxt_of_same_evt_on_same_cpu pour essayer de mettre une couleur correspondante a comm. Ca ne marche pas vraiment. Le pb est que comm n'est valide qu'a partir du exec pour un nouveau processus. On colorise donc avec l'info du pere. Il faudrait pouvoir tracer depuis un enqueue jusqu'a un exec, puis de l'exec jusqu'au dequeue du process.

SchedLog: add proc file that shows the mapping between event_id and event_name

Speedup compute

{
    "columns": {
        "y1":["+",["copy","cpu"],0.75],
        "x1":["nxt_of_same_evt_on_same_cpu","timestamp"]
    },
    "hdf5": ["./raw/hackbench/256-linux-comm.hdf5"],
    "renderers": [
        {
            "filter": ["&",["==","event",13],[">","arg0",0]],
            "label": "RQSIZE>0",
            "line_color": "#666666",
            "x0": "timestamp", "x1": "x1",
            "y0": "cpu", "y1": "cpu"
        },

         {
            "filter": ["&",["==","event",2], ["==","comm",8]],
            "label": "Wakeup gcc 8",
            "line_color": "#ffff00",
            "x0": "timestamp", "x1": "timestamp",
            "y0": "cpu", "y1": "y1"
        },
        {
            "filter": ["&",["|", ["==","event",3], ["==","event",0]], ["==","comm",8]],
            "label": "new gcc",
            "line_color": "#ffff0F",
            "x0": "timestamp", "x1": "timestamp",
            "y0": "cpu", "y1": "y1"
        },

         {
            "filter": ["&",["==","event",4], ["==","comm",6]],
            "label": "Block make 6",
            "line_color": "#AA0000",
            "x0": "timestamp", "x1": "timestamp",
            "y0": "cpu", "y1": "y1"
        },
        {
            "filter": ["&",["==","event",2], ["==","comm",6]],
            "label": "Wakeup make",
            "line_color": "#ff0000",
            "x0": "timestamp", "x1": "timestamp",
            "y0": "cpu", "y1": "y1"
        },
        {
            "filter": ["&",["==","event",3], ["==","comm",6]],
            "label": "new make",
            "line_color": "#ff000F",
            "x0": "timestamp", "x1": "timestamp",
            "y0": "cpu", "y1": "y1"
        },

       {
            "filter": ["&",["==","event",2], ["==","comm",9]],
            "label": "Wakeup bash 9",
            "line_color": "#00ff00",
            "x0": "timestamp", "x1": "timestamp",
            "y0": "cpu", "y1": "y1"
        },
        {
            "filter": ["&",["|", ["==","event",3], ["==","event",0]], ["==","comm",9]],
            "label": "new bash",
            "line_color": "#00ff0F",
            "x0": "timestamp", "x1": "timestamp",
            "y0": "cpu", "y1": "y1"
        }

    ],
    "truncate": {
        "cursor": 80000000,
        "mode": "index",
        "width": 10000
    }
}

Add option to disable automatic update_image

SchedLog: improve recording of task->comm. Add a new_comm event.

Some task->comm are missing

Check if

there are exec_evt with unknown comm and display them. (All exec_evt had a comm, there were multiple addr mapping to the same comm because bytes after '\0' are not cleared)
there are processes with unknown comm and display them
Miss reading the data (yes=> fix, no=>Error in data)
p->comm is never updated
p->comm is updated after exec_evt

Split app v0 in two. One which builds and saves images, the other that loads and shows images.

Forbid user from running multiple image updates in parallel.

Seamless transition between datashader and renderers

Je me rend compte qu'on a (au moins) deux besoins en terme d'affichage. Une vue globale qui permet de rechercher des phases, et une vue locale pour comprendre ce qui se passe avec le detail des evts.
Pour la vue locale c'est bon.
Pour la vue globale c'est trop lent. Il faut mettre une taille tres grande et le temps d'affichage devient prohibitif, meme avec un script simplifié qui n'affiche que rqsize!=0.

Upload hdf5 button OR FileExplorerView?

Upload hdf5 button

CON:

users have to store them locally
users need a good network bandwidth

PRO:

Reuse system file selection

FileExplorerView for files on server

CON:

wasting dev time
users cannot name/organize the files

PRO:

Ready to use by anyone
Can change file format independently from users

Configure image height

quickfix:

nr_row = len(np.unique(lines['y0']))

Use https://dask.org/

Add web service to download trace

Cache expensive computations made after loading data

Hide hover on specific data.

Could be possible with custom div.
https://bokeh.pydata.org/en/latest/docs/user_guide/tools.html

Add import file button to upload .json

Compare local_new to cfs

Add legend for categories

Script to generate a Table of completion time

At recording. Store t0 somewhere in the tar and subtract it to timestamp.

SchedLog: improve recording of BLOCK reason

SelectFileViewController: too many entries in Select

Show number of tasks in background to the user

Improve loading time

500MB hdf5 files takes too long to load.
Maybe decompress in parallel?
Maybe stop using pandas.
Maybe do not use hdf5, but tar with np.savez_compressed

Merge Baseline and Monitor by adding a option MONITOR=true

Change columns from dict to list to ensure order.

Put example/record in another repository

reset to xmin, xmax of data when window is wider

Annotations/Comments/Markers

un truc qui serait utile serait de pouvoir inserer des notes a des timestamps donné. Il faudrait pouvoir sauver les notes dans un fichier separé pour pouvoir les recharger.