gesalous/perfspect

System performance characterization tool based on linux perf

PerfSpect

Quick Start | Output | Deploy in Kubernetes | Requirements | Build from source

PerfSpect is a system performance characterization tool built on top of linux perf. It contains two parts:

perf-collect: Collects hardware events at a 5 second output interval with practically zero overhead since PMU's run in counting mode.

• Collection mode:
  • sudo ./perf-collect default system wide
  • sudo ./perf-collect --socket
  • sudo ./perf-collect --cpu
  • sudo ./perf-collect --pid <process-id>
  • sudo ./perf-collect --cid by default, selects the 5 containers using the most CPU at start of perf-collect. To monitor specific containers provide up to 5 comma separated cids i.e. <cid_1>,<cid_2>
• Duration:
  • sudo ./perf-collect default run until terminated
  • sudo ./perf-collect --timeout 10 run for 10 seconds
  • sudo ./perf-collect --app "myapp.sh myparameter" runs for duration of another process

perf-postprocess: Calculates high level metrics from hardware events

• ./perf-postprocess

Quick start (requires perf installed)

wget -qO- https://github.com/intel/PerfSpect/releases/latest/download/perfspect.tgz | tar xvz
cd perfspect
sudo ./perf-collect --timeout 10
./perf-postprocess

Output

perf-collect outputs:

1. perfstat.csv: raw event counts with system metadata

perf-postprocess outputs:

1. metric_out.sys.average.csv: average metrics
2. metric_out.sys.csv: metric values at every 5 second interval
3. metric_out.html: html view of a few select metrics

Deploy in Kubernetes

Modify the template deamonset.yml to deploy in kubernetes

Requirements

perf - PerfSpect uses the Linux perf tool to collect PMU counters

Different events require different minimum kernels (PerfSpect will automatically collect only supported events)

1. Base (CPU util, CPI, Cache misses, etc.)
   • 3.10
2. Uncore (NUMA traffic, DRAM traffic, etc.)
   • 4.9
3. TMA (Micro-architecture boundness breakdown)
   • ICX, SPR: 5.10
   • BDX, SKX, CLX: 3.10

Build from source

Requires recent python. On successful build, binaries will be created in dist folder

pip3 install -r requirements.txt
make

Note: Most metrics and events come from perfmon and TMA v4.5