microsoft/microsoft-performance-tools-linux-android

Linux, Android and Chromium Performance Tools built using the Microsoft Performance Toolkit. Cross-platform .NET Core + WPA GUI

Microsoft Performance Tools Linux / Android

This repo contains various Linux and Android Performance Analysis tools built with the Microsoft Performance Toolkit SDK.

Tools are built with open source .NET Core and can be run on the cmd-line or in the WPA GUI. All the logs that are supported are open source.

Not only are the raw logs parsed, but a lot of smart post processing / correlation is done to make your life easier as a perf analyst. We hope you can solve & debug tough issues on you or your customers systems with this toolset!

Tracing supported:

• LTTng (Kernel CPU scheduling, Processes, Threads, Block IO/Disk, Syscalls, File events, etc)
• perf CPU Sampling(cpu-clock)
• Perfetto Android & Chromium (CPU Scheduling, CPU Sampling, CPU Frequency, FTrace, Android Logs, Generic Events / Default Tracks, GPU Counters, Jank Detection, Processes, Android Packages)

Logs supported:

• Dmesg
• Cloud-Init
• WaLinuxAgent
• AndroidLogcat

Optional WPA GUI:

Presentations

If you want to see a demo or get more in-depth info on using these tools (LTTng) check out a talk given at the Linux Tracing Summit:

Linux & Windows Perf Analysis using WPA (LTTng), (slides) (video)

Blog Posts

Featuring some use-cases and some walkthroughs

• New tools for analyzing Android, Linux, and Chromium browser performance – Microsoft-Performance-Tools for Linux-Android
• Perfetto tooling for analyzing Android, Linux, and Chromium browser performance – Microsoft-Performance-Tools-Linux-Android
• New Android logcat support available in Microsoft Performance Tools for Linux & Android

Prerequisites

Runtime prereqs

• .NET Core Runtime 3.1.x

Dev prereqs

• .NET Core SDK 3.1.x
• Visual Studio, VSCode, or your favorite editor!

Download

• For plugins Download see Releases

• NuGet Pkgs see PerformanceToolkitPlugins

• (Windows Only GUI - Install) Using the WPA GUI to load these tools as plugins

  • Download the latest Store Windows Performance Analyzer (Preview)

How to run the tools

The tools can be run in several modes:

• Cross-platform with .NET Core (Any OS that .NET Core supports)
  • Used as a library to process traces / logs programatically in a .NET Core language like C#
    • Examples:
    • LTTng 1, LTTng 2
    • Perf
    • LinuxLogs
    • Perfetto
  • With a driver program for example dumping to screen or text format
    • ./LTTngDriver.exe LTTngKernelTraceFolder
    • ./LTTngDriver.exe LTTng-Kernel-Trace.ctf (trace folder is zipped and renamed to .ctf)
• (Windows Only - Run) Using the WPA GUI to load these tools as plugins
  • WPA needs to be told where to find these additional plugins.
  • Using Windows Launcher script

    .\LaunchWpaPerfToolsLinuxAndroid.ps1 -i c:\PATH\TO\lttng-kernel-trace.ctf

    • -LinuxPerfToolsPluginFolder path may need to be specified if not auto-found correctly
  • In Command Prompt with -addsearchdir and -i trace file:

      wpa.exe -addsearchdir %HOMEDRIVE%%HOMEPATH%\Downloads\Microsoft-Performance-Tools-Linux-Android-1.2\Microsoft-Performance-Tools-Linux-Android\MicrosoftPerfToolkitAddins -i c:\PATH\TO\lttng-kernel-trace.ctf
    

  • OR with Env Variable to pick file from UI

     SET WPA_ADDITIONAL_SEARCH_DIRECTORIES=%HOMEDRIVE%%HOMEPATH%\Downloads\Microsoft-Performance-Tools-Linux-Android-1.2\Microsoft-Performance-Tools-Linux-Android\MicrosoftPerfToolkitAddins
     wpa.exe
    

  • Optional Troubleshooting - Verify that this WPA version supports plugins
    • In Command Prompt - Example:

      wpa.exe /?
      "C:\Program Files\WindowsApps\Microsoft.WindowsPerformanceAnalyzerPreview_10.0.22504.0_x64__8wekyb3d8bbwe\10\Windows Performance Toolkit\wpa.exe" /?
      

    • Verify that these 2 command line WPA options are supported:
      • OPTIONS: -addsearchdir PATH. Adds a directory path to the plugin search path. ....
      • ENVIRONMENT VARIABLES: WPA_ADDITIONAL_SEARCH_DIRECTORIES - A semicolon (;) delimited list of additional directories to search for plugins. Equivalent to the -addsearchdir option.
• (Windows) Command-line dumping to a text format based on the WPA UI (say CSV) (wpaexporter.exe)

  "C:\Program Files\WindowsApps\Microsoft.WindowsPerformanceAnalyzerPreview_10.0.22504.0_x64__8wekyb3d8bbwe\10\Windows Performance Toolkit\wpaexporter.exe" -addsearchdir PLUGIN_FOLDER -i traceFile
  

How to capture a trace or logs

• Linux
  • Please see Linux Trace Log Capture
• Perfetto
  • Android - Please see Record traces on Android
  • Linux - Please see Record traces on Linux
  • Chromium - In chrome://tracing/ Record choose "Use protobuf" OR use UI at https://ui.perfetto.dev/
    • Also see https://www.chromium.org/developers/how-tos/trace-event-profiling-tool/recording-tracing-runs

How to load the logs in the UI

• LTTng - If you just need to open only a LTTng trace by itself in folder format
  • WPA -> Open -> Folder -> (Select CTF folder)
    • Note: Requires >= 1.2 release AND WPA >= 10.6.20.1 (via WPA Help -> About)
• Perfetto
  • WPA -> Open -> (Select Perfetto trace file)
    • Note: The Perfetto plugin explicitly supports the .perfetto-trace and .pftrace file types, but it does support more (e.g. Protobuf, Chrome JSON). You just need to rename to one of the stated supported types
• Unified (LTTng, Perfetto, or other multiple different logs files together)
  • Once you gather the data, there is a tiny bit of prep needed to open them in a single unified timeline (like the screenshot above)
  • If you want to open multiple logs together in single timeline - Copy all trace files and logs you want to open to single folder
  • Example: You want to open in the same timeline: LTTng, Perf CPU Sampling, Dmesg
    • Ensure that the Linux CTF folder/trace is zipped and renamed to .ctf in the same folder (hack so open Unified works)
  • WPA -> File -> Open -> Multi-select all files and choose "Open Unified"

How do I use WPA in general?

If you want to learn how to use the GUI UI in general see WPA MSDN Docs

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

microsoft-performance-tools-linux-android's Issues

[Feature Request] Add resolved callstacks to Execution Events

I'm hoping for the same New Thread Stack and Ready Thread Stack options that Windows CPU Usage (Precise) has.

LTTng ref trace with channel filenames like chan1_0 does not load

I am new to CTF. I have found from a LTTNG tutorial a sample file https://lttng.org/blog/2018/03/21/lttng-scope-0.3-a-basic-tutorial/ which has a tarball of https://github.com/lttng/lttng-ref-traces/raw/master/traces/basic-kernel-analysis.tar.xz
The trace data generated by lttng is a folder of channel files

To make it ctf it is sufficient to zip the complete folder, rename to ctf and load it in WPA? I have tried that but I get then

Unable to process 'file:///C:/basic-kernel-analysis/kernel.ctf': System.Exception: No events - failure processing .ctf
   at PerfCds.PerfSourceParser.ProcessSource(ISourceDataProcessor`3 dataProcessor, ILogger logger, IProgress`1 progress, CancellationToken cancellationToken) in D:\a\1\s\Microsoft-Performance-Tools-Linux\PerfCds\PerfSourceParser.cs:line 132
   at Microsoft.Performance.SDK.Runtime.Extensibility.SourceProcessingSession`3.ProcessSource(ILogger logger, IProgress`1 progress, CancellationToken cancellationToken)
   at Microsoft.Performance.SDK.Processing.CustomDataProcessorBaseWithSourceParser`3.ProcessAsyncCore(IProgress`1 progress, CancellationToken cancellationToken)
   at Microsoft.Performance.SDK.Processing.CustomDataProcessorBase.ProcessAsync(IProgress`1 progress, CancellationToken cancellationToken)
   at Microsoft.Performance.SDK.Runtime.CustomDataSourceExecutor.ExecuteAsync(CancellationToken cancellationToken)
Unable to process 'file:///C:/basic-kernel-analysis/kernel.ctf': System.Exception: No events - failure processing .ctf
   at LTTngCds.LTTngSourceParser.ProcessSource(ISourceDataProcessor`3 dataProcessor, ILogger logger, IProgress`1 progress, CancellationToken cancellationToken) in D:\a\1\s\Microsoft-Performance-Tools-Linux\LTTngCds\LTTngSourceParser.cs:line 135
   at Microsoft.Performance.SDK.Runtime.Extensibility.SourceProcessingSession`3.ProcessSource(ILogger logger, IProgress`1 progress, CancellationToken cancellationToken)
   at Microsoft.Performance.SDK.Processing.CustomDataProcessorBaseWithSourceParser`3.ProcessAsyncCore(IProgress`1 progress, CancellationToken cancellationToken)
   at Microsoft.Performance.SDK.Processing.CustomDataProcessorBase.ProcessAsync(IProgress`1 progress, CancellationToken cancellationToken)
   at Microsoft.Performance.SDK.Runtime.CustomDataSourceExecutor.ExecuteAsync(CancellationToken cancellationToken)
Unable to process any data sources.

I have copied the extensions of your drop to "C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\CustomDataSources\MicrosoftPerfToolkitAddins". WPA comes then up properly.

Originally posted by @Alois-xx in #6 (comment)

Loading perf artifact is not well documented, or not flexible

Hi team,

I did reverse engineering to load perf file. Then I felt it is not well documented and not flexible.

• Requires perf scripted output
• Requires XXX**.perf.data.txt** extension

Especially about the extension requirements, as WPA extends feature for .txt loading as shown in Diagnostic Console

WPA cannot find this plugin even SET WPA_ADDITIONAL_SEARCH_DIRECTORIES

Environment:

• Window 11
• C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Host.win-x64\3.1.32
• WPA Version 10.6.20.1

Result

C:\Users\GAME>SET WPA_ADDITIONAL_SEARCH_DIRECTORIES=C:\tools\Microsoft-Performance-Tools-Linux-Android\MicrosoftPerfToolkitAddins
C:\Users\GAME>wpa.exe -listplugins

C:\Users\GAME>
                             - AVAILABLE PLUGINS -

        * XPerf

    For help, use -help <plugin name>
    For version information, use -version <plugin name>

Create NuGet package

Package up plugin binaries into a NuGet package

hello

LTTngDataExtensions unable to process traces from NET6.0+ project

During trace processing, an exception is thrown (shown below) when processing an LTTng trace when run from a NET6.0 or NET7.0 project. When used from netcoreapp3.1 and NET5.0, the trace processing still works.

You can repro this by modifying the LTTng unit tests within the repo to target those versions of .NET

Repro

1. Change LTTngDataExtUnitTest/LTTngDataExtUnitTest.csproj targetframeworks to build NET5.0, NET6.0, and NET7.0
   a. <TargetFramework>netcoreapp3.1</TargetFramework> -> <TargetFrameworks>netcoreapp3.1;net5.0;net6.0;net7.0</TargetFrameworks>
2. Rebuild LTTngDataExtUnitTest (will probably have to re-open solution first)
3. Run all LTTngDataExtUnitTest tests in each framework
4. netcoreapp3.1 and net5.0 tests all pass
5. net6.0 and net7.0 unit tests all fail to process traces (although 3 of the unit tests still pass in each)

Exception:

CtfPlayback.Metadata.CtfMetadataException
  HResult=0x80131500
  Message=type_specifier does not have TypeSpecifier: line=1, startToken=new_setting
  Source=CtfPlayback
  StackTrace:
   at CtfPlayback.Metadata.AntlrParser.CtfListener.ExitDeclarationSpecifierTypeSpecifier(DeclarationSpecifierTypeSpecifierContext context) in D:\git\Microsoft-Performance-Tools-Linux-Android\CtfPlayback\Metadata\AntlrParser\CtfListener.cs:line 842
   at CtfParser.DeclarationSpecifierTypeSpecifierContext.ExitRule(IParseTreeListener listener) in D:\git\Microsoft-Performance-Tools-Linux-Android\CtfPlayback\Metadata\AntlrParser\Generated\CtfParser.cs:line 581
   at Antlr4.Runtime.Tree.ParseTreeWalker.ExitRule(IParseTreeListener listener, IRuleNode r)
   at Antlr4.Runtime.Tree.ParseTreeWalker.Walk(IParseTreeListener listener, IParseTree t)
   at Antlr4.Runtime.Tree.ParseTreeWalker.Walk(IParseTreeListener listener, IParseTree t)
   at CtfPlayback.Metadata.AntlrParser.CtfAntlrMetadataParser.Parse(Stream metadataStream) in D:\git\Microsoft-Performance-Tools-Linux-Android\CtfPlayback\Metadata\AntlrParser\CtfAntlrMetadataParser.cs:line 60
   at CtfPlayback.CtfPlayback.GeneratePlaybackStreams(ICtfInput source, CtfPlaybackOptions playbackOptions) in D:\git\Microsoft-Performance-Tools-Linux-Android\CtfPlayback\CtfPlayback.cs:line 142
   at CtfPlayback.CtfPlayback.Playback(ICtfInput source, CtfPlaybackOptions playbackOptions, IProgress`1 progress) in D:\git\Microsoft-Performance-Tools-Linux-Android\CtfPlayback\CtfPlayback.cs:line 69
   at LTTngCds.LTTngSourceParser.ProcessSource(ISourceDataProcessor`3 dataProcessor, ILogger logger, IProgress`1 progress, CancellationToken cancellationToken) in D:\git\Microsoft-Performance-Tools-Linux-Android\LTTngCds\LTTngSourceParser.cs:line 134
   at Microsoft.Performance.SDK.Runtime.Extensibility.SourceProcessingSession`3.ProcessSource(ILogger logger, IProgress`1 totalProgress, CancellationToken cancellationToken)
   at Microsoft.Performance.SDK.Processing.CustomDataProcessorWithSourceParser`3.ProcessAsyncCore(IProgress`1 progress, CancellationToken cancellationToken)
   at Microsoft.Performance.SDK.Processing.CustomDataProcessor.ProcessAsync(IProgress`1 progress, CancellationToken cancellationToken)
   at Microsoft.Performance.SDK.Runtime.ProcessingSourceExecutor.<ExecuteAsync>d__14.MoveNext()

Lack of LTTngCds implementation?

Hi Team,

I'm writing MsQuic LTTng trace extension for WPA.
microsoft/msquic#3294

I noticed lack of information in LTTngEvent which is actually in trace file.
As in image, vpid and vtid are stored in LTTngEvent.StreamDefinedEventContext as “_vpid” and “_vtid”.
However there is no cpu_id information which is stored in as stream.packet.context
Is this lack of LTTngCds implementation?

One of trace
quic.babel.txt
The trace file exactly has cpu_id in stream.packet.context

The cpu_id is not stored in LTTngEvent

[Feature] Perfetto - Support Table/Views for Jank Detection and Frames Expected vs Actual

There should be a table / view showing Expected vs Actual Frames. These should show up in Generic Events today but should be moved to their own table with appropriate UI.

https://perfetto.dev/docs/data-sources/frametimeline

The 2 tables available via the SQL API:
https://perfetto.dev/docs/analysis/sql-tables#expected_frame_timeline_slice
https://perfetto.dev/docs/analysis/sql-tables#actual_frame_timeline_slice

[Feature] Perfetto - GPU Counters tables

Add set of grapshs for GPU hardware counters

https://xdc2020.x.org/event/9/contributions/616/attachments/718/1321/xdc2020-fahien-presentation.pdf

LTTng traces do not show the thread id

I took a trace using the following commands (see the add-context line for the thread id).
The thread id does not show up in WPA but it shows in TraceCompass.
lttng create sess1 --output /home/user
lttng add-context --userspace --type=vtid --type=pthread_id
lttng enable-event --userspace --all
lttng start sess1
./myapp
lttng destroy sess1

Linux "perf" traces give a confusing error if the trace is not named "perf.data.txt"

I was trying to have multiple trace files in the same directory. I was really confused because only my original trace (named perf.data.txt) would open correctly. Eventually it was realized that the file name matters. Renaming the other trace to something else and my new trace to perf.data.txt allowed me to open it in WPA. This error message is counfusing/misleading to me because it doesn't provide any guidance or hint about what is wrong.

(Apologies for a screenshot of a screenshot)

[Feature] Perfetto - Support Android 12 CPU Scheduling CallStacks

Having this feature would enable the Perfetto plugin to show CPU Scheduling callstacks of both kernel & user-mode either at the system or process level. What is collected is dependent on the tracing config.

Users could easily see:
-In CPU Scheduling for a given High CPU% process (or any grouping/aggregation) the top callstacks aggregated together

This would mean users can perform manual wait analysis / critical path analysis; and improve system and app performance. Not only could they see the Process & Thread that led to a Wakeup, but the callstacks of what is about to run (what code was blocked before or not running) and the callstack of the alerter (the code that led to the wakeup)

Callstacks are dependent on this issue in Perfetto -
Feature - Support callstack capturing for cpu scheduling events

At the capture level, this is Android 12 only and supposedly works in that version although the documentation to capture a trace is a bit cryptic at the moment.

LTTng Parsing Library to Use

I see there are a number of NuGet packages published out of this repo that can possibly be used for LTTng parsing. In MsQuic we use LTTng for our tracing on Linux. We also have a custom WPA plugin, that currently only supports our ETW events (used on Windows). We're looking to add LTTng support to our plugin and were wondering if it's possible/reasonable to consume stuff out of here. Can we do this? What should we use? https://www.nuget.org/packages/Microsoft.Performance.Toolkit.Plugins.LinuxLogParser looks promising.

Linux "perf" traces should allow multiple file names, not just "perf.data.txt"

It is my understanding that the file name contains important information about which type of trace is contained in the .txt file. Fair enough. However, I think it would be nice to change this Equals call to an EndsWith so that I can have different traces in the same directory with useful names, so long as they end with .data.txt or some equivalent.

Microsoft-Performance-Tools-Linux-Android/PerfDataExtensions/SourceDataCookers/PerfDataProcessingSource.cs

Line 64 in 12065f9

return dataSource.IsFile() && StringComparer.OrdinalIgnoreCase.Equals("perf.data.txt", Path.GetFileName(dataSource.Uri.LocalPath));

Alternatively, an arbitrary file extension could be used for this even if the data in the file is actually just .txt.

[Bug] Perfetto - Can't open multiple Perfetto traces in the same session

You can't open one Perfetto trace and then open a second Perfetto trace in the same WPA session. This is due to a bug in the Performance SDK which is tracked with this issue microsoft/microsoft-performance-toolkit-sdk#112

Note: You can still open multiple traces (including a Perfetto trace ) at the same time using Unified Mode.

error while load trace with perf

it pop a wind title "error while querying table cpu scheduler events"

[PerfettoCds.PerfettoDataProcessor]: Verbose - Query for PerfettoExpectedFrameEvent returning 434 rows completed in 0.0064454s at 05/06/2023 06:26:49.376 UTC
Loaded Local Preferences from C:\Users\dd\AppData\Local\Windows Performance Analyzer\Preferences.xml
Error: Error while querying table CPU Scheduler Events. Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'index')
at System.Linq.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument)
at System.Linq.Enumerable.ElementAt[TSource](IEnumerable1 source, Int32 index) at PerfettoCds.Pipeline.CompositeDataCookers.PerfettoCpuSchedEventCooker.PopulateCpuWakeEvents(IDataExtensionRetrieval requiredData, ProcessedEventData1 threadData, ProcessedEventData1 processData) in D:\a\1\s\Microsoft-Performance-Tools-Linux-Android\PerfettoCds\Pipeline\CompositeDataCookers\PerfettoCpuSchedEventCooker.cs:line 143 at PerfettoCds.Pipeline.CompositeDataCookers.PerfettoCpuSchedEventCooker.PopulateCpuSchedulingEvents(IDataExtensionRetrieval requiredData, ProcessedEventData1 threadData, ProcessedEventData1 processData) in D:\a\1\s\Microsoft-Performance-Tools-Linux-Android\PerfettoCds\Pipeline\CompositeDataCookers\PerfettoCpuSchedEventCooker.cs:line 62 at PerfettoCds.Pipeline.CompositeDataCookers.PerfettoCpuSchedEventCooker.OnDataAvailable(IDataExtensionRetrieval requiredData) in D:\a\1\s\Microsoft-Performance-Tools-Linux-Android\PerfettoCds\Pipeline\CompositeDataCookers\PerfettoCpuSchedEventCooker.cs:line 54 at Microsoft.Performance.SDK.Runtime.Extensibility.DataExtensions.DataCookers.CompositeDataCookerReference.CreateInstance(IDataExtensionRetrieval requiredData) at Microsoft.Performance.SDK.Runtime.Extensibility.DataExtensions.ProcessingSystemCompositeCookers.GetOrCreateCompositeCooker(DataCookerPath cookerPath, Func2 createDataRetrieval)
at Microsoft.Performance.SDK.Runtime.Extensibility.DataExtensions.ProcessingSystemCompositeCookers.GetCookerOutput(DataCookerPath dataCookerPath)
at Microsoft.Performance.SDK.Runtime.Extensibility.FilteredDataRetrieval.GetCompositeDataRetrieval(DataCookerPath cookerPath)
at Microsoft.Performance.SDK.Runtime.Extensibility.FilteredDataRetrieval.GetCompositeDataRetrieval(DataOutputPath outputPath)
at Microsoft.Performance.SDK.Runtime.Extensibility.FilteredDataRetrieval.QueryCompositeOutput[T](DataOutputPath dataOutputPath)
at Microsoft.Performance.SDK.Runtime.Extensibility.FilteredDataRetrieval.QueryOutput[T](DataOutputPath dataOutputPath)
at PerfettoCds.Pipeline.Tables.PerfettoCpuSchedTable.IsDataAvailable(IDataExtensionRetrieval tableData) in D:\a\1\s\Microsoft-Performance-Tools-Linux-Android\PerfettoCds\Pipeline\Tables\PerfettoCpuSchedTable.cs:line 121
--- End of inner exception stack trace ---
at Microsoft.Performance.Threading.BackgroundWorkManager.Send(Action action)
at Microsoft.Performance.Threading.BackgroundWorkManager.SendFunc[TResult](Func`1 func)
at Microsoft.Performance.Shell.Bridge.ConvertToHdv.ProcessingSourceTableToHdvViewModelSourceAdaptor.IsDataAvailable()

Unable to load LTTNG trace with floating point fields

I have recorded a couple basic trace events in WSL using LTTNG, and when I try to load that trace directory in WPA GUI, I get an error about one of my floating point event fields. When I remove all the floating point fields and use integers instead, it works and the trace loads. Is there support for floating fields?

I am using binaries I got internally (). I was having trouble using binaries I got from this github.

Then a separate issue. When I did get the trace to load using integers, I didn't see the fields loaded in WPA. Is this something that should be working?

Thanks!

Zooming in the LTTng graph view does not filter the events in the table view

Repro steps:

• open an LTTng trace
• open the Generic events view
• select a time interval and zoom in
  The events in the table view are not filtered to the selection.

WPA sometimes fails to load perf data

Hi team,

WPA sometimes fails to load perf artifact. sometimes work.

Here is my data and diagnostic console log
handshake_f1000a.perf.data.txt
Diagnostic_Console.txt

[Feature] Perfetto - Support Processing of Android 12 CPU Callstack Sampling

Having this feature would enable the Perfetto plugin to show CPU Sampling of both kernel & user-mode either at the system or process level. What is collected is dependent on the tracing config.

Users could easily see:

• CPU Utilization % (dependent on scope of samples) (Note: CPU Util% via CPU Scheduling is already available)
• Top callstacks sorted by top cpu %
• Correlate cpu sampling with other data/tables, such as CPU Scheduling or other events

Perfetto trace format already seems to support collecting CPU Samples via PerfEventConfig callstack_sampling
PerfEventConfig
Configuration for the traced_perf profiler.

Example config for basic cpu profiling: perf_event_config { timebase { frequency: 80 } callstack_sampling { scope { target_cmdline: "surfaceflinger" target_cmdline: "system_server" } kernel_frames: true } }

Defined in perf_event_config.proto

In my testing, this did not seem to be supported on Android 11, but it did work on Android 12. This makes sense given the timestamps of the PerfEventConfig checkins.

I did verify that the data was present in the Android 12 trace and symbolic frame information was present via the trace processor CallStack Profilers tables

WPA crashes on data from latest lttng

Tested lttng-analyses-record on a kernel with the latest git lttng.
WPA Preview with the https://github.com/microsoft/Microsoft-Performance-Tools-Linux-Android/releases/tag/v1.3.3 release just crashes when trying to open the folder.
Note that lttng-iolatencystats ~/lttng-traces/lttng-analysis-*/ --debug also fails cause pid got removed from lttng_statedump_file_descriptor, though it's unclear in which version: https://lwn.net/Articles/816142/, https://lttng.org/docs/v2.13/#doc-whats-new

Traces

WPA is not displaying GenericEvent field names/values

Then a separate issue. When I did get the trace to load using integers, I didn't see the fields loaded in WPA. Is this something that should be working?

Yes that is desired be working but it looks unrelated to this float issue (as you said repro's on int). Not sure if this ever worked but I did try a few WPA versions and some older traces that seemed to have the same behavior not showing the fields values. LTTngDriver cmd-line accessed/displayed the fields fine so this is likely a separate UI issue we will need to work on.

Originally posted by @ivberg in #11 (comment)

"Open Unified" button not showing

I have been using it for data analysis recently, but have encountered doubts. There is no "Open Unified" option after the plugin is loaded successfully. Can you help me find out why, thanks a lot.


I need something like this picture below

perf artifact recorded without -a option failed to be loaded

Not sure here is a right place.

perf artifact generated without -a option failed to be loaded. the one can be loaded with -a option

step

perf record -F 5000 -g -o out.perf.data ${COMMAND}
perf script -i out.perf.data > out.perf.data.txt

here is the data and diagnostic console log
handshake_f5000.perf.data.txt
Diagnostic_Console.txt

No Graphs visible

I have downloaded the WPA Preview from the AppStore and used the launcher script.
When I load an lttng or a perfetto trace I do see either no graphs, or just Trace Stats, but no useful graphs.

I see this exception

Unable to process 'file:///C:/tmp/SharedFolder/lttng/SlowUbuntu.ctf': System.Exception: No events - failure processing .ctf
   at PerfCds.PerfSourceParser.ProcessSource(ISourceDataProcessor`3 dataProcessor, ILogger logger, IProgress`1 progress, CancellationToken cancellationToken) in D:\a\1\s\Microsoft-Performance-Tools-Linux-Android\PerfCds\PerfSourceParser.cs:line 132
   at Microsoft.Performance.SDK.Runtime.Extensibility.SourceProcessingSession`3.ProcessSource(ILogger logger, IProgress`1 totalProgress, CancellationToken cancellationToken)
   at Microsoft.Performance.SDK.Processing.CustomDataProcessorWithSourceParser`3.ProcessAsyncCore(IProgress`1 progress, CancellationToken cancellationToken)
   at Microsoft.Performance.SDK.Processing.CustomDataProcessor.ProcessAsync(IProgress`1 progress, CancellationToken cancellationToken)
   at Microsoft.Performance.SDK.Runtime.ProcessingSourceExecutor.ExecuteAsync(CancellationToken cancellationToken)
Loaded Local Preferences from C:\Users\xxxxx\AppData\Local\Windows Performance Analyzer\Preferences.xml

Am I doing something wrong?

I have tried that the example perfetto traces which are linked in from the perfetto UI. There are no errors, but many
disconnects?

[PerfettoCds.PerfettoDataProcessor]: Verbose - Querying for PerfettoStackProfileSymbol using SQL query: select id, type, name, source_file, line_number, symbol_set_id from stack_profile_symbol order by id
[PerfettoCds.PerfettoDataProcessor]: Verbose - trace_processor_shell.exe: [166.796] httpd.cc:193            [HTTP] Client disconnected
[PerfettoCds.PerfettoDataProcessor]: Verbose - trace_processor_shell.exe: [166.802] httpd.cc:186            [HTTP] New connection
[PerfettoCds.PerfettoDataProcessor]: Verbose - trace_processor_shell.exe: [166.802] httpd.cc:309            [HTTP] 0000 POST /rpc (body: 4 bytes).
[PerfettoCds.PerfettoDataProcessor]: Verbose - trace_processor_shell.exe: [166.803] httpd.cc:193            [HTTP] Client disconnected
[PerfettoCds.PerfettoDataProcessor]: Verbose - trace_processor_shell.exe: [166.810] httpd.cc:186            [HTTP] New connection
[PerfettoCds.PerfettoDataProcessor]: Verbose - trace_processor_shell.exe: [166.810] httpd.cc:309            [HTTP] 0000 POST /rpc (body: 109 bytes).
[PerfettoCds.PerfettoDataProcessor]: Verbose - Query for PerfettoStackProfileSymbol completed in 0.0143257s at 03/03/2022 06:21:00.169 UTC
[PerfettoCds.PerfettoDataProcessor]: Verbose - trace_processor_shell.exe: [166.811] httpd.cc:193            [HTTP] Client disconnected

In the end I get zero graphs from the example_android_trace_15s.pftrace

ETL files load fine.

[Feature request] Perfetto - Display browser tab title alongside renderer view

Thanks for this great tooling!

Feature request: It would be useful to display the corresponding browser tab title of a renderer. The Perfetto UI has that displayed and I find it a useful information for identifying interested processes/threads.

LTTng - UnsupportedFileException when trying to process LTTng trace folder (.ctf file works)

Seen in Release 0.8.0

LTTngDriver throws this error when trying to process a LTTng trace folder. The same folder zipped as .ctf file works. This used to work in the past which I re-verified on the same trace with an old drop. The same issue/error repro's in the WPA UI when opening the folder.

(base) DESKTOP:~/LTTngDriverDebug/netcoreapp3.1$ ./LTTngDriver.exe ~/LTTngDriverDebug/netcoreapp3.1/kernel
ExtensionDirectory:
Loaded all in 00:00:00.7319867
CTF Path:/home/ivberg/LTTngDriverDebug/netcoreapp3.1/kernel
Microsoft.Performance.Toolkit.Engine.UnsupportedFileException: The given file is not supported by any known processors
at Microsoft.Performance.Toolkit.Engine.Engine.AddFile(String filePath)
at LTTngDriver.Program.Run() in C:\src\Microsoft-Performance-Tools-Linux\LTTngDriver\Program.cs:line 59
at LTTngDriver.Program.Main(String[] args) in C:\src\Microsoft-Performance-Tools-Linux\LTTngDriver\Program.Main.cs:line 15
FilePath: /home/ivberg/LTTngDriverDebug/netcoreapp3.1/kernel

[Feature] Perfetto - Support Processing of Block Disk I/O events into processed table

Feature request for the Perfetto plugin to support processing Block I/O and Ext4 events into their own table.

Would give useful graphs and processed data like we have for LTTng like:

• IOs be Devices, ThreadId, Cmd
• Disk IO Utilization %
• IO Latencies Over Times
• Throughput - IOPS
• Throughput - MBps

Perfetto Android traces via FTrace can already capture Disk I/O events. These events do already show up in the Perfetto plugin as individual events in "Perfetto FTrace Events". Examples:
ext4_da_write_begin
ext4_da_write_end
block_rq_issue
block_rq_complete
ext4_sync_file_enter
ext4_sync_file_exit

These raw events are present in our Perfetto plugin, and give similar info as LTTng. They are just not useful unless correlated together to present more human useful views.

They don't seem to have 1st class support in Perfetto trace processor and end up as raw events in Perfetto SQL - https://perfetto.dev/docs/analysis/sql-tables#raw.

The work would be to do similar post processing as LTTng on these events based on the PerfettoFtraceEventCooker. Some of the work should be replicated, and some refactored into common code to share between the 2 implementations.