dotnet/diagnostics

This repository contains the source code for various .NET Core runtime diagnostic tools and documents.

.NET Core Diagnostics Repo

This repository contains the source code for various .NET Core runtime diagnostic tools. It currently contains SOS, the managed portion of SOS, the lldb SOS plugin and various global diagnostic tools. The goals of this repo is to build SOS and the lldb SOS plugin for the portable (glibc based) Linux platform (Centos 7) and the platforms not supported by the portable (musl based) build (Centos 6, Alpine, and macOS) and to test across various indexes in a very large matrix: OSs/distros (Centos 6/7, Ubuntu, Alpine, Fedora, Debian, RHEL 7.2), architectures (x64, x86, arm, arm64), lldb versions (3.9 to 9.0) and .NET Core (all in-development and supported major versions).

Another goal to make it easier to obtain a version of lldb (currently 3.9) with scripts and documentation for platforms/distros like Centos, Alpine, Fedora, etc. that by default provide really old versions.

This repo will also allow out of band development of new SOS and lldb plugin features like symbol server support for the .NET Core runtime and solve the source build problem having SOS.NETCore (managed portion of SOS) in the runtime repo.

See the GitHub Release tab for notes on SOS and diagnostic tools releases.

---

Building the Repository

The build depends on Git, CMake, Python and of course a C++ compiler. Once these prerequisites are installed the build is simply a matter of invoking the 'build' script (build.cmd or build.sh) at the base of the repository.

The details of installing the components differ depending on the operating system. See the following pages based on your OS. There is no cross-building across OS (only for ARM, which is built on x64). You have to be on the particular platform to build that platform.

To install the platform's prerequisites and build:

• Windows Instructions
• Linux Instructions
• MacOS Instructions
• FreeBSD Instructions
• NetBSD Instructions
• Testing on private runtime builds

SOS and Other Diagnostic Tools

• SOS - About the SOS debugger extension.
• dotnet-dump - Dump collection and analysis utility.
• dotnet-gcdump - Heap analysis tool that collects gcdumps of live .NET processes.
• dotnet-trace - Enable the collection of events for a running .NET Core Application to a local trace file.
• dotnet-counters - Monitor performance counters of a .NET Core application in real time.

Useful Links

• FAQ - Frequently asked questions.
• The LLDB Debugger - More information about lldb.
• SOS - More information about SOS.
• Debugging CoreCLR - Instructions for debugging .NET Core and the CoreCLR runtime.
• dotnet/runtime - Source for the .NET Core runtime.
• Official Build Instructions - Internal official build instructions.

Build Status

License

The diagnostics repository is licensed under the MIT license.

diagnostics's Issues

Add tests for RHEL 7.5+, Alpine 3.8+, Debian 9+

To complete the supported Linux distros.

[SOS][Linux] Managed breakpoints are lost after application restart

@Dmitri-Botcharnikov commented on Thu Jul 21 2016

Breakpoints on managed code are lost after application finishes and is restarted again. The issue can be reproduced on Linux/x64 as well as on Linux/ARM:

(lldb) bpmd helloworld4.exe ExceptionTest.Main
Adding pending breakpoints...
(lldb) c
Process 19495 resuming
(lldb) JITTED helloworld4!HelloWorld4.ExceptionTest.Main()
Setting breakpoint: breakpoint set --address 0x00007FFF7D3CD4AF [HelloWorld4.ExceptionTest.Main()]
Process 19495 stopped
* thread #1: tid = 19495, 0x00007fff7d3cd4af JIT(0x6f80c0) at helloworld4.cs:14, name = 'corerun', stop reason = breakpoint 3.1
    frame #0: 0x00007fff7d3cd4af JIT(0x6f80c0) at helloworld4.cs:14
   11               return x / y;
   12           }
   13           static void Main()
-> 14           {
   15               // Input for test purposes. Change the values to see
   16               // exception handling behavior.
   17               double a = 98, b = 0;
(lldb) c
Process 19495 resuming
Attempted divide by zero.
Process 19495 exited with status = 0 (0x00000000) 
(lldb) run
Process 19707 launched: '/home/dima/work/dotnet/demo/runtime/corerun' (x86_64)
Attempted divide by zero.
Process 19707 exited with status = 0 (0x00000000) 
(lldb) bpmd -list
!bpmd pending breakpoint list
 Breakpoint index - Location, ModuleID, Method Token
1 - helloworld4.exe!ExceptionTest.Main+0, 0x00007FFF7C0729B8, 0x06000002
2 - helloworld4.exe!ExceptionTest.Main+0, 0x0000000000000000, 0x00000000

We can see that pending breakpoints are retained across runs, but it seems that no notifications on module loading and code generation are sent to SOS.

---

@Dmitri-Botcharnikov commented on Thu Jul 21 2016

cc @mikem8361 @janvorli @chunseoklee @seanshpark @lucenticus @kvochko @papaslavik

---

@mikem8361 commented on Thu Jul 21 2016

Yes, it looks like those notifications need to be sent/set again on restart.

---

@Dmitri-Botcharnikov commented on Mon Jul 25 2016

@mikem8361 I've tried to set notification flags in each call of LoadClrDebugDll (which in turn called from HandleExceptionNotification) but without success. Am I missing something?

---

@mikem8361 commented on Mon Jul 25 2016

Can you share your changes. There are a lot of pieces that need to be looked at. LoadClrDebugDll is called for everything sos command invocation but putting something inside the "if (s_clrDataProcess == NULL)" would be called only once. Putting something outside it would be called on command (which is probably not that big a deal). The "exception breakpoint" set in g_ExtServices->SetExceptionCallback(HandleExceptionNotfication) may need to be setting again on restart. Not sure if lldb preserves this breakpoint on restart and the callback handler that is set. g_ExtServices->ClearExceptionCallback would need to be called first to clear the global state in the sosplugin.

`createdump` tool does not work with my app crash (Linux ARM32, Debian)

Hi,
I have a console app running on .NET CORE runtime 2.1.3-servicing-26807-02 on a Linux ARM32 Debian SBC (single board computer, like RaspberryPi3), and it will crash on runing for hours, I've got the crash dumps by enable linux core dump by doing like:
add kernel.core_pattern=/home/pi/crash_dumps/core.%e.%p.%h.%t line to sysctl.conf, and also set ulimit -c to unlimited.
and then, I've been told there's no lldb can analysis the dump.

I have to turn to another solution, I created an issue ticket(via commercial service from our company) to Microsoft, the support engineer build a createdump tool (merchanism explained here) for my environment (targeting .NET CORE 2.1.3-servicing-26807-02), but he didn't give me the detail for how to deploy and how use it to collect crash dump via this tool (he is absent for weekend), anyone could help?

My dotnet --info show:

.NET Core runtimes installed:
Microsoft.NETCore.App 2.1.3-servicing-26807-02 [/opt/dotnet/shared/Microsoft.NETCore.App]

which dotnet show:

/usr/local/bin/dotnet

thanks!

Need lldb version that works on Alpine 3.9

The installed by apk doesn't work and even building 5.0.2 (below) doesn't work.

Docker image build file:

RUN apk update

RUN apk add --no-cache \
        autoconf \
        bash \
        build-base \
        clang \
        clang-dev \
        cmake \
        coreutils \
        curl \
        curl-dev \
        gcc \
        gettext-dev \
        gdb \
        git \
        icu-dev \
        krb5-dev \
        libtool \
        libunwind-dev \
        linux-headers \
        make \
        openssl \
        openssl-dev \
        paxctl \
        python \
        python-dev \
        util-linux-dev \
        wget \
        zlib-dev

RUN apk -X https://dl-cdn.alpinelinux.org/alpine/edge/main add --no-cache \
        lttng-ust-dev \
        llvm \
        swig \
        userspace-rcu-dev

# Build and install lldb 5.0.2
RUN apk add --no-cache \
        doxygen \
        libedit-dev \
        libxml2-dev \
        libffi-dev \
        ncurses-dev \
        python2 \
        python2-dev && \
        \
        wget http://releases.llvm.org/5.0.2/cfe-5.0.2.src.tar.xz && \
        wget http://releases.llvm.org/5.0.2/llvm-5.0.2.src.tar.xz && \
        wget http://releases.llvm.org/5.0.2/lldb-5.0.2.src.tar.xz && \
        \
        tar -xf llvm-5.0.2.src.tar.xz && \
        mkdir llvm-5.0.2.src/tools/clang && \
        mkdir llvm-5.0.2.src/tools/lldb && \
        tar -xf cfe-5.0.2.src.tar.xz --strip 1 -C llvm-5.0.2.src/tools/clang && \
        tar -xf lldb-5.0.2.src.tar.xz --strip 1 -C llvm-5.0.2.src/tools/lldb && \
        rm cfe-5.0.2.src.tar.xz && \
        rm lldb-5.0.2.src.tar.xz && \
        rm llvm-5.0.2.src.tar.xz && \
        \
        mkdir llvmbuild && \
        cd llvmbuild && \
        CC=clang CXX=clang++ cmake -DCMAKE_BUILD_TYPE=Release ../llvm-5.0.2.src && \
        make -j $(($(getconf _NPROCESSORS_ONLN)+1)) && \
        cd tools/lldb && \
        make install && \
        cd ../../.. && \
        rm -r llvmbuild && \
        rm -r llvm-5.0.2.src

Add SOS testing for Windows and Linux arm architectures

This is where there seems a lot of problems/instability with lldb itself on arm.

Linux "minidumps" don't have all the metadata necessary to display stack functions

Both Windows and Linux minidumps don't save all the assemblies' metadata which can cause stack traces to display !Unknown. On Windows most debuggers like Windbg and VS can load the module from file for this kind of dumps. On Linux lldb/SOS doesn't load the native modules and really doesn't know anything about the managed assemblies.

There is an metadata provider interface that could be used to fix this for SOS on Linux.

Creating a proper version file (_version.h on Windows, version.cpp on xplat) for official builds

On Windows this is how sos.dll's version resources are defined. On Linux, this version is embedded as a string.

Microsoft.DiaSymReader.Native package needs an arm64 version

Need to add an arm64 version of the symbol reader to this package.

GC SOS commands are failing on macOS.

In the SOS.UnitTests, various commands are failing on macOS. These failures need to be fixed and that the SOSRunner doesn't detect these kind of failures.

(lldb) runcommand sos DumpObj -refs 000000018201fa58
Name:        GCWhere
MethodTable: 00000001084251a8
EEClass:     0000000109031098
Size:        24(0x18) bytes
File:        /Users/mikem/diagnostics/artifacts/Debuggees/GCWhere/bin/Debug/netcoreapp2.0/publish/GCWhere.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
0000000108dc5ba0  4000001        8        System.String  0 instance 000000018201fa18 _string
GC Refs:
              offset object
Failed to request number of entries.
DumpObj -refs 000000018201fa58  failed

None

(lldb) runcommand sos GCRoot -all 000000018201fa58
Failed to request number of entries.Thread 5948bc:
    00007FFF5FBF6F00 00000001090404E2 GCWhere.Main() [/Users/mikem/diagnostics/artifacts/Debuggees/GCWhere/GCWhere.cs @ 33]
        rbp-18: 00007fff5fbf6f08
            ->  000000018201FA58 <unknown>

    00007FFF5FBF6F00 00000001090404E2 GCWhere.Main() [/Users/mikem/diagnostics/artifacts/Debuggees/GCWhere/GCWhere.cs @ 33]
        rbp-8: 00007fff5fbf6f18
            ->  000000018201FA58 <unknown>

Failed to request number of entries.Failed to request number of entries.Failed to request number of entries.Failed to request number of entries.Found 2 roots.

None

Bundle libsos with coreclr builds for the dotnet cli

@TyOverby commented on Tue Mar 13 2018

I'm trying to use lldb/libsos to debug a dotnet core application, but the instructions ask me to build llvm, lldb, and presumably coreclr from scratch.

Would it be possible to package the correct version of libsos with dotnet cli installations?

---

@jkotas commented on Wed Mar 14 2018

cc @mikem8361

---

@mikem8361 commented on Wed Mar 14 2018

Are you talking about MacOS? Because that is the only platform that requires building lldb so libsosplugin.dylib can be built in the coreclr repo. libsos.dylib (the actual/core part of sos) is already built.

If it is a Linux platform, then the cli installs contain the libsosplugin.so that works with lldb 3.9 and of course libsos.so is also include.

The problem on MacOS is that libs/include files necessary to build the libsosplugin.dylib that works with the lldb that comes with MacOS are not available.

sosplugin is not working on ARM

@wfurt commented on Thu Jul 12 2018

I tested this in my RP3 as well as Ubuntu on Qemu. I'm not even able to get list of commands.

pi@raspberrypi:~/aa $ lldb-4.0 ../dotnet-2.1.302/dotnet -- bin/Debug/netcoreapp2.1/aa.dll
(lldb) target create "../dotnet-2.1.302/dotnet"
Current executable set to '../dotnet-2.1.302/dotnet' (arm).
(lldb) settings set -- target.run-args  "bin/Debug/netcoreapp2.1/aa.dll"
(lldb) r
Process 16576 launched: '../dotnet-2.1.302/dotnet' (arm)
Hello World!
Going to sleep!
Process 16576 stopped
* thread #1, name = 'dotnet', stop reason = signal SIGSTOP
    frame #0: 0x76f88ce8
->  0x76f88ce8: svc    #0x0
    0x76f88cec: cmn    r0, #4096
    0x76f88cf0: mov    r5, r0
    0x76f88cf4: ldrls  r4, [sp]
(lldb) plugin load ../dotnet-2.1.302/shared/Microsoft.NETCore.App/2.1.2/libsosplugin.so
(lldb) sos
(lldb) sos help
Stack dump:
0.	HandleCommand(command = "sos help")
Segmentation fault

I did same steps on x64 and lldb-4.0
(lldb) sos

SOS is a debugger extension DLL designed to aid in the debugging of managed
programs. Functions are listed by category, then roughly in order of
...
...

(lldb) sos help
SOS command 'help' not found /home/furt/dotnet-2.1/shared/Microsoft.NETCore.App/2.1.0/libsos.so: undefined symbol: help

---

@abbotware commented on [Thu Jul 12 2018](https://github.com/dotnet/coreclr/issues/18889#issuecomment-404672814)

FYI - Same problem I have on aarch64 using lldb-3.8(armhf) and lldb-3.9(arm64) - i installed both 32 and 64 bit on the same machine just to see if it was a 32bit vs 64bit library loading issue

---

@RussKeldorph commented on [Thu Jul 12 2018](https://github.com/dotnet/coreclr/issues/18889#issuecomment-404718273)

@echesakovMSFT 

---

@wfurt commented on [Thu Jul 12 2018](https://github.com/dotnet/coreclr/issues/18889#issuecomment-404730466)

just a note that it seems to work when I build it my self. (#18827) I'm suspicious this may be build problem . 

---

@echesakovMSFT commented on [Fri Jul 13 2018](https://github.com/dotnet/coreclr/issues/18889#issuecomment-404870130)

I also ended up building lldb5.0.2+sosplugin for arm myself. In my case "official" lldb couldn't even run target process with the following message `error: process launch failed: Lost debug server connection`.

---

@abbotware commented on [Tue Jul 24 2018](https://github.com/dotnet/coreclr/issues/18889#issuecomment-407495386)

It sounds like there are issues, but is there a process to get to a known working state so I can at least get some deadlock / hang analysis done on Linux Arm32?

From what I am reading, if I do the following:

Build LLDB 5.0.2?
Build SOS Plugin? (which source version)
Build .Net Core?  

on my 'target' hardware, should I have a working way to do dump analysis? (or attach to a hung process)

---

@wfurt commented on [Tue Jul 24 2018](https://github.com/dotnet/coreclr/issues/18889#issuecomment-407499242)

I simply build coreclr with what ever version of lldb you plan to use (5.0 in my case) 
You do not need to build lldb it self but you need lldb-dev (or equivalent package)

Also note, that on arm32 you can use Visual Studio for remote debug.  

dbgshm does not allow creation in a separate console

Hi,

I am implementing a debugger for .net core, but also written on .net core 2.0 and I am finding it not possible to call CreateProcess on the debug shim, and create a separate console for the application to run inside.

I think that it could be this line:
https://github.com/dotnet/coreclr/blob/master/src/dlls/dbgshim/dbgshim.cpp#L92

CREATE_NEW_CONSOLE
0x00000010

We would need to extend the shim api to allow creating a separate console. or is this by design?

LLDB hangs in Linux when opening a dotnet core 2.1 dump

Hi there,

I want to get a process dump of an application that is consuming more memory than expected and do a simple !dupmheap to try to figure out the problem.

Please check this self-contained example. It will create an image with all the tools already installed, then create a dotnet core 2.1 application that runs in a continuous loop printing messages with a 1s sleep between messages, and use it as entry point:

FROM microsoft/dotnet:2.1-sdk
WORKDIR /app
RUN apt-get update && apt-get install curl -y && \ 
    apt-get install gpg -y && \
    apt-get install apt-transport-https -y && \
    curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg && \
    mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg && \
    sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-xenial-prod xenial main" > /etc/apt/sources.list.d/microsoft.list' && \
    apt-get update && \
    apt-get install procdump -y && \
    apt-get install lldb-3.9 -y
RUN dotnet tool install -g dotnet-symbol && \
    export PATH="$PATH:/root/.dotnet/tools" && \
    dotnet symbol coredump && \
    dotnet new console && \
    echo 'using System; namespace app { class Program { static void Main(string[] args) { while(true) { Console.WriteLine($"{DateTime.Now:HH:mm:ss} Hello!"); System.Threading.Thread.Sleep(1000);} } } } '  > Program.cs && \ 
    dotnet build -c Release
WORKDIR /app/bin/Release/netcoreapp2.1
ENTRYPOINT [ "dotnet", "app.dll" ]

Then you need to run it with ptrace support and connect to it:

$ docker build --no-cache --rm -t lldb-poc .
$ docker run --cap-add=SYS_PTRACE --security-opt seccomp:unconfined -d --name lldb-poc-c lldb-poc 
$ docker exec -it --privileged lldb-poc-c /bin/bash

Once in, you will see that the application has the PID 1 (as usual when it is the entry point).
You can take the dump using procdump that is also installed:

# procdump --pid 1

ProcDump v1.0.1 - Sysinternals process dump utility
Copyright (C) 2017 Microsoft Corporation. All rights reserved. Licensed under the MIT license.
Mark Russinovich, Mario Hewardt, John Salem, Javid Habibi
Monitors a process and writes a dump file when the process exceeds the
specified criteria.

Process:                dotnet (1)
CPU Threshold:          n/a
Commit Threshold:       n/a
Threshold Seconds:      10
Number of Dumps:        1

Press Ctrl-C to end monitoring without terminating the process.

[07:31:51 - INFO]: Timed:
[07:31:54 - INFO]: Core dump 1 generated: dotnet_time_2018-10-22_07:31:51.1

Now that the dump is take, I try to open it in LLDB...

# lldb-3.9 `which dotnet` -c dotnet_time_2018-10-22_07\:31\:51.1
(lldb) target create "/usr/bin/dotnet" --core "dotnet_time_2018-10-22_07:31:51.1"

But it hangs forever... the prompt never gets to (lldb) again. I tried several variants, like loading the SOS plugin first, and then do the target create after, but the outcome is always the same.

I tried also to attach to the running process, but no luck:

# lldb-3.9 `which dotnet`
(lldb) target create "/usr/bin/dotnet"
Current executable set to '/usr/bin/dotnet' (x86_64).
(lldb) plugin load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/libsosplugin.so
(lldb) process attach -p 1
error: attach failed: unable to attach

Could you please give me some light on how to explore the process' memory with LLDB in linux?

Add symbol server support to SOS.NETCore

Add symbol server support to the lldb plugin. First, download PDBs for managed assemblies in SOS.NETCore. Second, download SOS/DAC/DBI for the version of libcoreclr loaded in the libsosplugin.

Document and assert the common coreclr/diagnostics structs/include files

Identify and document all the common coreclr/daignostic structs/include files (both in a summary doc and in the header file itself) like msodw.h, gcinfo.h, etc.

CLRStack -i in SOS causes SegmentationFault on LLDB / Linux

@chsienki commented on Tue Mar 27 2018

ClrStack -i is causing a segmentation fault in SOS on Linux

Repro Steps:

Environment:
Ubuntu 14.04
LLDB 3.8.1
CLR 2.1.0-preview3-26324-03

lldb steps:
target create pathtoclr/corerun
settings append target.run-args netcoreexe
plugin load pathtoclr/libsosplugin.so

Step or continue until libcoreclr.so is loaded

sos ClrStack -i

Expected Output:

Dumping managed stack and managed variables using ICorDebug

Actual Output:

Segmentation fault (core dumped)

---

@mikem8361 commented on Tue Mar 27 2018

This may be that the libsosplugin.so is built for lldb 3.9 for .NET Core 2.1 and we are stilling testing against 3.8.1.

Add/enable embedded PDB in the SOS tests

We only current test on Windows and external Portable PDBs

Investigate building the macOS lldb plugin

There is currently no easy way to build the libsosplugin for macOS for the lldb version that comes with the OS. This also prevents any SOS testing on Mac.

Move and build the lldb plugin into diagnostics repo

Move the Linux/MacOS lldb plugin (#3) into the diagnostic repo (not started yet).

• Build for all the supported versions of lldb so instead of being locked into lldb 3.9 we can support whatever version is available for the Linux distro as long as it is greater than 3.8.
• I also hope to solve the MacOS lldb plugin build problems and provide version that works with the installed MacOS lldb.

minidump can't tell source/line info for the managed stack frame

env:

COMPlus_DbgEnableMiniDump=1
COMPlus_DbgMiniDumpType=4
DOTNET_SDK_VERSION=2.1.401
DOTNET_USE_POLLING_FILE_WATCHER=true
DOTNET_RUNNING_IN_CONTAINER=true

program:

using System;

namespace CrashApp
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
	    throw new Exception("crasing");
        }
    }
}

my debugcore script:

#!/usr/bin/env bash

if [ "$#" -ne 1 ]; then
    echo 'USEAGE: debugcore $CORE_FILE_PATH'
    exit 0
fi

COREFILE=$1
RUNTIME_PATH='/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.3/'
PATH_TO_LIBSOSPLUGIN='/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.3/libsosplugin.so'
HOST_PATH='/usr/share/dotnet/dotnet '

echo $COREFILE


lldb-3.9 -O "settings set target.exec-search-paths $RUNTIME_PATH" -o "plugin load $PATH_TO_LIBSOSPLUGIN" --core $COREFILE $HOST_PATH

(lldb) bt
* thread #1: tid = 157, 0x00007fcfe0972b3a libpthread.so.0`__waitpid + 106, name = 'dotnet', stop reason = signal SIGABRT
  * frame #0: 0x00007fcfe0972b3a libpthread.so.0`__waitpid + 106
    frame #1: 0x00007fcfdf498d3e libcoreclr.so`::PROCCreateCrashDumpIfEnabled() + 110 at process.cpp:3039
    frame #2: 0x00007fcfdf498d6e libcoreclr.so`::PROCAbort() + 14 at process.cpp:3065
    frame #3: 0x00007fcfdf497bdb libcoreclr.so`PROCEndProcess(hProcess=<unavailable>, uExitCode=<unavailable>, bTerminateUnconditionally=<unavailable>) + 235 at process.cpp:1394
    frame #4: 0x00007fcfdf207668 libcoreclr.so`UnwindManagedExceptionPass1(ex=<unavailable>, frameContext=<unavailable>) + 760 at exceptionhandling.cpp:4681
    frame #5: 0x00007fcfdf2077e3 libcoreclr.so`DispatchManagedException(ex=<unavailable>, isHardwareException=<unavailable>) + 275 at exceptionhandling.cpp:4752
    frame #6: 0x00007fcfdf1841da libcoreclr.so`IL_Throw(obj=<unavailable>) + 714 at jithelpers.cpp:4862
    frame #7: 0x00007fcf6604171b // EXPECTED symbol/source/line etc
    frame #8: 0x00007fcfdf213eaf libcoreclr.so`CallDescrWorkerInternal + 124
    frame #9: 0x00007fcfdf1345dc libcoreclr.so`MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 83 at callhelpers.cpp:78
    frame #10: 0x00007fcfdf134589 libcoreclr.so`MethodDescCallSite::CallTargetWorker(this=<unavailable>, pArguments=<unavailable>, pReturnValue=0x0000000000000000, cbReturnValue=0) + 857 at callhelpers.cpp:620
    frame #11: 0x00007fcfdf235046 libcoreclr.so`RunMain(MethodDesc*, short, int*, PtrArray**) [inlined] MethodDescCallSite::Call(this=<unavailable>, pArguments=<unavailable>) + 23 at callhelpers.h:433
    frame #12: 0x00007fcfdf23502f libcoreclr.so`RunMain(MethodDesc*, short, int*, PtrArray**) + 340 at assembly.cpp:1705
    frame #13: 0x00007fcfdf234edb libcoreclr.so`RunMain(MethodDesc*, short, int*, PtrArray**) + 87 at assembly.cpp:1720
    frame #14: 0x00007fcfdf234e84 libcoreclr.so`RunMain(pFD=<unavailable>, numSkipArgs=<unavailable>, piRetVal=<unavailable>, stringArgs=<unavailable>) + 292 at assembly.cpp:1720
    frame #15: 0x00007fcfdf2352e5 libcoreclr.so`Assembly::ExecuteMainMethod(this=0x00000000020f3600, stringArgs=0x00007ffeeda6e488, waitForOtherThreads=YES) + 229 at assembly.cpp:1817
    frame #16: 0x00007fcfdf07de92 libcoreclr.so`CorHost2::ExecuteAssembly(this=<unavailable>, dwAppDomainId=<unavailable>, pwzAssemblyPath=<unavailable>, argc=<unavailable>, argv=<unavailable>, pReturnValue=<unavailable>) + 482 at corhost.cpp:491
    frame #17: 0x00007fcfdf058fb4 libcoreclr.so`::coreclr_execute_assembly(hostHandle=<unavailable>, domainId=<unavailable>, argc=<unavailable>, argv=<unavailable>, managedAssemblyPath=<unavailable>, exitCode=<unavailable>) + 212 at unixinterface.cpp:407
    frame #18: 0x00007fcfdf83087d libhostpolicy.so`coreclr::execute_assembly(host_handle=0x0000000002110060, domain_id=1, argc=0, argv=0x0000000000000000, managed_assembly_path="/workspace/CrashApp/bin/Debug/netcoreapp2.1/CrashApp.dll", exit_code=0x00007ffeeda6e97c) + 125 at coreclr.cpp:104
    frame #19: 0x00007fcfdf8232fd libhostpolicy.so`run(args=0x00007ffeeda6ed30, out_host_command_result=error: summary string parsing error) + 6781 at hostpolicy.cpp:289
    frame #20: 0x00007fcfdf823b4c libhostpolicy.so`::corehost_main(argc=2, argv=0x00007ffeeda6fca8) + 236 at hostpolicy.cpp:386
    frame #21: 0x00007fcfdfad6cbf libhostfxr.so`___lldb_unnamed_symbol1564$$libhostfxr.so + 207
    frame #22: 0x00007fcfdfae236c libhostfxr.so`___lldb_unnamed_symbol1609$$libhostfxr.so + 5436
    frame #23: 0x00007fcfdfae34c5 libhostfxr.so`___lldb_unnamed_symbol1612$$libhostfxr.so + 437
    frame #24: 0x00007fcfdfae2a39 libhostfxr.so`___lldb_unnamed_symbol1610$$libhostfxr.so + 1001
    frame #25: 0x00007fcfdfad6f0c libhostfxr.so`hostfxr_main_startupinfo + 156
    frame #26: 0x000000000040ac74 dotnet`___lldb_unnamed_symbol30$$dotnet + 1572
    frame #27: 0x000000000040af05 dotnet`___lldb_unnamed_symbol31$$dotnet + 165
    frame #28: 0x00007fcfdfd462e1 libc.so.6`__libc_start_main + 241
    frame #29: 0x0000000000408a54 dotnet`_start + 41

in frame 7, no symbol/source/line info:

    frame #7: 0x00007fcf6604171b 

Make it easy to acquire all SOS binaries needed for debugging

@noahfalk commented on Mon Apr 09 2018

Tracking a request originally made here: https://github.com/dotnet/cli/issues/8872

Depending on the OS, debugger and runtime a developer needs different versions of the sos binaries (sos, libsos, libsosplugin). We are working to make it easier to acquire these binaries and filed this work item to track the work. I imagine a few things may need to happen (sub-divide this issue if needed, but retain this one to track overall status)

• Produce any binary variations that are currently non-existent
• Figure out the most convenient mechanism for distributing the binaries (likely in the SDK?)
• Provide documentation

---

@mikem8361 commented on Mon Apr 09 2018

Customer feedback:

Using SOS with LLDB is very difficult for a few reasons

• Finding the correct version of LLDB is not simple for all the Linux distributions. I had problems with CentOS which is very popular in the server environment (x64)
• The SOS commands are slightly different from the ones used on Windows. Therefore, even if you already are familiar with SOS on Windows, it is hard to understand how to use them
• I said to the audience that SOS should not be their primary debugging choice, but also that is extremely important to have this tool because it is the last resort if they would ever need to debug in production. So, it is good to have even if it is not the best ever developer experience.

---

@danmosemsft commented on Mon Apr 09 2018

@janvorli

---

@ghost commented on Tue Apr 10 2018

Coming from aspnet/DotNetTools#428, can SOS be also published as an official dotnet-cli tool?

@mikem8361, can you explain a bit what kind of version issues did you notice? With current state of CoreCLR master, libsos builds with LLB 3.[4-9], 4.0, 5.0 and the latest 6.0 and works with any one of them available at runtime. Perhaps things have improved in package managers for llvm toolchain?

---

@janvorli commented on Tue Apr 10 2018

@kasper3 Until lldb 3.9, the plugin built for previous versions was crashing in new versions. It seems that the interface has either stabilized or just not changed since.

---

@ghost commented on Tue Apr 10 2018

@janvorli, would it help if we abstract out the methods (of interest) in a shim like openssl shim in corefx?

---

@mikem8361 commented on Tue Apr 10 2018

The complains seems to be mostly about arm versions and MacOS except the CentOS x64 in the above which I don't understand exactly. Like you and Jan said the current plugin works on a lot wider set of lldb versions.

Add tests for OpenBSD, NetBSD and FreeBSD

At least of these BSDs in the internal builds.

SOS: Wide char output

@bendono commented on Wed Oct 12 2016

While recently debugging an issue, I noticed that SOS was unable to output Unicode text properly. This has resulted in various obscure workarounds such replacing non-ASCII characters with dots as seen here. Note that printf here is really ExtOut, which uses IDebugControl2 to output text. The problem is that IDebugControl2 only supports multibyte strings. A better option is to use IDebugControl4 which also supports Unicode strings.

I would suggest the following changes which fallback to IDebugControl2 if the IDebugControl4 interface is for whatever reason unavailable:

void ExtOut(PCWSTR Format, ...)
{
    if (Output::IsOutputSuppressed())
        return;

    // Get instance of IDebugControl4 to output wide strings.
    IDebugControl4* debugControl4 = nullptr;
    if (SUCCEEDED(g_ExtClient->QueryInterface(__uuidof(IDebugControl4), (void**)&debugControl4)))
    {
        ExtOutIndent();

        va_list Args;
        va_start(Args, Format);
        debugControl4->OutputVaListWide(DEBUG_OUTPUT_NORMAL, Format, Args);
        va_end(Args);

        EXT_RELEASE(debugControl4);
    }
    else
    {
        // Unable to obtain IDebugControl4. Use IDebugControl as an alternative.
        // Convert wide string to multibyte string.
        IDebugControl* debugControl = nullptr;
        if (SUCCEEDED(g_ExtClient->QueryInterface(__uuidof(IDebugControl), (void**)&debugControl)))
        {
            // Convert wide string to multibyte.
            // Determine required size of multibyte buffer.
            int bufferSize = WideCharToMultiByte(CP_ACP, 0, Format, -1, NULL, 0, NULL, NULL);

            if (bufferSize != 0)
            {
                LPSTR multibyteFormat = new CHAR[bufferSize];

                if (WideCharToMultiByte(CP_ACP, 0, Format, -1, multibyteFormat, bufferSize, NULL, NULL) != 0)
                {
                    ExtOutIndent();

                    va_list Args;
                    va_start(Args, Format);
                    debugControl->OutputVaList(DEBUG_OUTPUT_NORMAL, multibyteFormat, Args);
                    va_end(Args);
                }

                delete multibyteFormat;
            }

            EXT_RELEASE(debugControl);
        }
    }
}

Questions:

1. Would a pull request be accepted changing ExtOut(PCSTR format, ...) to ExtOut(PCWSTR format, ...)? Similar changes would be needed for ExtWarn, ExtErr, and DMLOut.
2. This change would require prefixing MANY string literals as Unicode. Is there a preference for L, W, T(), TEXT() or another such prefix? This change alone is the largest, so I'd prefer to get this right the first time.
3. Are modern techniques such as variadic templates permitted in the codebase? This could enable code share between the ExtOut, ExtWarn, and ExtErr which only differ in the DEBUG_OUTPUT_* level.
4. Another option is to leave ExtOut, ExtWarn and ExtErr alone but add new ExtOutEx, ExtWarnEx, and ExtErrEx functions that take a wide string. Or just add overloads that accept wide strings. Then select portions of the code can be updated to use the newer functions as necessary. This would result in a lot less code churn.

Please let me know your opinions.

---

@jkotas commented on Wed Oct 12 2016

cc @mikem8361

---

@mikem8361 commented on Tue Oct 18 2016

I wouldn't recommend actually changing the existing output functions, but either add the Unicode overrides or new "Ex" functions like you suggest.

But the biggest problem is that for xplat support (Linux/OS X) the IDebugControl2 is emulated on top of LLDB apis. It will take some more work to write the Unicode output functions on top of the LLDB apis. The LLDB output functions only takes ANSI strings so conversion needs to be done. And the LLDB/C++ runtime defines Unicode strings (wchar_t *) as UTF32 instead of UTF16 which is what SOS uses.

All this adds up to a chunck of work that isn't that high of priority compared to my other workitems. I might eventually get to it but I have no time frame right now.

LLDB and the SOS tests fail on the Windows Subsystem for Linux

lldb doesn't seem to work on WSL but gdb does. This may be WSL or lldb.

lldb hangs on "process continue" and the test timeouts.

Add the SOS tests for WIndows/Linux from the diagnostictests repo

This would move the SOS test runner and some of the common test runner logic and the SOS scripts from the private diagnostictests repo to the diagnostics repo.

This will include Windows SOS test coverage and more Linux coverage.

Document and assert any SOS/coreclr dependencies

The include files that are now shared between coreclr and the diagnostics repo need to be documented and asserted if possible.

sethostruntime cmd problem on Linux

The tracepointprovider .so module gets registered twice and fails when the host runtime is different from the target runtime. Not a problem on macOS.

Implement/port clru -gcinfo option on xplat SOS

@mikem8361 commented on Mon May 02 2016

The current windows implementation of the -gcinfo option support uses Windows Fibers which need to be removed/recoded.

This seriously impacts our ability to investigate heap corruption failures from our stress runs and partner teams.

SOS tests fail when cdb has debug symbols for CoreCLR/CoreRun

When running the debugger tests in a configuration where cdb loads symbols for CoreCLR and/or CoreRun, SOSRunner gets confused by templated functions in the output of !DumpStack. In particular, templated destructors and lambdas both can end in a >, which causes SOSRunner to incorrectly recognize these lines as containing a prompt. Two example functions that confuse SOSRunner under the current scheme: * CoreCLR!CAutoTryCleanup::~CAutoTryCleanup * CoreRun!__acrt_lock_and_call< > (apparently cdb adds an extra space between >s when demangling the lambda argument to the template) When I hardcode the prompt to be 0:000>, which it always seems to be on my machine, the tests pass, though I'm not sure that's the correct solution.

The fix will require a way to run commands and print something when the command is finished that isn't so ambiguous. Similar to what we do for LLDB.

[Linux] How to view static fields of generics in SOS?

@ayuckhulk commented on Mon May 29 2017

Consider the following code:

class Program
{
	public static void Main()
	{
        S1<int> s1 = new S1<int>();
        s1.bar = 2;
	}

    public class S1<T>
    {
        public static int fooInt = 99;
        public T bar;
    }
}

I'd like to view the value of s1.fooInt. However, SOS plugin for lldb does not show its value or native address:

(lldb) sos ClrStack -i s1



Dumping managed stack and managed variables using ICorDebug.
=============================================================================
Child SP         IP               Call Site
00007FFFFFFFC340 00007fff7d7408dd [DEFAULT] Void Program.Main() (/home/i.kulaychuk/work/overlay/Program.dll)

PARAMETERS: (none)

LOCALS:
  + Program+S1`1<int> s1 @ 0x7fff580337d0
     |- < unknown type > fooInt
     |- int bar  = 0

00007FFFFFFFC360 00007ffff62a1f87 [NativeStackFrame]
Stack walk complete.
=============================================================================

(lldb) sos DumpObj 0x00007fff580337d0
Name:        Program+S1`1[[System.Int32, System.Private.CoreLib]]
MethodTable: 00007fff7c196b00
EEClass:     00007fff7d792808
Size:        24(0x18) bytes
File:        /home/i.kulaychuk/work/overlay/Program.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007fff7d471740  4000002        8         System.Int32  1 instance                0 bar
00007fff7d471740  4000001       10         System.Int32  1   static dynamic statics NYI                 fooInt
(lldb) 

The command sos ClrStack -i s1 uses ICorDebug API ICorDebugType::GetStaticFieldValue to show static fields (see https://github.com/dotnet/coreclr/blob/master/src/ToolBox/SOS/Strike/strike.cpp#L11734). But for some reason it does not work for dynamic statics.
Is it a bug? Is there any other way to get dynamic static fields?

CC @jkotas, @janvorli, @mikem8361, @Dmitri-Botcharnikov

---

@jkotas commented on Mon May 29 2017

I do not think that there is anything fundamental why this cannot work. It is just unimplemented.

---

@ayuckhulk commented on Mon May 29 2017

@jkotas The API ICorDebugType::GetStaticFieldValue actually returns CORDBG_E_STATIC_VAR_NOT_AVAILABLE. However, C# debugger (clrdbg) is always able to show static variable value. Does it force the runtime to initialize the dynamic static field of the class? Is it by design that ICorDebugType::GetStaticFieldValue does not initialize the static field of the class? Is there another API to force the initialization?

Sorry, I got confused with ICorDebur*, IDacDbiInterface* etc. APIs 😄

---

@0xd4d commented on Mon May 29 2017

You'll need to create an instance of the generic class, see this comment in rsclass.cpp:

// The debugger may chose to func-eval the creation of an instance of this type and try again.

---

@ayuckhulk commented on Mon May 29 2017

@0xd4d The instance of the class is already created (when I put a breakpoint on line s1.bar = 2;) but ICorDebugType::GetStaticFieldValue still returns CORDBG_E_STATIC_VAR_NOT_AVAILABLE. What API should I use to func-eval the creation of an instance of this type?

---

@ayuckhulk commented on Tue May 30 2017

@jkotas Is this https://blogs.msdn.microsoft.com/davidnotario/2005/02/08/jit-compiler-and-type-constructors-cctors/ the reason why static fields are not available? Then I guess I need to call the .cctor method using ICorDebugEval API.

---

@jkotas commented on Tue May 30 2017

The short answer is that many things in the runtime are done lazily, including allocation of storage for static fields.

Then I guess I need to call the .cctor method using ICorDebugEval API.

If you want to mimic the Visual Studio experience, then yes - you would need to run the .cctor. It is not just to allocate the storage for the fields, it is also to actually run the .cctor if it has not run already.

---

@ayuckhulk commented on Tue May 30 2017

@jkotas I tried to use ICorDebugEval for .cctor from SOS but it failed with E_NOTIMPL. I guess that's because SOS uses V4 debug API and in order for ICorDebugEval to work the API should be V2.
Is there any working API to actually run .cctor and other methods (like getters for properties)? Or am I missing something? 😄

---

@jkotas commented on Tue May 30 2017

It is because of funceval requires infrastructure that was never implemented for the low-level SOS debugger. I do not think it has anything to do with V2 vs. V4 debug APIs.

---

@ayuckhulk commented on Tue May 30 2017

@jkotas Could you please point me to the things that should be implemented to get funceval working in SOS?

---

@jkotas commented on Tue May 30 2017

The funceval is pretty complex. You can an idea about what is involved from https://blogs.msdn.microsoft.com/jmstall/tag/funceval/ .

---

@ayuckhulk commented on Tue May 30 2017

Thank you!

Debugging instructions should clearly carry information as to how acquire symbols

@lt72 commented on Thu Nov 02 2017

Debugging instructions should carry the information on how to acquire symbols. Whether that happens through a tool, or by manual download, or both, all locations of the symbols should be clearly documented.
E.g.: the debugging instruction page should carry a link to the core-setup and the download links page on DLC.

Can not load or initialize libmscordaccor.so when debugging a coredump in lldb

@JoeStead commented on Tue May 01 2018

We have a coredump from our application that was built using the .NET Core 2.0.3 SDK, a standalone app. The application was running on CentOS7.

To perform some analysis on the dump, I've exported the file onto an Ubuntu 16.04 VM, with lldb-3.6 installed, and the same .NET Core SDK Version that was used to build the application.

I'm launching lldb like this:

lldb-3.6 /usr/bin/dotnet --core ./mydumpfilehere

The dump is loaded, and I'm able to use the standard lldb commands, so far, so good. Now I want to load the sos plugin, which I do by running:

plugin load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libsosplugin.so

Now, if I type help, I see the commands from the plugin.

Finally, I setclrpath to setclrpath /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3 and to my knowledge, I should be good to go.

Now, as soon as I use any sos command, I get the following message:

Failed to load data access DLL, 0x80004005
Can not load or initialize libmscordaccore.so. The target runtime may not be initialized.

I thought, that perhaps, the SDK versions weren't quite matching up, so I built the coreclr repo, and used the plugin from there instead, still with no luck.

Finally, I thought I might be able to use the plugin that's actually shipped with our self contained app, again, that does not work.

The only thing I can think of, is I need to analyse this on a CentOS 7 machine, but my (limited) understanding is that shouldn't matter?

---

@JoeStead commented on Tue May 01 2018

Thought I'd give 2.1 a go too. Downloaded the 2.1 preview, built a standalone app for Centos, which just printed hello world and threw an exception. Used (still on Ubuntu 16.04) lldb-3.9 to load the dump, and load the plugin from the 2.1 install, exactly the same issue.

---

@RussKeldorph commented on Tue May 01 2018

@dotnet/dotnet-diag

---

@mikem8361 commented on Tue May 01 2018

I think the problem might be creating the dump on Centos but using the Ubuntu runtime/DAC to load it even though it should be the same runtime unless it is a slight different build. It does look like you are using 2.0.3 on both Centos and Ubuntu. I'll try creating a dump on Centos and loading it on my Ubuntu 14.04 machine to see what might be the problem.

Another problem you will run into is using lldb 3.6 on to load the core dump. SOS doesn't work on lldb 3.6 on core dumps. You need 3.9. The problem is that the libsosplugin.so is built for 3.6 on Centos/Ubuntu for 2.0.x. Attached is a 3.9 libsosplugin.so if you want to try that.

For .NET Core 2.1 libsosplugin.so is built for lldb 3.9.

libsosplugin.zip

---

@JoeStead commented on Wed May 02 2018

Thanks for the plugin

If there is a problem with creating a dump on Centos and loading on Ubuntu, am I right in thinking I could compile lldb on Centos and hopefully everything will work? Something I've been keen to avoid because it takes so long to build, but needs must :)

---

@mikem8361 commented on Wed May 02 2018

I have repro'ed the problem of creating a core dump on Centos 7 and attempting to load it on Ubuntu (14.04). I'm currently investigating.

---

@mikem8361 commented on Wed May 02 2018

I found a simple workaround for now: copy /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so to the directory containing the core dump which I assume is the current directory. That seems to fix the problem. I'll continue to investigate an actual fix.

---

@mikem8361 commented on Wed May 02 2018

The libcoreclr.so needs to be in the same directory as the host ("dotnet" or "corerun") used to run the .NET Core program used to create the dump. When you load a core dump lldb needs the host program as part of the --core command line.

lldb-3.9 --core coredumpfile dotnet

---

@JoeStead commented on Thu May 03 2018

I've not been able to use the workaround you've suggested, just so I'm not misunderstanding the steps:

Use lldb-3.9 (with the 2.0.3 SDK)
Use the version of the plugin you included above
Copy the libcoreclr.so to the same directory as the coredump file
Load the plugin included above
set the clr path?

I'm getting a little confused with the latter steps I think, and it's definitely my fault, this is a whole new world for me

---

@mikem8361 commented on Thu May 03 2018

You have correct steps (yes on setclrpath). The only thing that seems to be missing in your above comments is the host ("dotnet" or "corerun") used to run your test program. lldb doesn't seem to find any of the other modules if I don't add the host program to the --core comand line.

Here is the script I use to load my core dump on Ubuntu:

BUILD_DIR=$HOME/coreclr/bin/Product/Linux.x64.Debug
DOTNET=/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3
HOST=$HOME/save/centosdump/dotnet
$LLDB_PATH -o "plugin load $BUILD_DIR/libsosplugin.so" -o "setclrpath $DOTNET" --core $HOME/save/centosdump/core.43690 $HOST

If you only have 2.0.3 installed then the HOST path should be HOST=/usr/share/dotnet/dotnet

$ ls
core.43690*  dotnet*  loadcore*

Hopefully that is clearer.

---

@mikem8361 commented on Fri May 04 2018

I investigated the problem some more and it is a lldb problem than a plugin/SOS. The lldb APIs that SOS uses to initialize the DAC do not work unless the actual "libcoreclr.so" file matching the one in core dump can be loaded from lldb. Hence requiring libcoreclr.so to be in the directory where the "host" is which is the way lldb finds and loads the module/symbol info for all the modules in the core dump. I've tried all the different symbol and module related settings with no luck. There seems to be no programmatic way to fix this either, but I'll continue to look at this.

---

@raffaeler commented on Mon Jun 04 2018

I tried with the 2.1.300 on Ubuntu18 + lldb-3.9 and the problem is still there.
The easiest workaround is to work with a self contained deployment.
But even in this case, certain commands such as clrstack make lldb crash randomly and you have to restart the debugging from the beginning.

---

@mikem8361 commented on Mon Jun 04 2018

@raffaeler Can you give me some more details? Are you debugging a core dump? Do you have any details on how commands like clrstack make lldb crash randomly (repro steps, etc.)?

I'm currently working on addressing problems that make SOS/lldb unreliable or problematic by increasing the test coverage, etc. in the new diagnostics repo.

---

@raffaeler commented on Mon Jun 04 2018

It is a console app that throws right after pressing a key from the console.
Instead of throwing manually I iterate a string characters longer than its length
Lldb catches the exception and I can use sos commands only because I published a self contained deployment.
Commands like pe works, bit clrstack made lldb crash.
Ubuntu is version 18.04 LTS.

---

@raffaeler commented on Mon Jun 04 2018

BTW I attach the process instead of running it through corerun.

---

@raffaeler commented on Mon Jun 04 2018

@mikem8361
I love the fact there is a repo entirely for diagnostics... good luck ;)
Do you want me to open an issue in the new diagnostic repo? It looks more appropriate...

---

@mikem8361 commented on Tue Jun 05 2018

@raffaeler opening a new issue in the diagnostics repo sound good especially sense your problem seems to have nothing to do with this original issue.

---

@raffaeler commented on Tue Jun 05 2018

Done: #22

---

@eshenem commented on Mon Jun 25 2018

I am getting the same error. I have lldb-3.9 installed on debian VM (core dump was generated in docker container based on alpine). I copied libcoreclr.so to the directory where core dump file exists and that did not help. Also, while I wasn't sure the build was based on net core 2.1.0 or 2.1.1, I installed both and tried both to no avail. Any ideas?

---

@raffaeler commented on Mon Jun 25 2018

@eshenem I would suggest you to post a comment with the details in the new thread I opened

---

@mikem8361 commented on Mon Jun 25 2018

If you are loading a core dump on debian generated on alpine you need the alpine runtime bits (libcoreclr.so, etc.) because the runtime installed on debian will not match the alpine runtime bits. Alpine is one of the distros that does run the "portable" .NET Core runtime (RID: linux-x64). See
https://www.microsoft.com/net/download/linux.

And it gets more complicated when it comes to using SOS if you are loading this alpine core dump on debian because the libsos.so and some of the runtime modules it depends on (libmscordaccore.so) won't work on debian either.

I would suggest loading this core dump in an alpine docker container, but I've found and I'm working through that the alpine lldb doesn't work that well or not at all. I'm currently working on docs/directions to build a lldb for alpine that works, but it will be a few weeks or so before it is all ready (see diagnostics repo).

---

@eshenem commented on Tue Jun 26 2018

Thanks @mikem8361 - will keep an eye on diagnostics repo for docs / direction.

createdump crash while writing minidump

setup info:

docker container `microsoft/dotnet:2.1-sdk`
host VM: Centos 7

I tried to create mindump for a live dotnet core app.
But createdump itself crash...

createdump output:

Writing minidump with heap to file /workspace/SceneServer/SceneServer/core.123
terminate called after throwing an instance of 'PAL_SEHException'
Aborted (core dumped)

createdump stack trace:

(lldb) bt
* thread #1: tid = 249, 0x00007f4ef8667fff libc.so.6`gsignal + 207, name = 'createdump', stop reason = signal SIGABRT
  * frame #0: 0x00007f4ef8667fff libc.so.6`gsignal + 207
    frame #1: 0x00007f4ef866942a libc.so.6`abort + 362
    frame #2: 0x00007f4ef8f800ad libstdc++.so.6`__gnu_cxx::__verbose_terminate_handler() + 349
    frame #3: 0x00007f4ef8f7e066 libstdc++.so.6`___lldb_unnamed_symbol56$$libstdc++.so.6 + 6
    frame #4: 0x00007f4ef8f7e0b1 libstdc++.so.6`std::terminate() + 17
    frame #5: 0x00007f4ef8f7e2c9 libstdc++.so.6`__cxa_throw + 105
    frame #6: 0x00007f4ef943c9af libmscordaccore.so`ThrowExceptionHelper + 79
    frame #7: 0x00007f4ef940f870 libmscordaccore.so
    frame #8: 0x00007f4ef935f391 libmscordaccore.so`ClrDataFrame::ValueFromDebugInfo(MetaSig*, bool, unsigned int, unsigned int, IXCLRDataValue**) + 337
    frame #9: 0x00007f4ef9360438 libmscordaccore.so`ClrDataFrame::GetExactGenericArgsToken(IXCLRDataValue**) + 200
    frame #10: 0x00007f4ef92d662f libmscordaccore.so`ClrDataAccess::EnumMemWalkStackHelper(CLRDataEnumMemoryFlags, IXCLRDataStackWalk*, Thread*) + 2655
    frame #11: 0x00007f4ef92d31b2 libmscordaccore.so`ClrDataAccess::EnumMemDumpAllThreadsStack(CLRDataEnumMemoryFlags) + 3026
    frame #12: 0x00007f4ef92d0fe3 libmscordaccore.so`ClrDataAccess::EnumMemoryRegionsWorkerHeap(CLRDataEnumMemoryFlags) + 643
    frame #13: 0x00007f4ef92d94d9 libmscordaccore.so`ClrDataAccess::EnumMemoryRegionsWrapper(CLRDataEnumMemoryFlags) + 137
    frame #14: 0x00007f4ef92d975f libmscordaccore.so`ClrDataAccess::EnumMemoryRegions(ICLRDataEnumMemoryRegionsCallback*, unsigned int, CLRDataEnumMemoryFlags) + 239
    frame #15: 0x000000000040776b createdump`CrashInfo::EnumerateMemoryRegionsWithDAC(_MINIDUMP_TYPE) + 219
    frame #16: 0x00000000004063f7 createdump`CrashInfo::GatherCrashInfo(_MINIDUMP_TYPE) + 663
    frame #17: 0x0000000000405693 createdump`CreateDumpCommon(char const*, _MINIDUMP_TYPE, CrashInfo*) + 179
    frame #18: 0x00000000004054ee createdump`main + 606
    frame #19: 0x00007f4ef86552e1 libc.so.6`__libc_start_main + 241
    frame #20: 0x00000000004051c9 createdump

Run the SOS tests against the daily runtime builds

We run SOS against 1.0.9, 2.0.9 and 2.1.0. We need to run against daily builds. This is more difficult for the existing diagnostics repo infrastructure because they are available as nuget packages instead of tar files.

Part of this is creating an internal daily test job.

Need to decide to test against master, 2.2 or other branches that are in developement.

Extend documentation for reading EventCounter data

I've gone through the documentation to gain knowledge about how these should be used but in my opinion it lacks documentation about how to read the data in-process using for example an EventListener.

This issue does mention

EventCounters are based on top of EventSource, so the consumption should be the same.

but it does not mention the need to use the EventCounterIntervalSec argument and the structure of the payload.

I am sure there are other developers wanting to read this data by other means than PerfView. I suppose something like this could improve the documentation. I am more than willing to provide a PR for it

Reading data in-process using EventListener

To enable event counter data reading using an EventListener pass an argument named "EventCounterIntervalSec" with the desired interval in seconds as the value. Eventcounter data will be aggregated during this interval.

var options = new Dictionary<string, string>();
options.Add("EventCounterIntervalSec", "1");
eventListener.EnableEvents(ServiceEventSource.Log, EventLevel.Informational, EventKeywords.All, options);

Given this code an event named "EventCounters" will be emitted each second with a payload of type System.Diagnostics.Tracing.EventPayload.

This can be cast to an IDictionary<string, object> with the following keys: "Name", "Mean", "StandardDeviation", "Count", "Min", "Max", "IntervalSec" where "Name" contains the name of the EventCounter as defined in the EventSource

Note that, although EventCounter is supported in versions of .Net Core earlier than 2.0.3 but it won't actually work unless version > 2.0.3 is targetted.

Documentation for EventPipe

We should have a walk through and some documentation on how to use EventPipe.

Allow BPs before host has initialized during mixed mode debugging

@mikem8361 commented on Fri Mar 23 2018

Yes, interop debugging is .NET Core has the restriction that you cannot hit breakpoints in native code before the .NET runtime starts.

@mikem8361 is there an issue that tracks removing that restriction?

Fixing this would require a major architectural change to how interop debugging works, so I don't think it will be resolved any time soon. I would probably recomend switching back and forth between your C++ project and your .NET Core project depending on which type of code you want to debug. This is what I do all the time while debugging the .NET Core debugger (which has lots of both C# and C++ code).

See #16776

Add DumpHeapStat code to SOS.

@mikem8361 commented on Tue Jan 24 2017

The SOS!DumpHeap implementation has two main perf hot spots: there is a check for break in AddStat function for each step in searching a binary tree, there is a linear search in allocation context array every time moving from one object to next one.

I'm adding more stat information. The latest addition is adding where certain type in certain generation appears most in a heap segment. HeapDumpW has a /crossRef option to dump every object into .csv files by heap segment and generation. So combining the two, I just go to a .csv file to look for certain type I have concern with.

Feng Yuan

---

@danmosemsft commented on Tue Jan 24 2017

UPloaded file and removed internal link
DumpHeapStat.cs.txt

How to use SOS with lldb under OSX

@livarcocc commented on Thu Jun 15 2017

From @NinoFloris on June 12, 2017 22:0

I've tried to follow the loosely translated equivalent of the linux steps but trying do plugin load libsos.dylib gives an lldb error of error: plug-in is missing the required initialization: lldb::PluginInitialize(lldb::SBDebugger)

How to proceed?

Copied from original issue: dotnet/cli#6842

---

@livarcocc commented on Thu Jun 15 2017

From @NinoFloris on June 12, 2017 22:3

@gkhanna79 @piotrpMSFT Would you perhaps have experience with this for OSX?

---

@livarcocc commented on Thu Jun 15 2017

From @gkhanna79 on June 12, 2017 22:53

@mikem8361 @lt72 should be able to help.

---

@livarcocc commented on Thu Jun 15 2017

From @mikem8361 on June 12, 2017 22:58

"libsos.dylib" isn't the lldb plugin to load; it is loaded by the lldb plugin (libsosplugin.dylib). For OS X because of signing issues, etc. you have to first build a version of lldb locally and then build the libsosplugin in the coreclr repo against that.

Some directions:

https://github.com/mikem8361/coreclr/blob/master/Documentation/building/buildinglldb.md

---

@livarcocc commented on Thu Jun 15 2017

From @NinoFloris on June 12, 2017 23:42

Noess, I was hoping to avoid all these native build shenanigans. Is that really the only way? Could this not be made a bit easier? I reckon there are quite a few dotnet core people that work on OSX.

Would a brew formula be an option to abstract the hard and error prone work to build and setup SOS?

(I could devote some of my time to that, if that sounds like something that could work)

---

@livarcocc commented on Thu Jun 15 2017

From @mikem8361 on June 13, 2017 23:58

There are two parts of SOS on Linux and OSX: the lldb plugin (libsosplugin.dylib) and the actual SOS code (libsos.dylib). The lldb plugin needs to be built against the version lldb you are going to use. The libsos.dylib version needs to match the coreclr you are running. The problem is that the includes/libs (LLDB.h, etc. and liblldb.dylib) needed to build the plugin for the default lldb (lldb-370.0.42 on Sierra) on Mac don’t seem to exist anywhere.

The only solution we have come up with for now is to clone and build lldb which results in the includes/libs needed to build the plugin. Those includes/libs don’t match the default lldb because it isn’t clear what build of lldb apple ships. The other problem is the lldb must be built locally with a signing certificate generated by you. The lldb you build or the one I build won’t work on any other machine.

There might be a way to create a cert that could be installed on a machine and package (via brew) lldb and the plugin built against that cert. I don’t know enough about the signing process to know if that would work or the time to research it.

The other solution is to find the matching LLDB.h, etc and lib files for the lldb installed by default (i.e. lldb-370.0.42 on Sierra). I could never figure this one out either.

---

@livarcocc commented on Thu Jun 15 2017

From @NinoFloris on June 14, 2017 1:19

I think they are actually building from this repo https://github.com/apple/swift-lldb, and it is quite likely to be the build tagged with https://github.com/apple/swift-lldb/releases/tag/swift-3.1-RELEASE

As my terminal reports the exact string format as changed in that commit when I do lldb --version

About brew and the cert, letting the formula build it all locally is already miles better than having to do all that manually, it's still a brew install for anyone that wants to quickly get going, the build is just going to take some time.

---

@livarcocc commented on Thu Jun 15 2017

What is the CLI issue here? I will confess that I don't quite follow this thread. If this is not a CLI problem, and it does not look to be, I would like to close this issue.

---

@livarcocc commented on Thu Jun 15 2017

From @mikem8361 on June 15, 2017 5:28

The coreclr issue is that the lldb part of the SOS isn't being provided for OS X for all the reason above. This makes it difficult to use lldb/SOS against a CLI/managed app on OS X.

---

@livarcocc commented on Thu Jun 15 2017

Should this be moved to the coreclr repo then?

---

@livarcocc commented on Thu Jun 15 2017

From @mikem8361 on June 15, 2017 8:18

Yes.

---

@jijiechen commented on Sun Mar 11 2018

Just wanted to note, after having built lldb, you need to build this repo by yourself to get a libsosplugin.dylib on your macOS. Since it's not provided along with published .net core runtime.

Allow the "bpmd" to set source/line or function breakpoints before coreclr is loaded

Allow pending source/line and function (by name) breakpoints before coreclr is loaded.

Which LLDB I should use for analysis crash dump (app run on .NET CORE runtime 2.1.3-servicing-26807-02) from Linux ARM32

I've got several my app crash under my linux iot boards (RaspberryPi3B+ and ASUS Tinker Board S), they are all Linux (Debian based) ARM32 system, I've tried varies of LLDB(from 3.5 to 4.0) for loading libsosplugin.so to analysis the dump but all failed, either could not load the core dump, or after I loaded the plugin, after I typed in sos command, it just crashes itself with Segmentation fault, any suggestion for this?

Remove SOS COM ClrCreateInstance code

This code uses mscoree/installed desktop runtime to find interfaces (currently only the diasymreader interfaces). .NET Core and SOS should not depend on any global state.

Debugging linux dump

@ashishnegi commented on Fri Oct 13 2017

Hi
I am trying to debug a crash dump on Linux. But I am not able to load any symbols.

Here are the steps I tried :

1. I downloaded coreclr symbols from : 2.0.0 download which are symbols.

2. I unzipped *.symbols.nupkg into a folder with all *.dbgs in one folder :

All *.so files are in folder allso.

asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $ ls -l
total 1181308
drwxrwxr-x 2 asnegi asnegi       4096 Oct 13 03:03 allso
-rwxrwxrwx 1 asnegi asnegi      29376 Jul 19 19:25 corerun
-rwxrwxrwx 1 asnegi asnegi     118776 Jul 19 19:25 corerun.dbg
-rwxrwxrwx 1 asnegi asnegi      86296 Jul 19 19:25 createdump
-rwxrwxrwx 1 asnegi asnegi     467480 Jul 19 19:25 createdump.dbg
-rwxrwxrwx 1 asnegi asnegi 1015279616 Oct 12 10:21 dotnet.49126.dmp
-rwxrwxrwx 1 asnegi asnegi    1646224 Jul 19 19:25 ilasm
-rwxrwxrwx 1 asnegi asnegi    4556520 Jul 19 19:25 ilasm.dbg
-rwxrwxrwx 1 asnegi asnegi    1606912 Jul 19 19:25 ildasm
-rwxrwxrwx 1 asnegi asnegi    4124184 Jul 19 19:25 ildasm.dbg
-rwxrwxrwx 1 asnegi asnegi   11713248 Jul 19 19:25 libclrjit.so.dbg
-rwxrwxrwx 1 asnegi asnegi   79269552 Jul 19 19:25 libcoreclr.so.dbg
-rwxrwxrwx 1 asnegi asnegi     860440 Jul 19 19:25 libcoreclrtraceptprovider.so.dbg
-rwxrwxrwx 1 asnegi asnegi    2965160 Jul 19 19:25 libdbgshim.so.dbg
-rwxrwxrwx 1 asnegi asnegi   52992576 Jul 19 19:25 libmscordaccore.so.dbg
-rwxrwxrwx 1 asnegi asnegi   11612952 Jul 19 19:25 libmscordbi.so.dbg
-rwxrwxrwx 1 asnegi asnegi     391152 Jul 19 19:25 libsosplugin.so.dbg
-rwxrwxrwx 1 asnegi asnegi    2783600 Jul 19 19:25 libsos.so.dbg
-rwxrwxrwx 1 asnegi asnegi     979464 Jul 19 19:25 libSystem.Globalization.Native.a
-rwxrwxrwx 1 asnegi asnegi      65869 Jul 19 19:25 sosdocsunix.txt
-rwxrwxrwx 1 asnegi asnegi     436024 Jul 19 19:25 System.Globalization.Native.so.dbg
-rwxrwxrwx 1 asnegi asnegi   10742784 Jul 19 19:25 System.Private.CoreLib.dll
-rwxrwxrwx 1 asnegi asnegi     967300 Jul 19 19:25 System.Private.CoreLib.pdb

3. Dump is also present in same folder : dotnet.49126.dmp

4. I then start lldb with :

asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $ lldb-3.6 -c dotnet.49126.dmp /usr/share/dotnet/dotnet

(lldb) plugin load allso/libsosplugin.so
(lldb) setsostid c049 1
Mapped sos OS tid 0xc049 to lldb thread index 1
(lldb) set ClrStack
Ambiguous command 'set'. Possible matches:
        settings
        setclrpath
        setsostid
(lldb) sos ClrStack
OS Thread Id: 0xc049 (1)
        Child SP               IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000 <unknown>
(lldb) target modules list
[  0] DAAB9A3E-C6DC-6F4B-E681-71B4F4A013FB-8B431DF0                    /usr/share/dotnet/dotnet
[  1] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032                    /lib/x86_64-linux-gnu/libdl.so.2
      /lib/x86_64-linux-gnu/libdl-2.23.so
[  2] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2                    /lib/x86_64-linux-gnu/libpthread.so.0
[  3] 144E588F-94CA-FAFD-BD0B-D1499C74190F-678DAD88                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6
      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
[  4] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8                    /lib/x86_64-linux-gnu/libm.so.6
      /lib/x86_64-linux-gnu/libm-2.23.so
[  5] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434                    /lib/x86_64-linux-gnu/libgcc_s.so.1
[  6] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518                    /lib/x86_64-linux-gnu/libc.so.6
      /lib/x86_64-linux-gnu/libc-2.23.so
[  7] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA                    /lib64/ld-linux-x86-64.so.2
[  8] FA3EC34F-FF22-C058-FE6C-120AD8466ACE-A05DCEBB                    /usr/share/dotnet/host/fxr/2.0.0/libhostfxr.so
[  9] 82FA3DDE-91A5-5710-F6CD-42274FF2701E-5A2E9BE3                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libhostpolicy.so
[ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so
[ 11] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9                    /lib/x86_64-linux-gnu/librt.so.1
      /lib/x86_64-linux-gnu/librt-2.23.so
[ 12] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1                    /usr/lib/x86_64-linux-gnu/libunwind.so.8
[ 13] 13BF33CF-CE9A-8E60-360A-390912876AC2-75C8AD93                    /lib/x86_64-linux-gnu/libuuid.so.1
[ 14] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB                    /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
[ 15] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260                    /lib/x86_64-linux-gnu/liblzma.so.5
[ 16] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclrtraceptprovider.so
      /home/asnegi/ryu/ryu/coreclr/runtimes/linux-x64/native/./libcoreclrtraceptprovider.so.dbg
[ 17] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C                    /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
[ 18] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55                    /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0
[ 19] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE                    /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4
[ 20] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25                    /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4
[ 21] 92F4379E-B144-FCB9-47CD-A120F46B67A7-A38D39F9                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libclrjit.so
[ 22] F3FA2CBA-8B15-ABC5-E785-BE2B38B6F620-E4E617A4                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/System.Globalization.Native.so
[ 23] 23A28B8C-35CF-747B-C507-EE9900D71790-47415517                    /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
[ 24] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD                    /usr/lib/x86_64-linux-gnu/libicudata.so.55
[ 25] 6A9FD2EF-FD0E-B6D3-EE33-CAC7131B30E8-F6C2C6E1                    /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1
[ 26] 7B4BE8CE-1CC5-DCF8-629A-A008500A93F0-E77577A0                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/System.Native.so
[ 27] 29F5E5B3-04BE-ED20-55CF-467E97187F20-6D9E3312                    /usr/lib/x86_64-linux-gnu/libssh2.so.1
[ 28] 675F454A-D6FD-0B6C-A2E4-1127C7B98079-DA37F7B6                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
[ 29] 2DA08A7E-5BF6-1003-0DD3-3B70DB951399-626B7496                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
[ 30] 85CA824C-BE57-3813-B1E3-59894F743EBB-100AB034                    /usr/lib/x86_64-linux-gnu/libxml2.so.2
[ 31] 523E5F7F-4262-BD15-56B1-57D57F13ED16-722E804E                    /lib/x86_64-linux-gnu/libbz2.so.1.0
[ 32] 340B7B46-3F98-1B8A-0FB3-451751F881DF-1B0C2F74                    /lib/x86_64-linux-gnu/libz.so.1
[ 33] 2EB15945-45CC-A1B5-258D-D030B398DA17-4F4541DA                    /lib/x86_64-linux-gnu/libgcrypt.so.20
[ 34] 3A325585-F22D-AE6B-A918-CDC3D2B62760-F666FE16                    /lib/x86_64-linux-gnu/libgpg-error.so.0
[ 35] CE5C34DF-F6A9-8121-F82E-ED4547861645-5801DB1D                    /usr/lib/x86_64-linux-gnu/libsnappy.so.1
      /usr/lib/x86_64-linux-gnu/libsnappy.so.1.3.0
[ 36] D5C47E54-D8F2-BA13-26E2-8B7C8BAB45FD-5B537660                    /usr/lib/x86_64-linux-gnu/liblz4.so.1
      /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
[ 37] 85F3D1F1-BE82-5AF2-FC27-936CD9F113D4-636B0F61                    /lib/x86_64-linux-gnu/libcgroup.so.1

Observation :
As you can see, libcoreclr.so is at location :
[ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so

It is not picking unstripped binaries from my current folder. I am not able to see any stack.

Expected :
It should load symbols from local folder.

Please tell me the right steps to debug.

Extra details :
dotnet version 2.0.0
lldb version 3.6

Note : I also tried with *.so and *.dbg files in same folder with crash_dump.

---

@jkotas commented on Fri Oct 13 2017

cc @lt72

---

@mikem8361 commented on Fri Oct 13 2017

Try adding -O "settings set target.exec-search-paths $HOME/ryu/ryu/coreclr/runtimes/linux-x64/native" to your lldb command line. This is neccessary if the .dbg files are not in the same place as the where they were in the core dump. If "target modules list -s" still doesn't show the .dbg file for libcoreclr.so then it maybe that *.dbg is the wrong version some how. You can use readelf on the libcoreclr.dbg file to get the build id and see if it matches the one in the target modules command (FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 ).

The reason clrstack isn't working is because you need to try another thread index. The setsosid command is a hacky workaround for a lldb 3.6 bug that requires guessing which thread index maps to which OS thread id.

lldb 3.9 has this problem fixed, but you need an libsosplugin.so that works with 3.9. You can try the one I attached (built on ubuntu 14.04 but should work on 16.xx).

libsosplugin.so.zip

---

@danmosemsft commented on Thu Oct 26 2017

@ashishnegi is this resolved for you now? If so, let's close it.

---

@ashishnegi commented on Sat Oct 28 2017

Hi, I will be able to update on this thread in next 3 days.
Thanks for your support.

---

@ashishnegi commented on Mon Oct 30 2017

I see that build id is different for me :

asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $ readelf -n libcoreclr.so.dbg

Displaying notes found at file offset 0x00000200 with length 0x00000024:
  Owner                 Data size       Description
  GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: f09424a9ee0addca898872e72b24a51357677af5

Displaying notes found at file offset 0x049e1d5c with length 0x0000001c:
  Owner                 Data size       Description
  GNU                  0x00000009       NT_GNU_GOLD_VERSION (gold version)
    Version: gold 1.11
asnegi@asnegi-vm ~/ryu/ryu/coreclr/runtimes/linux-x64/native $

and in target modules list :

[ 10] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.0/libcoreclr.so

target modules list -s does not show any libcoreclr.so.dbg.
I see one libcoreclrtraceptprovider.so.dbg in target moduels list -s. readelf shows same target id as that of target modules list.

so, I think my symbols are not right.

1. Is this right inference ?

Also, I think lldb-3.9 will not be able to help here as symbols are not right.

2. Is it possible to find right symbols corresponding to build id : FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 of coreclr.so ?

---

@ashishnegi commented on Mon Oct 30 2017

One question :
In lldb, executing target modules list gives me FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 as build id. This is same as my local installed dotnet's build id. This dotnet has version : 2.0.0.

Is lldb showing build id of dotnet which was crashed on the remote machine Or build id of dotnet present in my local machine ?

---

@mikem8361 commented on Mon Oct 30 2017

"target modules list -u -s" is giving you the build id of the modules in the core dump not what is on your local machine. Your build id of the libcoreclr.so in the dump doesn't match libcoreclr.so.dbg so lldb doesn't load it. What version of .NET Core was on the remote machine that the core dump was generated? It could be either 1.x or maybe a 2.0.x service release which could be the reason for the mismatch.

---

@mikem8361 commented on Mon Oct 30 2017

It does look like the symbol zip files don't match the product. I'm looking into a solution. Not sure what time frame they will be fixed.

---

@ashishnegi commented on Mon Oct 30 2017

@mikem8361 FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 is build id for libcoreclr.so for 2.0.0.
My local install has 2.0.0 and same build id.

So, to confirm, Does it mean that symbols for linux 2.0.0 are not right in coreclr's download archives of 2.0.0 ?

I will try building coreclr locally on linux to get the symbols.
I think many people are using 2.0.0 in linux, so it will be great if right symbols can be provided.

---

@mikem8361 commented on Mon Oct 30 2017

We are working on fixing them now.

-------- Original message --------
From: Ashish Negi [email protected]
Date: 10/30/17 8:21 PM (GMT-08:00)
To: dotnet/coreclr [email protected]
Cc: Mike McLaughlin [email protected], Mention [email protected]
Subject: Re: [dotnet/coreclr] Debugging linux dump (#14490)

@mikem8361https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmikem8361&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=Xz3mCMRORKdIKrsAhZ1yDaKAg1uKDVVvvRO0e67MtvA%3D&reserved=0 FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119 is build id for libcoreclr.so for 2.0.0.
My local install has 2.0.0 and same build id.

So, to confirm, Does it mean that symbols for linux 2.0.0 are not right in coreclr's download archives of 2.0.0https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcore%2Fblob%2Fmaster%2Frelease-notes%2Fdownload-archives%2F2.0.0-download.md&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=hW%2B4p7uMStSDr7NxPAUsM8W9whKD021XlieiXU2Wpls%3D&reserved=0 ?

I will try building coreclr locally on linux to get the symbols.
I think many people are using 2.0.0 in linux, so it will be great if right symbols can be provided.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcoreclr%2Fissues%2F14490%23issuecomment-340648585&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=PCOH6Z%2FFWAn3jbbEy%2Fs3pPzerB2PVFOGC4pBAYJUTNc%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJ-Fa_MRtvKb4Cs-KnDmIpW1IiEypX1pks5sxpJJgaJpZM4P4Rvp&data=02%7C01%7Cmikem%40microsoft.com%7C9e62d61da74d4e2e493608d5200e8472%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636450169077420893&sdata=RQ3N%2B1u1C3TBuPPwd4Xwtu8gUnOeQyLkl4SwEc3%2B6kM%3D&reserved=0.

---

@mikem8361 commented on Tue Oct 31 2017

For now if you get the runtime.linux-x64.Microsoft.NETCore.App.2.0.0.symbols.nupkg package from the core-setup symbol zip, it will match the 2.0.0 binaries in your core dump.

---

@mikem8361 commented on Wed Nov 01 2017

The download site symbol link has been fixed for 2.0.0 and 2.0.2.

---

@ashishnegi commented on Thu Nov 02 2017

Hi @mikem8361 Thanks for sosplugin.
I made good progress.
I installed lldb-3.9 on Ubuntu 16.04 and used your libsosplugin.so.
sos Threads shows real os thread id.
I switched to it. I can use sos PrintException and it works.

I am still not able to get any stack on sos ClrStack. Can you help understand why that's happening ?

(lldb) sos PrintException
Exception object: 00007f5ec808ce00
Exception type:   System.IO.FileNotFoundException
Message:          Could not load the specified file.
InnerException:   <none>
StackTrace (generated):
    SP               IP               Function
    00007F5E9D7F27D0 00007F5EEE0749AD System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(System.Reflection.AssemblyName)+0x3c357d
    00007F5E9D7F2800 00007F5EEDCB12AC System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr, System.Reflection.AssemblyName)+0x5c

StackTraceString: <none>
HResult: 80070002
(lldb) sos ClrStack
OS Thread Id: 0xc049 (57)
        Child SP               IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000 <unknown>

Here is my full debug session :

asnegi@asnegi-vm ~/ryu/dump $ lldb-3.9 -O "settings set target.exec-search-paths /home/asnegi/ryu/dump/" -c dotnet.49126.dmp /usr/share/dotnet/dotnet
(lldb) settings set target.exec-search-paths /home/asnegi/ryu/dump/
(lldb) target create "/usr/share/dotnet/dotnet" --core "dotnet.49126.dmp"
Core file '/home/asnegi/ryu/dump/dotnet.49126.dmp' (x86_64) was loaded.
(lldb) plugin load libsosplugin.so
libclrjit.so was compiled with optimization - stepping may behave oddly; variables may not be available.
(lldb) (lldb)
(lldb) sos Threads
ThreadCount:      20
UnstartedThread:  0
BackgroundThread: 19
PendingThread:    0
DeadThread:       0
Hosted Runtime:   no
                                                                                                        Lock
       ID OSID ThreadOBJ           State GC Mode     GC Alloc Context                  Domain           Count Apt Exception
  11    1 bfef 000000000178BEC0    21220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn (Finalizer)
  13    2 bfe6 00000000017A26B0  2020020 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn
  38    3 c01a 0000000001822D10  1021220 Preemptive  00007F5EC8652450:00007F5EC8652F60 0000000001788130 0     Ukn (Threadpool Worker)
  46    4 c01c 00007F5EC0004BC0  1021220 Preemptive  00007F5EC8625A98:00007F5EC8626F60 0000000001788130 0     Ukn (Threadpool Worker)
  57    5 c049 00007F5E5C00D360    20220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn System.IO.FileNotFoundException 00007f5ec808ce00
   9    6 c045 00007F5E6401CF60    20220 Cooperative 00007F5EC86A5718:00007F5EC86A6F60 0000000001788130 0     Ukn
  49    7 c0c2 00007F5EBC0066C0  1021220 Cooperative 00007F5EC865FC20:00007F5EC8660F60 0000000001788130 0     Ukn (Threadpool Worker)
  31    8 c10e 00007F5E4C0044F0  1020220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn (Threadpool Worker)
  55    9 c14e 00007F5E1C0009F0    20220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn
  52   10 c153 00007F5E20000CD0    20220 Preemptive  00007F5EC8646EE0:00007F5EC8646F60 0000000001788130 0     Ukn
  24   11 c744 00007F5EC055BEB0  1021220 Preemptive  00007F5EC865C438:00007F5EC865CF60 0000000001788130 0     Ukn (Threadpool Worker)
  51   12 c746 00007F5EBC00C780  1021220 Preemptive  00007F5EC860ACE0:00007F5EC860AF60 0000000001788130 0     Ukn (Threadpool Worker)
  53   13 c04c 00007F5E183178B0    20220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn
  47   14 c04a 00007F5E5801B8D0    20220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn
  37   15 c775 00007F5E480086D0  1021220 Cooperative 00007F5EC867CE60:00007F5EC867CF60 0000000001788130 0     Ukn (Threadpool Worker)
   4   16 c776 00007F5DF41909B0  1021220 Cooperative 00007F5EC867D1A0:00007F5EC867EF60 0000000001788130 0     Ukn (Threadpool Worker)
  20   17 c777 00007F5EBC011B30  1021220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn (Threadpool Worker)
  21   18 c778 00007F5EC0560820  1021220 Preemptive  00007F5EC85FAEE0:00007F5EC85FAF60 0000000001788130 0     Ukn (Threadpool Worker)
  27   19 c7f3 00007F5E6405FC50    21220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn
  48   20 c953 00007F5DEC0009F0    20220 Preemptive  0000000000000000:0000000000000000 0000000001788130 0     Ukn
(lldb) thread select 57
(lldb) * thread #57: tid = 49225, 0x00007f5f68ad8c1d libpthread.so.0`__GI___nanosleep + 45, stop reason = signal SIGSEGV
    frame #0: 0x00007f5f68ad8c1d libpthread.so.0`__GI___nanosleep + 45
libpthread.so.0`__GI___nanosleep:
->  0x7f5f68ad8c1d <+45>: addb   %al, (%rax)
    0x7f5f68ad8c1f <+47>: addb   %al, (%rax)
    0x7f5f68ad8c21 <+49>: addb   %al, (%rax)
    0x7f5f68ad8c23 <+51>: addb   %al, (%rax)
(lldb) sos PrintException
Exception object: 00007f5ec808ce00
Exception type:   System.IO.FileNotFoundException
Message:          Could not load the specified file.
InnerException:   <none>
StackTrace (generated):
    SP               IP               Function
    00007F5E9D7F27D0 00007F5EEE0749AD System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(System.Reflection.AssemblyName)+0x3c357d
    00007F5E9D7F2800 00007F5EEDCB12AC System.Private.CoreLib.dll!System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr, System.Reflection.AssemblyName)+0x5c

StackTraceString: <none>
HResult: 80070002
(lldb) sos ClrStack
OS Thread Id: 0xc049 (57)
        Child SP               IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000 <unknown>
(lldb) thread select 1
(lldb) * thread #1: tid = 49481, 0x00007f5f4af52fd9 libclrjit.so`::CONTEXTFromNativeContext(native=0x00007f5e54ff7c80, lpContext=0x00007f5e301bc010, contextFlags=<unavailable>) + 249 at context.cpp:545, name = 'dotnet', stop reason = signal SIGSEGV
    frame #0: 0x00007f5f4af52fd9 libclrjit.so`::CONTEXTFromNativeContext(native=0x00007f5e54ff7c80, lpContext=0x00007f5e301bc010, contextFlags=<unavailable>) + 249 at context.cpp:545 [opt]
(lldb) sos PrintException
The current thread is unmanaged
(lldb) sos ClrStack
OS Thread Id: 0xc149 (1)
Unable to walk the managed stack. The current thread is likely not a
managed thread. You can run !threads to get a list of managed threads in
the process
Failed to start stack walk: 80070057
(lldb)

---

@lt72 commented on Tue Nov 07 2017

@mikem8361 to investigate

---

@mikem8361 commented on Tue Nov 07 2017

@ashishnegi: Is there anyway you could attach the core dump? I would need it to debug why clrstack isn't working.

---

@kirpichiki commented on Thu Dec 21 2017

Hi everyone and sorry for invasion. I have another situation with debugging core dump. I can't determine how to load symbols for my own code.

`clrstack` works fine for clr libraries, but when meets the library from my project it marks it as !Unknown

clrstack of thread 4b (core dump)

OS Thread Id: 0x4b (1)
        Child SP               IP Call Site
00007F78D37FD210 00007f7bb07c7360 [GCFrame: 00007f78d37fd210]
00007F78D37FD300 00007f7bb07c7360 [HelperMethodFrame_1OBJ: 00007f78d37fd300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F78D37FD430 00007F7B359EB503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD4C0 00007F7B359AB6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
00007F78D37FD520 00007F7B359AB559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)
00007F78D37FD5E0 00007F7B359AB273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD610 00007F7B39A4725C /app/Hangfire.Mongo.dll!Unknown
00007F78D37FD6F0 00007F7B39A19AEA /app/Hangfire.Core.dll!Unknown
00007F78D37FD750 00007F7B39A19A4D /app/Hangfire.Core.dll!Unknown
00007F78D37FD770 00007F7B39A1964D /app/Hangfire.Core.dll!Unknown
00007F78D37FD7F0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FD840 00007F7B359AAB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
00007F78D37FD8E0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FDC58 00007f7baf116067 [GCFrame: 00007f78d37fdc58]
00007F78D37FDD30 00007f7baf116067 [DebuggerU2MCatchHandlerFrame: 00007f78d37fdd30]

When i attaching to process directly all symbols is loaded

clrstack of thread 4b (process attach)

OS Thread Id: 0x4b (1)
        Child SP               IP Call Site
00007F78D37FD210 00007f7bb07c7360 [GCFrame: 00007f78d37fd210]
00007F78D37FD300 00007f7bb07c7360 [HelperMethodFrame_1OBJ: 00007f78d37fd300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F78D37FD430 00007F7B359EB503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD4C0 00007F7B359AB6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
00007F78D37FD520 00007F7B359AB559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)
00007F78D37FD5E0 00007F7B359AB273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)
00007F78D37FD610 00007F7B39A4725C Hangfire.Mongo.Signal.Mongo.MongoSignal.Listen(System.Threading.CancellationToken)
00007F78D37FD6F0 00007F7B39A19AEA Hangfire.Server.AutomaticRetryProcess.Execute(Hangfire.Server.BackgroundProcessContext)
00007F78D37FD750 00007F7B39A19A4D Hangfire.Server.InfiniteLoopProcess.Execute(Hangfire.Server.BackgroundProcessContext)
00007F78D37FD770 00007F7B39A1964D Hangfire.Server.ServerProcessExtensions.RunProcess(Hangfire.Server.IServerProcess, Hangfire.Server.BackgroundProcessContext)
00007F78D37FD7F0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FD840 00007F7B359AAB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
00007F78D37FD8E0 00007F7B359712F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F78D37FDC58 00007f7baf116067 [GCFrame: 00007f78d37fdc58]
00007F78D37FDD30 00007f7baf116067 [DebuggerU2MCatchHandlerFrame: 00007f78d37fdd30]

As far as I can gather the difference consists in loading of characters from vdso. After the core dump is loaded, `image list` contains of 65 positions

image list (core dump)

[  0] 147E9425-FAF3-931B-7847-70F59A293B75-D3AFE15A                    /usr/share/dotnet/dotnet
[  1] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032                    /lib/x86_64-linux-gnu/libdl.so.2
[  2] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2                    /lib/x86_64-linux-gnu/libpthread.so.0
[  3] B2EAB689-A85C-BE4F-D4FF-4E26C85B78ED-1C1BB649                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[  4] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8                    /lib/x86_64-linux-gnu/libm.so.6
[  5] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434                    /lib/x86_64-linux-gnu/libgcc_s.so.1
[  6] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518                    /lib/x86_64-linux-gnu/libc.so.6
[  7] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA                    /lib64/ld-linux-x86-64.so.2
[  8] D86DD5BB-C39B-CBE9-A9B1-A38AB649D9C0-B71E0B10                    /usr/share/dotnet/host/fxr/2.0.3/libhostfxr.so
[  9] 4913D69C-1CC0-875A-2F6D-9EE01CD6CB37-E868EB6B                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libhostpolicy.so
[ 10] 6E4920E8-4B41-E313-6DD1-C81372830F35-41B97699                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so.dbg
[ 11] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9                    /lib/x86_64-linux-gnu/librt.so.1
[ 12] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1                    /usr/lib/x86_64-linux-gnu/libunwind.so.8
[ 13] 3CEF054D-42C9-88DE-9868-87E7DB371868-DCA3788A                    /lib/x86_64-linux-gnu/libuuid.so.1
[ 14] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB                    /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
[ 15] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260                    /lib/x86_64-linux-gnu/liblzma.so.5
[ 16] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so.dbg
[ 17] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C                    /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
[ 18] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55                    /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0
[ 19] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE                    /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4
[ 20] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25                    /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4
[ 21] 856752E3-54D0-A7BD-B459-EB387FEB2424-C0EE031B                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libclrjit.so
[ 22] 4204CB9E-E445-DEE6-A5FA-2663837DFC4D-EB02E9C9                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so.dbg
[ 23] 01B48531-F68A-F8C3-3E79-B1A46CAF69C7-3D369192                    /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
[ 24] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD                    /usr/lib/x86_64-linux-gnu/libicudata.so.55
[ 25] 628F069D-8645-434A-D88E-502966359B23-213456EB                    /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1
[ 26] 6D534237-9F9B-96DE-7E69-800B026F1CBD-EB4FC321                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Native.so
[ 27] 087255FF-BEB3-B875-9FFA-E5B50E7D5E12-B5C0D628                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Security.Cryptography.Native.OpenSsl.so
[ 28] 42D1D917-636C-64CF-9904-65F087B18169-7E4CDD7F                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
[ 29] C09D2E07-5184-E1CA-71F7-E08957982A94-7A377939                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
[ 30] 5D383C4B-E609-2A9A-0D05-DD67CBB28E49-45CF3EA6                    /usr/share/dotnet/store/x64/netcoreapp2.0/libuv/1.10.0/runtimes/linux-x64/native/libuv.so
[ 31] A62AB5DA-17C5-DBC4-8859-304CE066B7A2-A26A0C7D                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Net.Http.Native.so
[ 32] AFBF17C0-C8A6-D0C7-7E4D-D56CF850E76C-A89EDAA6                    /usr/lib/x86_64-linux-gnu/libcurl.so.4
[ 33] E09D3783-AD1D-0BBC-D320-4FA01E4EF6D7-56E18F57                    /usr/lib/x86_64-linux-gnu/libidn.so.11
[ 34] 8D1CC120-4D6B-6D33-BD1D-2C5A2A0516A2-234322CF                    /usr/lib/x86_64-linux-gnu/librtmp.so.1
[ 35] DB5180B5-6809-7E2A-4690-A5B40D36BD13-4C893FEE                    /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
[ 36] 213C3A67-5647-0FF9-61EE-7FD63BAC177B-C6B72A49                    /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
[ 37] 698C15E7-BD2C-D966-1858-3CADC6EBCE07-B293CEB2                    /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
[ 38] 8D9BD4CE-26E4-5EF1-6075-C67D5F5EEAFD-8B562832                    /lib/x86_64-linux-gnu/libz.so.1
[ 39] 3CE893F6-D138-2C2C-7648-DCCB06E71B1C-7E0861CC                    /usr/lib/x86_64-linux-gnu/libgnutls.so.30
[ 40] B11678F5-6019-9547-DCF7-26384EA39153-EE0DFABF                    /usr/lib/x86_64-linux-gnu/libhogweed.so.4
[ 41] D6B36C5A-463E-E0FA-84FD-D6D5FD3F7726-EDB90D54                    /usr/lib/x86_64-linux-gnu/libnettle.so.6
[ 42] 7B3533D5-998D-20EE-1A1B-E3F87789B690-41E7F620                    /usr/lib/x86_64-linux-gnu/libgmp.so.10
[ 43] 16E3DBC6-D048-1459-39BB-43BBFD7954D2-7421B00F                    /usr/lib/x86_64-linux-gnu/libkrb5.so.3
[ 44] AEB4C08F-C47F-86C4-75E9-D3996DFE5E9B-403ACEBF                    /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
[ 45] 1E16CB57-F699-E215-A2A8-D4EFEF90883B-C749B12D                    /lib/x86_64-linux-gnu/libcom_err.so.2
[ 46] DF3219B8-9E86-920E-901B-AC4A80AA60F2-B6134588                    /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
[ 47] E7196EFC-C044-747C-45DA-6705CBE6E3EB-2CE5DAE2                    /lib/x86_64-linux-gnu/libresolv.so.2
[ 48] 96BCC7EB-28D8-1B14-69EE-D6F24FC083CB-D58577BC                    /usr/lib/x86_64-linux-gnu/libsasl2.so.2
[ 49] 1FE877BE-52A4-24D0-636A-FD4D35BB330E-41D6E0F3                    /usr/lib/x86_64-linux-gnu/libgssapi.so.3
[ 50] A0E2D03F-F5CF-6593-7F44-25D4EFD4D655-243809EB                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
[ 51] A6EE7754-728D-688B-0409-5ACB34F1A071-F99ED6C3                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6
[ 52] 3364D4BF-2113-C4E8-D17E-F533867ECC99-A53413D6                    /lib/x86_64-linux-gnu/libkeyutils.so.1
[ 53] 73A8EADB-C858-6066-2B24-850E71D4AFBE-22C33359                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0
[ 54] 59E74230-6A4E-A287-2E06-1ECCE92F35FA-DDA75357                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26
[ 55] E5C159E4-1540-6AE7-9D21-056D752BA949-C408B5B1                    /usr/lib/x86_64-linux-gnu/libasn1.so.8
[ 56] 7D15576E-1F09-6614-D360-784E4A01A1F5-FAF908C9                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4
[ 57] 481DB33C-28D8-8E43-DA6B-ED65E1A75994-07D4D818                    /usr/lib/x86_64-linux-gnu/libroken.so.18
[ 58] 9D9C958F-1F48-94AF-EF6A-ECD90D1C430E-A29AC34F                    /usr/lib/x86_64-linux-gnu/libffi.so.6
[ 59] 57E25072-866B-2D30-CF02-EBE7AE623B84-F96FA700                    /usr/lib/x86_64-linux-gnu/libwind.so.0
[ 60] F6F1B4E9-F89B-716C-4A0B-A5819BDFFAF4-A13EFB91                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1
[ 61] C60082E3-BB78-D0D4-2868-D9B359B89BF6-6CE5A1A7                    /usr/lib/x86_64-linux-gnu/libhx509.so.5
[ 62] D9782BA0-23CA-EC26-B15D-8676E3A5D07B-55E121EF                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0
[ 63] 390CD89E-F95E-5B2C-9E32-652DDBA75DA6-88A9CAF9                    /lib/x86_64-linux-gnu/libcrypt.so.1
[ 64] D8E2912D-8CEA-A52A-AD78-1D76526BF205-CF746752                    /lib/x86_64-linux-gnu/libnss_files.so.2
[ 65] D6921D29-D0C8-48A2-4379-504B2C6E6D45-BB26D416                    /lib/x86_64-linux-gnu/libnss_dns.so.2

After attaching to process there are 66 including [vdso] (0x00007fff2978d000)

image list (core dump)

[  0] 147E9425-FAF3-931B-7847-70F59A293B75-D3AFE15A                    /usr/share/dotnet/dotnet
[  1] 41FD7D4C-745E-681C-706B-ED73B4CC46C3-E422C5E6 0x00007fff2978d000 [vdso] (0x00007fff2978d000)
[  2] 0FC788F0-8618-4625-7B5F-1773FBD438E9-5DFC1032                    /lib/x86_64-linux-gnu/libdl.so.2
[  3] 27F189EF-8DB8-C373-4C6A-678E6EF3CB0B-206D58B2                    /lib/x86_64-linux-gnu/libpthread.so.0
[  4] B2EAB689-A85C-BE4F-D4FF-4E26C85B78ED-1C1BB649                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[  5] FF7A33D3-89E7-56CA-381A-8189291A968E-A5E1F4F8                    /lib/x86_64-linux-gnu/libm.so.6
[  6] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434                    /lib/x86_64-linux-gnu/libgcc_s.so.1
[  7] 088A6E00-A181-4622-219F-346B41E775B8-DD46C518                    /lib/x86_64-linux-gnu/libc.so.6
[  8] 9157F205-547F-0EB5-88E2-AB1F2F120B74-253A43EA                    /lib64/ld-linux-x86-64.so.2
[  9] D86DD5BB-C39B-CBE9-A9B1-A38AB649D9C0-B71E0B10                    /usr/share/dotnet/host/fxr/2.0.3/libhostfxr.so
[ 10] 4913D69C-1CC0-875A-2F6D-9EE01CD6CB37-E868EB6B                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libhostpolicy.so
[ 11] 6E4920E8-4B41-E313-6DD1-C81372830F35-41B97699                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclr.so.dbg
[ 12] F951C1E0-765F-CAE4-8F82-CAFE35D1ADD3-6D6C9AF9                    /lib/x86_64-linux-gnu/librt.so.1
[ 13] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1                    /usr/lib/x86_64-linux-gnu/libunwind.so.8
[ 14] 3CEF054D-42C9-88DE-9868-87E7DB371868-DCA3788A                    /lib/x86_64-linux-gnu/libuuid.so.1
[ 15] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB                    /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
[ 16] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260                    /lib/x86_64-linux-gnu/liblzma.so.5
[ 17] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libcoreclrtraceptprovider.so.dbg
[ 18] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C                    /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
[ 19] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55                    /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0
[ 20] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE                    /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4
[ 21] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25                    /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4
[ 22] 856752E3-54D0-A7BD-B459-EB387FEB2424-C0EE031B                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/libclrjit.so
[ 23] 4204CB9E-E445-DEE6-A5FA-2663837DFC4D-EB02E9C9                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so
      /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Globalization.Native.so.dbg
[ 24] 01B48531-F68A-F8C3-3E79-B1A46CAF69C7-3D369192                    /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
[ 25] E8C2C2CC-46D6-EF2B-507A-DC2C663B9F42-7F5CA5DD                    /usr/lib/x86_64-linux-gnu/libicudata.so.55
[ 26] 628F069D-8645-434A-D88E-502966359B23-213456EB                    /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1
[ 27] 6D534237-9F9B-96DE-7E69-800B026F1CBD-EB4FC321                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Native.so
[ 28] 087255FF-BEB3-B875-9FFA-E5B50E7D5E12-B5C0D628                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Security.Cryptography.Native.OpenSsl.so
[ 29] 42D1D917-636C-64CF-9904-65F087B18169-7E4CDD7F                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
[ 30] C09D2E07-5184-E1CA-71F7-E08957982A94-7A377939                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
[ 31] 5D383C4B-E609-2A9A-0D05-DD67CBB28E49-45CF3EA6                    /usr/share/dotnet/store/x64/netcoreapp2.0/libuv/1.10.0/runtimes/linux-x64/native/libuv.so
[ 32] A62AB5DA-17C5-DBC4-8859-304CE066B7A2-A26A0C7D                    /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/System.Net.Http.Native.so
[ 33] AFBF17C0-C8A6-D0C7-7E4D-D56CF850E76C-A89EDAA6                    /usr/lib/x86_64-linux-gnu/libcurl.so.4
[ 34] E09D3783-AD1D-0BBC-D320-4FA01E4EF6D7-56E18F57                    /usr/lib/x86_64-linux-gnu/libidn.so.11
[ 35] 8D1CC120-4D6B-6D33-BD1D-2C5A2A0516A2-234322CF                    /usr/lib/x86_64-linux-gnu/librtmp.so.1
[ 36] DB5180B5-6809-7E2A-4690-A5B40D36BD13-4C893FEE                    /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
[ 37] 213C3A67-5647-0FF9-61EE-7FD63BAC177B-C6B72A49                    /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
[ 38] 698C15E7-BD2C-D966-1858-3CADC6EBCE07-B293CEB2                    /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
[ 39] 8D9BD4CE-26E4-5EF1-6075-C67D5F5EEAFD-8B562832                    /lib/x86_64-linux-gnu/libz.so.1
[ 40] 3CE893F6-D138-2C2C-7648-DCCB06E71B1C-7E0861CC                    /usr/lib/x86_64-linux-gnu/libgnutls.so.30
[ 41] B11678F5-6019-9547-DCF7-26384EA39153-EE0DFABF                    /usr/lib/x86_64-linux-gnu/libhogweed.so.4
[ 42] D6B36C5A-463E-E0FA-84FD-D6D5FD3F7726-EDB90D54                    /usr/lib/x86_64-linux-gnu/libnettle.so.6
[ 43] 7B3533D5-998D-20EE-1A1B-E3F87789B690-41E7F620                    /usr/lib/x86_64-linux-gnu/libgmp.so.10
[ 44] 16E3DBC6-D048-1459-39BB-43BBFD7954D2-7421B00F                    /usr/lib/x86_64-linux-gnu/libkrb5.so.3
[ 45] AEB4C08F-C47F-86C4-75E9-D3996DFE5E9B-403ACEBF                    /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
[ 46] 1E16CB57-F699-E215-A2A8-D4EFEF90883B-C749B12D                    /lib/x86_64-linux-gnu/libcom_err.so.2
[ 47] DF3219B8-9E86-920E-901B-AC4A80AA60F2-B6134588                    /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
[ 48] E7196EFC-C044-747C-45DA-6705CBE6E3EB-2CE5DAE2                    /lib/x86_64-linux-gnu/libresolv.so.2
[ 49] 96BCC7EB-28D8-1B14-69EE-D6F24FC083CB-D58577BC                    /usr/lib/x86_64-linux-gnu/libsasl2.so.2
[ 50] 1FE877BE-52A4-24D0-636A-FD4D35BB330E-41D6E0F3                    /usr/lib/x86_64-linux-gnu/libgssapi.so.3
[ 51] A0E2D03F-F5CF-6593-7F44-25D4EFD4D655-243809EB                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
[ 52] A6EE7754-728D-688B-0409-5ACB34F1A071-F99ED6C3                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6
[ 53] 3364D4BF-2113-C4E8-D17E-F533867ECC99-A53413D6                    /lib/x86_64-linux-gnu/libkeyutils.so.1
[ 54] 73A8EADB-C858-6066-2B24-850E71D4AFBE-22C33359                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0
[ 55] 59E74230-6A4E-A287-2E06-1ECCE92F35FA-DDA75357                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26
[ 56] E5C159E4-1540-6AE7-9D21-056D752BA949-C408B5B1                    /usr/lib/x86_64-linux-gnu/libasn1.so.8
[ 57] 7D15576E-1F09-6614-D360-784E4A01A1F5-FAF908C9                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4
[ 58] 481DB33C-28D8-8E43-DA6B-ED65E1A75994-07D4D818                    /usr/lib/x86_64-linux-gnu/libroken.so.18
[ 59] 9D9C958F-1F48-94AF-EF6A-ECD90D1C430E-A29AC34F                    /usr/lib/x86_64-linux-gnu/libffi.so.6
[ 60] 57E25072-866B-2D30-CF02-EBE7AE623B84-F96FA700                    /usr/lib/x86_64-linux-gnu/libwind.so.0
[ 61] F6F1B4E9-F89B-716C-4A0B-A5819BDFFAF4-A13EFB91                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1
[ 62] C60082E3-BB78-D0D4-2868-D9B359B89BF6-6CE5A1A7                    /usr/lib/x86_64-linux-gnu/libhx509.so.5
[ 63] D9782BA0-23CA-EC26-B15D-8676E3A5D07B-55E121EF                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0
[ 64] 390CD89E-F95E-5B2C-9E32-652DDBA75DA6-88A9CAF9                    /lib/x86_64-linux-gnu/libcrypt.so.1
[ 65] D8E2912D-8CEA-A52A-AD78-1D76526BF205-CF746752                    /lib/x86_64-linux-gnu/libnss_files.so.2
[ 66] D6921D29-D0C8-48A2-4379-504B2C6E6D45-BB26D416                    /lib/x86_64-linux-gnu/libnss_dns.so.2

If my conclution is right, how can I load vdso while dedugging core dump?
@mikem8361 Won't you mind to gide me how to deal with this situation?

Misc info:

• All work takes place in docker container with Ubuntu:xenial base
• dotnet version is 2.0.3
• lldb version is 4.0 with manually builded libsosplugin.so
• Changing settings set target.exec-search-paths to folder with my program dlls seems to have no effect

---

@mikem8361 commented on Thu Dec 21 2017

I don't think vso has anything to do with your problem. I think sos is having a problem finding the pdb's for your app or having problem unwinding the managed frames in the core dump. How did you generate the core dump? sos should load the portable pdbs from the same directory as the your app assemblies/program. What was the exact command used to load it in LLDB?

Is there any way you could zip up the core dump and all your app binaries and attach them to this issue?

---

@kirpichiki commented on Thu Dec 21 2017

Thanks for the answer. I am generating coredump with the createdump utility using default parameters. Something like:
/usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/createdump 6
To start debugger I am using this command:
lldb-4.0 -O "settings set target.exec-search-paths /app/" -o "plugin load /libsosplugin.so" --core /tmp/coredump.6 /usr/share/dotnet/dotnet
Here /app - folder with my asp.net core app (dlls and pdbs), builded with Release configuration,
/libsosplugin.so - manually builded plugin

I'm afraid sources is pretty confidential, so I cant share it with you =(

---

@mikem8361 commented on Thu Dec 21 2017

This may be a bug in createdump. Try building the createdump from the coreclr master branch (I can also send it to you if you want) and/or try the “--full” option.

---

@ashishnegi commented on Thu Dec 21 2017

@mikem8361 I somehow missed earlier notification about dump.

I can confirm that sos ClrStack is working for me for few weeks.
I think I didn't do any thing new to make it work.
Otherwise, even then sos EEStack -EE was working for me.

We can close this thread now.

---

@kirpichiki commented on Fri Dec 22 2017

Yep, I see my symbols only with full dump. Thanks for your guidance @mikem8361. It's frustrating I have to load all 24 Gb of virtual memory just to watch stacktrace, but if thats only option I'll find some workaround.

---

@mikem8361 commented on Fri Dec 22 2017

Did you try the createdump built from the coreclr master branch (with the default options)? This will let me know that the bug/problem is fixed for 2.1. It should work with the 2.0.3 runtime.

createdump.zip

---

@kirpichiki commented on Mon Dec 25 2017

@mikem8361 your createdump fails with
relocation error: /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.3/createdump: symbol PAL_VirtualUnwindOutOfProc, version V1.0 not defined in file libmscordaccore.so with link time reference

I compiled coreclr from the master and tried to run createdump with the default options - same result

OS Thread Id: 0x7f (1)
        Child SP               IP Call Site
00007F85347F7210 00007f8808245360 [GCFrame: 00007f85347f7210]
00007F85347F7300 00007f8808245360 [HelperMethodFrame_1OBJ: 00007f85347f7300] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
00007F85347F7430 00007F878D45B503 System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)
00007F85347F74C0 00007F878D41B6EC System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken)
00007F85347F7520 00007F878D41B559 System.Threading.Tasks.Task.InternalWait(Int32, System.Threading.CancellationToken)
00007F85347F75E0 00007F878D41B273 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)
00007F85347F7610 00007F87914B4BFC /app/Hangfire.Mongo.dll!Unknown
00007F85347F76F0 00007F8791486F2A /app/Hangfire.Core.dll!Unknown
00007F85347F7750 00007F8791486E8D /app/Hangfire.Core.dll!Unknown
00007F85347F7770 00007F8791486A8D /app/Hangfire.Core.dll!Unknown
00007F85347F77F0 00007F878D3E12F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F85347F7840 00007F878D41AB6E System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
00007F85347F78E0 00007F878D3E12F1 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007F85347F7C58 00007f8806b94067 [GCFrame: 00007f85347f7c58]
00007F85347F7D30 00007f8806b94067 [DebuggerU2MCatchHandlerFrame: 00007f85347f7d30]

---

@mikem8361 commented on Tue Jan 02 2018

Sorry about that. The standard createdump has some dependencies on libmscordaccore.so. Try this createdump:

createdump.zip

---

@kirpichiki commented on Wed Jan 10 2018

No changes. Still getting !Unknown.

Add support of standard streams redirection in dbgshim

Currently it's impossible to redirect input/output streams of the process created with CreateProcessForLaunch(). Such feature can be useful for debugger implementation for Unix.
However PAL version of CreateProcessW supports stream redirection.
I propose to extend the API of the dbgshim library with something like this:

EXTERN_C HRESULT
CreateProcessForLaunchEx(
    __in LPWSTR lpCommandLine,
    __in BOOL bSuspendProcess,
    __in LPVOID lpEnvironment,
    __in LPCWSTR lpCurrentDirectory,
    __out PDWORD pProcessId,
    __out HANDLE *pResumeHandle,
    __out HANDLE *pStdInHandle,
    __out HANDLE *pStdOutHandle,
    __out HANDLE *pStdErrHandle);

Also a special portable API for HANDLE reading and writing will be added:

typedef VOID (*PREADHANDLE_CALLBACK)(PVOID pBuffer, DWORD dwBufferSizeInBytes, PVOID parameter);

EXTERN_C HRESULT
RegisterForStandardOutput(
    __in HANDLE hStdOutHandle,
    __in PREADHANDLE_CALLBACK pfnOutCallback,
    __in HANDLE hStdErrHandle,
    __in PREADHANDLE_CALLBACK pfnErrCallback,
    __in PVOID parameter,
    __out PVOID *ppListenerToken);

EXTERN_C HRESULT
WriteStandardHandle(
    __in HANDLE hStdInHandle,
    __in LPCVOID lpBuffer,
    __in DWORD dwNumberOfBytesToWrite,
    __out DWORD *pdwNumberOfBytesWritten
);

EXTERN_C HRESULT
CloseStandardHandle(
    __in HANDLE hStandardHandle);

In such case debugger can handle input/output of the newly created process in a portable way.

Profiling: Make better docs about what metadata APIs are legal to call in which situations

The profiler has pretty poor documentation in this area. What little that exists on David Broman's blog or MSDN is typically either overly loose (we give the profiler writer the entire IMetadatEmit API and somehow expect them to know which modifications will work?) or overly restrictive (we say do all edits during ModuleLoadFinished, but some of them actually work later too)

We likely need to bucket all the possible modifications into 4 categories:
a) Never legal
b) Legal at Module load for images that aren't precompiled
c) Legal at Module load for any image
d) Legal at any time

vscode `remote lanuch debug` .NET CORE 2.1 console app( run on LINUX ARM32) crashed with `Error from pipe program plink.exe`

Hi,

I've setup the VS code to remote lanuch debugging my console app which running on a LINUX ARM32 computer, I'v write a simple throw unhandled exception on purpose app, and tested with break point can be hit in VS CODE, but, with my real application, when it crashed for running for hours, the vscode debug console printed:

Error from pip program 'plink.exe': Process is terminating due to StackOverflow exception

does it mean my app crashed due to stackoverflow exception and the vscode can't catpure? then what is the solution for this kind of exception debug?

ARM64: take a look at todo in sos code

Issue copied from https://github.com/dotnet/coreclr/issues/6580

The following @ARM64TODO in debug/SOS code (using dotnet/coreclr paths):

ToolBox\SOS\Strike\disasmARM64.cpp:    // ARM64TODO: needs to be implemented for jump stubs for ngen case
ToolBox\SOS\Strike\disasmARM64.cpp:    // ARM64TODO: not (yet) implemented. perhaps we don't need it at all.
ToolBox\SOS\Strike\disasmARM64.cpp:            // ARM64TODO: we could possibly handle adr(p)/ldr pair too.

Add and trim PAL tests

Add the coreclr PAL tests for the diagnostics repo PAL subset.

Cannot use SOS from netcore 2.1.300 under lldb-3.9 on Ubuntu18

I created two scripts to attach a managed process (netcore 2.1) from lldb-3.9:
This one is used to load sos from the self contained deployment of the app

#!/bin/bash
pid=$1
sudo lldb-3.9 -o "process attach -p $pid" -o "plugin load ~/crash/scd/libsosplugin.so" -o "setclrpath ~/crash/scd/" -o "process handle -s false SIGUSR1 SIGUSR2"

This is the one for non-scd deployment (sos is loaded from the sdk) which does not work because of a 'Failed to load data access DLL, 0x801331c4f'

#!/bin/bash
pid=$1
sudo lldb-3.9 -o "process attach -p $pid" -o "plugin load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.0/libsosplugin.so" -o "setclrpath /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.0/" -o "process handle -s false SIGUSR1 SIGUSR2"

After running a managed process compiled with 2.1.300, I get the pid with ps -ejH and attach it using one of the above scripts (~/lldb)

The process is attached and the plugin gets loaded:

raf@ubuntu18:~/crash/scd$ ~/lldb.sh 16436
(lldb) process attach -p 16436
Process 16436 stopped
* thread #1: tid = 16436, 0x00007fbf76ceb384 libpthread.so.0`__libc_read + 68, name = 'crash', stop reason = signal SIGSTOP
    frame #0: 0x00007fbf76ceb384 libpthread.so.0`__libc_read + 68
libpthread.so.0`__libc_read:
->  0x7fbf76ceb384 <+68>: cmpq   $-0x1000, %rax            ; imm = 0xF000 
    0x7fbf76ceb38a <+74>: ja     0x7fbf76ceb3c4            ; <+132>
    0x7fbf76ceb38c <+76>: movl   %r8d, %edi
    0x7fbf76ceb38f <+79>: movq   %rax, 0x8(%rsp)
  thread #2: tid = 16437, 0x00007fbf760c0839 libc.so.6`syscall + 25 at syscall.S:38, name = 'crash-ust', stop reason = signal SIGSTOP
    frame #0: 0x00007fbf760c0839 libc.so.6`syscall + 25 at syscall.S:38
  thread #3: tid = 16438, 0x00007fbf760c0839 libc.so.6`syscall + 25 at syscall.S:38, name = 'crash-ust', stop reason = signal SIGSTOP
    frame #0: 0x00007fbf760c0839 libc.so.6`syscall + 25 at syscall.S:38
  thread #4: tid = 16439, 0x00007fbf760b9bf9 libc.so.6`__GI___poll(fds=0x00007fbf72d5ade8, nfds=1, timeout=-1) + 73 at poll.c:29, name = 'crash', stop reason = signal SIGSTOP
    frame #0: 0x00007fbf760b9bf9 libc.so.6`__GI___poll(fds=0x00007fbf72d5ade8, nfds=1, timeout=-1) + 73 at poll.c:29
  thread #5: tid = 16440, 0x00007fbf76cebdae libpthread.so.0`__libc_open64 + 206, name = 'crash', stop reason = signal SIGSTOP
    frame #0: 0x00007fbf76cebdae libpthread.so.0`__libc_open64 + 206
libpthread.so.0`__libc_open64:
->  0x7fbf76cebdae <+206>: cmpq   $-0x1000, %rax            ; imm = 0xF000 
    0x7fbf76cebdb4 <+212>: ja     0x7fbf76cebde6            ; <+262>
    0x7fbf76cebdb6 <+214>: movl   %r8d, %edi
    0x7fbf76cebdb9 <+217>: movl   %eax, 0x8(%rsp)
  thread #6: tid = 16441, 0x00007fbf76ce79f3 libpthread.so.0`__pthread_cond_wait + 579, name = 'crash', stop reason = signal SIGSTOP
    frame #0: 0x00007fbf76ce79f3 libpthread.so.0`__pthread_cond_wait + 579
libpthread.so.0`__pthread_cond_wait:
->  0x7fbf76ce79f3 <+579>: cmpq   $-0x1000, %rax            ; imm = 0xF000 
    0x7fbf76ce79f9 <+585>: movq   0x30(%rsp), %r8
    0x7fbf76ce79fe <+590>: ja     0x7fbf76ce7b30            ; <+896>
    0x7fbf76ce7a04 <+596>: movl   %r9d, %edi
  thread #7: tid = 16442, 0x00007fbf76ce7ed9 libpthread.so.0`__pthread_cond_timedwait + 649, name = 'crash', stop reason = signal SIGSTOP
    frame #0: 0x00007fbf76ce7ed9 libpthread.so.0`__pthread_cond_timedwait + 649
libpthread.so.0`__pthread_cond_timedwait:
->  0x7fbf76ce7ed9 <+649>: cmpq   $-0x1000, %rax            ; imm = 0xF000 
    0x7fbf76ce7edf <+655>: ja     0x7fbf76ce81c0            ; <+1392>
    0x7fbf76ce7ee5 <+661>: movl   %r8d, %edi
    0x7fbf76ce7ee8 <+664>: callq  0x7fbf76ceb060            ; __pthread_disable_asynccancel
  thread #8: tid = 16443, 0x00007fbf76ceb384 libpthread.so.0`__libc_read + 68, name = 'crash', stop reason = signal SIGSTOP
    frame #0: 0x00007fbf76ceb384 libpthread.so.0`__libc_read + 68
libpthread.so.0`__libc_read:
->  0x7fbf76ceb384 <+68>: cmpq   $-0x1000, %rax            ; imm = 0xF000 
    0x7fbf76ceb38a <+74>: ja     0x7fbf76ceb3c4            ; <+132>
    0x7fbf76ceb38c <+76>: movl   %r8d, %edi
    0x7fbf76ceb38f <+79>: movq   %rax, 0x8(%rsp)

Executable module set to "/home/raf/crash/scd/crash".
Architecture set to: x86_64-pc-linux.
(lldb) plugin load ./libsosplugin.so
(lldb) setclrpath ./
(lldb) process handle -s false SIGUSR1 SIGUSR2
NAME         PASS   STOP   NOTIFY
===========  =====  =====  ======
SIGUSR1      true   false  true 
SIGUSR2      true   false  true 

At this point the pe is issued and does not return any result (correct).
But the clrstack results in a Sementation fault.

(lldb) pe
(lldb) clrstack
(lldb) 
/home/raf/lldb.sh: line 5: 16459 Segmentation fault      sudo lldb-3.9 -o "process attach -p $pid" -o "plugin load ./libsosplugin.so" -o "setclrpath ./" -o "process handle -s false SIGUSR1 SIGUSR2"

After a number of other tests I believe the problem resides in problems loading the required libraries.

SOS/DAC stack unwinding asserts on a stack overflow exception on Linux

When we service stack overflow and other exceptions on Linux we switch to an alternate stack and the SOS/DAC stack unwinder asserts in debug because the stack pointer addresses are not in the threads normal stack range.