microsoft/krabsetw

KrabsETW provides a modern C++ wrapper and a .NET wrapper around the low-level ETW trace consumption functions.

Overview

krabsetw is a C++ library that simplifies interacting with ETW. It allows for any number of traces and providers to be enabled and for client code to register for event notifications from these traces.

krabsetw also provides code to simplify parsing generic event data into strongly typed data types.

Microsoft.O365.Security.Native.ETW is a C++ CLI (.NET) wrapper around krabsetw. It provides the same functionality as krabsetw to .NET applications and is used in production by the Office 365 Security team. It's affectionately referred to as Lobsters.

Examples & Documentation

• An ETW Primer.
• Simple examples can be found in the examples folder.
• Please refer to KrabsExample.md and LobstersExample.md for detailed examples.
• SampleKrabsCSharpExe is a non-trivial example demonstrating how to manage the trace objects.
• Using Message Analyzer to find new ETW event sources.

Important Notes

• krabsetw and Microsoft.O365.Security.Native.ETW only support x64. No effort has been made to support x86.
• krabsetw and Microsoft.O365.Security.Native.ETW are only supported on Windows 7 or Windows 2008R2 machines and above.
• Throwing exceptions in the event handler callback or krabsetw or Microsoft.O365.Security.Native.ETW will cause the trace to stop processing events.
• The call to "start" on the trace object is blocking so thread management may be necessary.
• The Visual Studio solution is krabs\krabs.sln.
• When building a native code binary using the krabsetw package, please refer to the compilation readme for notes about the TYPEASSERT and NDEBUG compilation flags.

NuGet Packages

NuGet packages are available both for the krabsetw C++ headers and the Microsoft.O365.Security.Native.ETW .NET library:

• https://www.nuget.org/packages/Microsoft.O365.Security.Native.ETW/
• https://www.nuget.org/packages/Microsoft.O365.Security.Native.ETW.Debug/ (for development - provides type asserts)
• https://www.nuget.org/packages/Microsoft.O365.Security.Krabsetw/

For verifying the .NET binaries, you can use the following command: sn -T Microsoft.O365.Security.Native.ETW.dll

The expected output is:

Microsoft (R) .NET Framework Strong Name Utility  Version 4.0.30319.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Public key token is 31bf3856ad364e35

Community & Contact

Please feel free to file issues through GitHub for bugs and feature requests and we'll respond to them as quickly as we're able.

krabsetw's Issues

.Net and Service Fabric provider(s) issue

I am having issues using the managed wrapper to consume Service Fabric events from a simple .Net console program. The only provider which seems to be working is the Lease provider where ServiceFabricETWProviderGuids() is defined as per ServiceFabricETWProviderGuids

I would like to be able to consume the following List of Service Fabric events.

O365.Security.Etw.EventRecord.Name does not return the event name.

It returns the provider name:

virtual property String^ Name
{
    String^ get() { return gcnew String(schema_->provider_name()); }
}

find_property method in parser.hpp - cache having duplicated entries

row 179, parser.hpp
needto to following fix (move caching inside if statement)
otherwise it is occasionally duplicated
...

• property_info propInfo(pBufferIndex_, currentPropInfo, propertyLength);
• cache_property(pName, propInfo);

// The property was found, return it
if (name == pName) {

• property_info propInfo(pBufferIndex_, currentPropInfo, propertyLength);

• cache_property(pName, propInfo);

  return propInfo;
  }

Support socket addresses in krabsetw and lobsters

IPSEC ETW provider ({0C478C5B-0351-41B1-8C58-4A6737DA32E3}) has a socket address field on some events. It's a bytestream that we could parse out correctly into a sane data type.

Sample code as follows:

SocketAddress localSocketAddress = new SocketAddress((AddressFamily)localAddress[0]);
for (int i = 0; i < localAddress.Length; i++)
{
localSocketAddress[i] = localAddress[i];
}


// Link: http://msdn.microsoft.com/en-us/library/windows/desktop/aa814468(v=vs.85).aspx
pattern AddressFamily = enum SHORT
{
    AFUnspec = 0,
    AFInet = 2,
    AFInet6 = 23
    ...
};

//
// http://msdn.microsoft.com/en-us/library/windows/desktop/ms740496(v=vs.85).aspx
// Use value parameter type since there's no aspect to limit a field's size.
type SocketAddress[uint SockAddrLen]
{
    (SockAddrIn | SockAddrIn6 | SockAddr) Address;
    // Padding to the total size in bytes
    optional ([|SockAddrLen > AddressSize()|] binary) Padding with BinaryEncoding{Length = SockAddrLen - AddressSize()};
    
    // Get the size of Address field in bytes
    uint AddressSize()
    {
        switch (Address)
        {
            case SockAddrIn6 =>
                return 28;
            case SockAddrIn =>
                return 16;
            case SockAddr =>
                return 16;
        }
        return 0;
    }
    
    // Get the IPv4/IPv6 address
    (IPv4Address | IPv6Address) GetAddress()
    {
        switch (Address)
        {
            case addr:SockAddrIn =>
                return addr.Address;
            case addr:SockAddrIn6 =>
                return addr.Address;
        }
        return null;
    }

    bool IsIPv4()
    {
        return Address is SockAddrIn || Address is SockAddr;
    }
    
    bool IsIPv6()
    {
        return Address is SockAddrIn6;
    }
    
    override string ToString()
    {
        return Address.ToString();
    }
}

// sockaddr
// Link: http://msdn.microsoft.com/en-us/library/windows/desktop/aa814468(v=vs.85).aspx
type SockAddr
{
    USHORT Family where ValidationCheck(value in {2, 23}, null, "WindowsReference: The Family in type SockAddr must be 2 (AF_INET) or 23 (AF_INET6).");
    binary Data with BinaryEncoding{Length = 14};
} with BinaryEncodingDefaults{Endian = Endian.Big};

// sockaddr_in
type SockAddrIn
{
    AddressFamily Family where value == AddressFamily.AFInet
        with BinaryEncoding{Endian = Endian.Little};
    USHORT Port;
    IPv4Address Address;
    binary Zero with BinaryEncoding{Length = 8};
    
    override string ToString()
    {
        return Address.ToString() + ":" + (Port as string);
    }
} with BinaryEncodingDefaults{Endian = Endian.Big};

// sockaddr_in6
type SockAddrIn6
{
    AddressFamily Family where value == AddressFamily.AFInet6
        with BinaryEncoding{Endian = Endian.Little};
    USHORT Port;
    ULONG FlowInfo;
    IPv6Address Address;
    ULONG ScopeId;
    
    override string ToString()
    {
        return Address.ToString() + ":" + (Port as string);
    }
} with BinaryEncodingDefaults{Endian = Endian.Big};

// Link: http://msdn.microsoft.com/en-us/library/ms740504(VS.85).aspx
type SockAddrStorage
{
    AddressFamily SockAddrStorageFamily where ValidationCheck(value in {0, 2, 23}, null, "WindowsReference: The SockAddrStorageFamily in type SockAddrStorage must be 0 (AF_UNSPEC), 2 (AF_INET), or 23 (AF_INET6).");
    array<byte> SockAddrStoragePad1 with BinaryEncoding{Length = 6};
    INT64 SockAddrStorageAlign;
    array<byte> SockAddrStoragePad2 with BinaryEncoding{Length = 112}; // Length = 128 - 2 - 6 - 8
    
    override string ToString()
    {
        return "AddressFamily = " + EnumToString(SockAddrStorageFamily, "WindowsReference.AddressFamily");
    }
}

Cross platform and .Net Core Support

Consider adding cross platform and .Net Core Support. Ideally this will include full support for the new & evolving .Net Core EventCounter design.

I would like to use this high performance library in .Net Core microservices that run on Windows and Linux.

Missing break statement in String->GUID lookup when creating user mode providers.

When you create a new provider with code like the following:

krabs::provider<> provider(L"Microsoft-Windows-PowerShell");

This code results in a longer startup code path that looks up the GUID associated with the provided friendly name.

However, once we find the match and copy the GUID required, we don't break out of the loop:

https://github.com/Microsoft/krabsetw/blob/0b42d12c337d2ea954ebc89c1547be48a552cf13/krabs/krabs/provider.hpp#L439

This should just require a break statement.

Incorrect EventRecordMetadata->Opcode Size

Maybe I'm confused but EventRecordMetadata specifies that the Opcode type is byte:
https://github.com/Microsoft/krabsetw/blob/master/O365.Security.Native.ETW/EventRecordMetadata.hpp#L43

However Opcode should be a UInt32 (I think). Consider the Kernel trace example here:
https://github.com/Microsoft/krabsetw/blob/master/examples/ManagedExamples/KernelTrace001.cs#L39

Opcode=0x1 -> EVENT_TRACE_FLAG_PROCESS

Looking at the other options on MSDN shows most of them actually don't fit into into a UInt8.
https://docs.microsoft.com/en-gb/windows/desktop/ETW/msnt-systemtrace

Linking error with c++ project.

I am seeing below linking error

unresolved external symbol "private: static class krabs::schema_locator krabs::schema_locator::singleton_" (?singleton_@schema_locator@krabs@@0V12@A)

My code is only

#include
#include
#include
#include <condition_variable>
#include "../packages/Microsoft.O365.Security.Krabsetw.2.0.2/lib/native/include/krabs.hpp"
int main()
{
const EVENT_RECORD record;
krabs::schema schema(record);
}

I have installed Nuget package Microsoft.O365.Security.Krabsetw version 2.0.2.
I tried both VS 2017(15.9.9) & 2019.

Appreciate any help.
Cheers.

provider ctor with provider name instead of Guid has a couple of COM bugs

Exceptions are thrown when using CoInitialize. A wrapper should be created to ensure the appropriate CoUninitialize occurs.

Remove Boost library dependency

KrabsETW has a dependency on Boost but only because of a single source file (comparers.hpp). The four Boost-dependent functions in here can be easily re-coded to eliminate the Boost dependency altogether. Moreover, this results in a (marginally) faster implementation. Is this something you could consider doing? The need to download a huge Boost nuget completely unnecessarily is a bit of a turn-off in some projects.

Missing Registry events.

Hi Zac

I have created a registry etw session using krabsetw. However it seems I am missing quite a few events. I wrote a test exe that sets reg value & never see those events. I created a session using Message Analyzer & it catches those events

See the image below comparing both sessions. Don't see much different.

I tried increasing info.properties.MaximumBuffers to make total 512kb buffer. With this I start seeing the events but delayed by almost 1-2 minutes.

Is this because my consumption rate is low? How can MMA consume faster? Is it possible to add multi-threaded consumers?

Appreciate any pointers.

Cheers
Rushikesh

view_of and predicates don't work for fixed-width strings

What API souuld I use to consume ETW?

What API souuld I use to consume ETW:

1. System.Diagnostics.Tracing.EventSource (BCL)
2. Microsoft.Diagnostics.Tracing.EventSource (NuGet)
3. Microsoft.Diagnostics.Tracing.TraceEvent (NuGet)
4. krabsetw (NuGet)
   ?

I read this at https://blogs.technet.microsoft.com/office365security/hidden-treasure-intrusion-detection-with-etw-part-2/:

TraceEvent is a library used by the PerfView tool and has the benefits of being a well-designed .NET API. Unfortunately, it doesn’t perform well for scenarios where we want to keep memory usage to a minimum. System.Diagnostics.Tracing has the advantage of being part of the .NET BCL but we’ve observed intermittent exceptions and unexpected behavior in the past. Additionally, it suffers from the same memory consumption issue that TraceEvent does.
In response to these challenges, Office 365 Security chose to implement our own API with three primary goals:
•Intuitive and flexible API
•High performance – filtering events in the native layer
•Available both in .NET and native C++
The result of this work is krabsetw, a library we’ve open-sourced under the MIT license. It contains both a native C++ API as well as a .NET API. This library is used in production today across Office 365 workloads on more than 100,000 machines. With filtering, we’re able to process more than more than 500 billion events per day, generating more than 7TB of data per day across the machines.

Is this still true?.. It is written one year ago (May 9, 2017), and other API's have updates since.
I want to know what API I can depend on, if not now, at least in upcoming updates.

Another question: Do I have to use these NuGets too?
Microsoft.Diagnostics.Tracing.TraceEvent.SupportFiles
Microsoft.Diagnostics.Tracing.EventRegister

Add ability to configure trace performance characteristics

re: https://github.com/microsoft/krabsetw/blob/master/krabs/krabs/nightmare.hpp#L218

I'd love to be able to configure BufferSize as well as MinimumBuffers, MaximumBuffers and FlushTimer (including via the .NET wrapper) for new traces. :-)

My trace was dropping events so I was hoping to play with the buffer settings to see if I could resolve this.

Update nuget package

Nuget package needs updating for VS2017.

Does krabsetw support resolving syscall address and return address in call-stacks?

syscall field in SysCallEnter event is a memory address
values in the call-stack event are also memory address.

Does krabsetw support resolving syscall and call-stacks?
e.g. 0x12312412312 -> NtCreateFile

Support for x86

Hi,

What are the limitations of using native (C++) krabsetw on x86 / 32-bit machine?
The README mentioned that:

krabsetw and Microsoft.O365.Security.Native.ETW only support x64. No effort has been made to support x86.

But I made a simple program compiled for x86 which used native krabsetw, similar to kernel_trace_001 in NativeExamples, and it seemed to work just fine.

Maybe a few bullet points on README regarding the internal / implementation limitations of krabsetw on x86 would help to clarify it.

Doesn't compile under C++17

I have had to downgrade the Visual Studio C++ compiler to the C++14 standard (/std:c++14) to compile because some types, such as unary_function have been deprecated in C++17.

Is it possible to update the code to be compatible with the latest version of the Visual Studio C++ compiler?

Note that I'm using the krabsetw NuGet package.

Publish official NuGet packages for krabsetw and O365.Security.Native.ETW.

File IO provider schema_not_found error!

Hi Zac,

While logging kernel events for File IO events, I came across the run time error "krabs::could_not_find_schema at memory location ...". Can you please help me to solve this issue?

There are two scenarios:

1. The above error occurs only when the following flag is enabled: EVENT_TRACE_FLAG_FILE_IO_INIT in the kernel_providers.hpp at
   CREATE_CONVENIENCE_KERNEL_PROVIDER(
   file_init_io_provider,
   EVENT_TRACE_FLAG_FILE_IO_INIT,
   krabs::guids::file_io);

2. If I enable the following flag EVENT_TRACE_FLAG_DISK_FILE_IO (and disable the flag EVENT_TRACE_FLAG_FILE_IO_INIT) in kernel_providers.hpp, the program runs fine. I need to enable both the flags to get a wide range of file events.

After digging into the code, I think (not sure) that the problem occurs when retrieving the cached data in function schema_locator::get_event_schema(const EVENT_RECORD& record) of schema_locator.hpp. It is because it locks certain schema using scope_lock.

My code is here:
#include
#include
#include
#include <condition_variable>
#include "....\krabs\krabs.hpp"
#include "examples.h"
#include <atlstr.h>
#include <Windows.h>
#include
#include <stdio.h>
#include <psapi.h>
#include <tlhelp32.h>
#include
#include

static void setup_fileio_provider(krabs::kernel::file_io_provider& fileioprovider);
std::ofstream trace_file;

void kernel_trace_001::start()
{

    krabs::kernel_trace trace(L"MyMagicTrace");
    trace_file.open ("c:\\mytrace.txt", std::ios::out | std::ios::binary);

    krabs::kernel::file_io_provider fileio_provider; 
krabs::kernel::file_init_io_provider fileioinit_provider;

setup_fileio_provider(fileio_provider);

trace.enable(fileio_provider);
trace.enable(fileioinit_provider); 

trace.start();
   trace_file.close();

}

void setup_fileio_provider(krabs::kernel::file_io_provider& provider) {
provider.add_on_event_callback([](const EVENT_RECORD &record) {
krabs::schema schema(record);
krabs::parser parser(schema);

	if ( schema.event_opcode() == 64 ) {
                   std::wstring filename = parser.parse<std::wstring>(L"OpenPath");
		std::uint32_t fileattr = parser.parse<std::uint32_t>(L"FileAttributes");
		std::uint32_t shareaccess = parser.parse<std::uint32_t>(L"ShareAccess");
		std::uint32_t* fileobj = parser.parse<std::uint32_t*>(L"FileObject");
		
		std::wcout << "File " << schema.event_name() << "\t" << schema.timestamp() << "\t" << schema.process_id() << "\t" << schema.thread_id() << "\t" << fileobj << "\t" << shareaccess << "\t" << filename << "\n";
	}
	
	if (schema.event_opcode() == 70 || schema.event_opcode() == 71) 
	{
		std::uint32_t ttid = parser.parse<std::uint32_t>(L"TTID");
		std::uint32_t* filekey = parser.parse<std::uint32_t*>(L"FileKey");
		std::uint32_t* fileobj = parser.parse<std::uint32_t*>(L"FileObject");

		std::wcout << "File " << schema.event_name() << L"\t" << schema.timestamp() << "\t" << schema.process_id() << "\t" << schema.thread_id() << "\t" << filekey << "\t" << fileobj << "\n";
	} 
	
	if (schema.event_opcode() == 32 || schema.event_opcode() == 35 || schema.event_opcode() == 36)
	{
		std::wstring filename = parser.parse<std::wstring>(L"FileName");
		std::uint32_t* fileobj = parser.parse<std::uint32_t*>(L"FileObject");

		std::wcout << "File " << schema.event_name() << "\t" << schema.timestamp() << "\t" << schema.process_id() << "\t" << schema.thread_id() << "\t" << fileobj << "\t" << filename << "\n";
	} 
	
	if (schema.event_opcode() == 68 )
	{
		std::uint32_t ttid = parser.parse<std::uint32_t>(L"TTID");
		std::uint32_t* filekey = parser.parse<std::uint32_t*>(L"FileKey");
		std::uint32_t iosize = parser.parse<std::uint32_t>(L"IoSize");
		std::uint32_t ioflags = parser.parse<std::uint32_t>(L"IoFlags");

		std::wcout << "File " << schema.event_name() << "\t" << schema.timestamp() << "\t" << schema.process_id() << "\t" << ttid << "\t" << filekey << "\n";
	}

});

}

Unable to parse Schannel events

I'm unable to get the schema properties or event_name from the Schannel (1f678132-5938-4686-9fdc-c8ff68f15c85) provider. I note that the same event is processed correctly in Microsoft Message Analyzer when I capture live events simultaneously on the same Windows 10 machine.

The provider_name and event_id values appear to be valid, however, I only get gibberish from the event_name method, and an exception is thrown when I try to enumerate the properties via parser.

The same code works well for the Microsoft-Windows-SChannel-Events provider. Is there something that I should be doing differently for the Schannel provider?

compilation errors in the library

Hi, I'm trying to get the C++ version to compile, but I'm getting some errors from the library's code.

line 100 of kt.hpp
C2440 'return': cannot convert from 'const char [17]' to 'std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t>>'

 inline const std::wstring kt::enforce_name_policy(
        const std::wstring &name_hint)
    {
        if (IsWindows8OrGreater()) {
            return krabs::details::ut::enforce_name_policy(name_hint);
        }

        return KERNEL_LOGGER_NAME;
    }

Fixing that is simple, I can change it to
return KERNEL_LOGGER_NAMEW;

After fixing that, there are 3 errors in nightmare.hpp, also type errors, so that's probably not the end of it.

Did I miss a crucial step for getting the library to compile?

Make a Debug version of the O365.Security.Native.ETW NuGet package

Feedback for Windows diagnostics team about Event Viewer UI improvements

Hello, I know this isn't the repository by Windows / TechNET diagnostics team, yet I would like to share some thoughts on the user experience. I hope this community feedback will be forwarded to the internal team responsible for Event Viewer.

In year 2018, if we look at Event Viewer UI and compare it to other products like Seq (a Windows service, that provides an interactive web interface with useful controls):

turns out that Event Viewer is not only slow in delivering relevant information in time (load, refresh, search timing) but lacks very many rich usability features that renders it a past-century product.

Please consider introducing a new UI for Event Viewer, preferably a Web-based product reachable by intr[ae]nets with Windows auth etc. If web is not an option, new WPF based viewer will be equally welcome!

Thanks!

Parsing erros for fields at the end of the record

Hi,

I've ran into some problems while consuming events from the Microsoft-Windows-WinINet provider ({43d1a55c-76d6-4f7e-995c-64c711e5cafe}) on Windows 10.

Sometimes the last properties in the record cannot be parsed.

Sometimes the problematic properties (e.g. Event id: 1008 Prop: StatusLine, Event id: 506 Prop: URL) seem to be zero-length, non-null-terminated strings at the end of the record (i.e. there is a record with zero bytes of data). These hit the assertion 'invariant: if we've exhausted our buffer, then we must've exhausted the properties as well'. It seems that the end of the buffer might not strictly mean the end of properties.

There are also integer types that either throw an exception ('Property length past end of property buffer') or hit the assertion.

I have not observed similar issues with other providers so far.

Please find attached a list of event ids and property names from the Microsoft-Windows-WinINet provider that I have observed causing problems. Note that these don't always cause problems, but some properties (StatusLine, URL) seem to have a high probability to hit the assertion.

--Markus
Microsoft-Windows-WinINet_parsing_issues.txt

Lobsters examples seems outated

Hi Zac, i was trying to use the lobster examples and Visual Studio displayed the following errors:

1. The delegate in "powershellProvider.OnEvent += (EventRecord record)" expects IEventRecord, not EventRecord

2. I coudn't find the classes Schema and Parser. If i have to implement those classes, could you please provide an example of how to do it?

I'm new to ETW by the way, thanks for your lib.

Add ability for krabsetw trace sessions to be updated while running

A user has requested that krabsetw support modifying trace options while the trace is running. This would require updates to the API surface for user-mode and kernel-mode traces.

The Win32 APIs to use for this are distinct for kernel-mode vs user-mode trace sessions:

• kernel-mode: ControlTrace using the EVENT_TRACE_CONTROL_UPDATE value for the ControlCode argument
• user-mode: EnableTraceEx2 using the EVENT_CONTROL_CODE_DISABLE_PROVIDER value for the ControlCode argument

Ideally, updates to the krabsetw APIs will keep track of the current state of settings before they're changed. This isn't a hard requirement but it would reduce some burden on API users.

Tracing Microsoft-Windows-Security-Auditing

Not sure if its krabsetw issue but I have been failing to trace Microsoft-Windows-Security-Auditing provider & hoping someone would already know this.

I tried running my code as LocalSystem as well tried different privileges. Also compared EventLog's process token with my process.

Logman query provider pid (EventLog Pid) doesn't even show Security-Auditing, yet it gets the .
EventLog service's call stack does seem to indicate it gets the security events with realtime trace.

I looked at some other thread where it was indicated that subscribing to Eventlog-Security is not possible with krabsetw & has some known issues. Not sure if its related but would be good to know what issues.

Appreciate any help.

Use with Powershell

Dear, is it possible to use this library from Powershell?
If so, would you be so kind to provide some examples..

Possible bugs when capturing events.

Hi again Zac.
I'm having a strange result when trying to use lobster to process the events of a example EventSource i made.

Here is the event source and console aplication that runs it:

using System;
using System.Collections.Generic;
using System.Diagnostics.Tracing;
using System.IO;
using System.Linq;
using System.Threading.Tasks;

namespace ETWeEnerator
{
    [EventSource(Name = "EventTrigger", Guid = "835828ae-38d2-51c8-51b8-0a6ee280d4a8")]
    public sealed class EventTrigger : EventSource
    {
        public class Keywords
        {
            public const EventKeywords Network = (EventKeywords)1;
            public const EventKeywords Database = (EventKeywords)2;
            public const EventKeywords Diagnostics = (EventKeywords)4;
            public const EventKeywords Performance = (EventKeywords)8;
        }

        public class Taks
        {
            public const EventTask LogInfo = (EventTask)1;
            public const EventTask LogError = (EventTask)2;
        }

        public class Tags
        {
            public const EventTags Info = (EventTags)1;
            public const EventTags Error = (EventTags)2;
        }

        public EventTrigger() : base("EventTrigger")
        {
        }

        public static EventTrigger Log = new EventTrigger();

        [Event(1, Opcode = EventOpcode.Info, Level = EventLevel.Informational, Keywords = Keywords.Network, Message = "{0}", Tags = Tags.Info)]
        public void LogEvent(string message)
        {
            base.WriteEvent(1, message);
        }

        [Event(2, Opcode = EventOpcode.Info, Level = EventLevel.Error, Keywords = Keywords.Network, Message = "{0}", Tags = Tags.Error)]
        public void LogError(string message, string ex)
        {
            base.WriteEvent(2, message, ex);
        }
    }
}

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading.Tasks;

namespace ETWeEnerator
{
    public class Program
    {
        private static void RaiseEvents()
        {
            Console.WriteLine("Loging info");
            EventTrigger.Log.LogEvent("Some message");
            Console.WriteLine("Creating Error");
            try
            {
                throw new Exception("Some exception");
            }
            catch (Exception ex)
            {
                EventTrigger.Log.LogError("some error", ex.Message);
            }
            Console.WriteLine("Finished");
            Console.ReadKey();
            RaiseEvents();
        }

        public static void Main(string[] args)
        {
            Console.WriteLine("Starting");
            Console.WriteLine("Creating Xml");
            string xml = EventTrigger.GenerateManifest(typeof(EventTrigger), ".");
            File.WriteAllText("eventrigger.xml", xml);
            RaiseEvents();
        }
    }
}

And here is the lobster consumer:

using O365.Security.ETW;
using System;
using System.Collections.Generic;
using System.Diagnostics.Tracing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Testing = O365.Security.ETW.Testing;

namespace krabs462
{
    public class EventTrigger : EventListener
    {
        protected override void OnEventWritten(EventWrittenEventArgs eventData)
        {
            Console.WriteLine(eventData.EventId);
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            var trace = new UserTrace();
            
            var provider = new Provider(Guid.Parse("{835828ae-38d2-51c8-51b8-0a6ee280d4a8}"));
            
            provider.Any = Provider.AllBitsSet;
            
            provider.OnEvent += (IEventRecord record) =>
            {

                if (record.Id == 1)
                {
                    Console.WriteLine("Message: " + record.GetAnsiString("message"));
                }

                if (record.Id == 2)
                {
                    Console.WriteLine("Message: " + record.GetAnsiString("message"));
                }
            };
            
            trace.Enable(provider);
            
            Task.Factory.StartNew(() => trace.Start());

            Console.WriteLine("Tarce started");
            Console.WriteLine("Press any key to stop");
            Console.ReadKey();
            trace.Stop();
            Console.WriteLine("Press any key to exit");
            Console.ReadKey();
        }
    }
}

Now the issues...

1. The ids of the events captured by the lobster consumer are 4 and 5, not the values 1 (info) and 2 (error) set in the methods LogEvent and LogError of the EventTrigger class.
2. The properties "message" and "ex" exists on those events, but their values are not the ones i passed when they are raised ("some message" and "some exception").

I used perfview.exe to check if the events are raised by EventTrigger are correct and seems everything is ok:

Is this a bug or am i missing something?

Thanks again.

kernel-network

for event 10, 12; sport, dport does not seem to be parsed out. Is it just me?

Does not compile with /permissive-

Latest versions of VS2017 have /permissive- as the default setting but unfortunately krabsetw does not compile with it enabled:

1>c:\users\<foobar>\documents\visual studio 2017\projects\krabstesting\packages\krabsetw.1.0.11\lib\native\include\krabs\filtering\comparers.hpp(100): error C2760: syntax error: unexpected token '}', expected ';'

1>c:\users\<foobar>\documents\visual studio 2017\projects\krabstesting\packages\krabsetw.1.0.11\lib\native\include\krabs\filtering\comparers.hpp(101): note: see reference to class template instantiation 'krabs::predicates::comparers::iequal_to<T>' being compiled

Used versions:
krabsetw: 1.0.11 nuget package
VS 2017 15.6.5

Wrong LogFileMode default flag initialization

Noticed this when I was trying to disable EVENT_TRACE_NO_PER_PROCESSOR_BUFFERING in a kernel tracer. When looking by this flag that is actually enabled in the tracer, I didn't find any reference to it. That's when I discovered that the info.properties.LogFileMode is wrongly been filled with a flag that do not belong to it. PROCESS_TRACE_MODE_EVENT_RECORD. This should or removed or changed to EVENT_TRACE_NO_PER_PROCESSOR_BUFFERING as they share the exact same value.

https://github.com/Microsoft/krabsetw/blob/master/krabs/krabs/nightmare.hpp#L204

MSDN:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa364080(v=vs.85).aspx
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363780(v=vs.85).aspx

Tdh_helpers pointer and size_t support

Hi Zac,
Krabs/Krabs/parser.hpp(251) when callkng parser::parser
With size_t, the function Krabs::debug:: asset_valid_assugnment passing to
Tdh_helpers.hpp throw_if_invalid
Compares TDH_INTYPE_UINT64 and TDH_INTYPE_POINTER (if event property if pointer)
e.g in page_fault event property: BaseAddress and RegionSize
Thanks alot

Feature Proposition: adding support for LINUX.... Would like having your opinions about that.

Last half year I am working on Linux programming and using Netlink api. I think there is an potion to create krabs provider running on Linux. It is not a simple project. Netlink is the socket which can be opened by application running as administrator (root) to kernel and it gets messages about what is happening, e.g. process start stop, ip routing etc... there are additional apis that can be used to leasten for filesystem, etc...
The code can be easily organized as VS project and compiled using Linux add in in Visual studio 2015 and later.
The question is the krabs authors are interested in such an addition. (and in future - may be adding Mac).
I would like to know your opinions about that.
Thanks.

How to extract Processor Number field for Syscall/Cswitch events?

I've been testing the Kernel Trace "syscall" event provider as follows:

logman start "NT Kernel Logger" -p "Windows Kernel Trace" (syscall,cswitch) -o systemevents.etl -ets
tracerpt systemevents.etl -o systemevents.csv -of CSV

The output from tracerpt indicates that the "syscall" and "cswitch" events include a "Processor Number" property somewhere within the event data. I've tried creating a "syscall" trace via krabsetw, but I can't seem to find a way to extract the "Processor Number" property - am I missing something obvious, or has this not been implemented within krabsetw?

Add RelatedActivityId to the events

ActivityId and RelatedActivityId header fields aren't available through the API but were available in the old TraceEvent library are useful for correlating events.

Possible synchronization issue in schema_locator

Hello, in the file schema_locator.hpp line 145 about the following function:

inline const PTRACE_EVENT_INFO schema_locator::get_event_schema(const EVENT_RECORD& record)
    {
        // check the cache
        auto key = schema_key(record);
        auto& buffer = cache_[key];

        // return if there's a cache hit
        if(buffer) return (PTRACE_EVENT_INFO)(&buffer[0]);

        auto temp = get_event_schema_from_tdh(record);

        // multiple threads may end up trying to save their
        // temp objects to the cache so we need to lock
        // so only the first entry should be cached.

        if(!buffer)
        {
            scope_lock l(sync_);
            if (!buffer) buffer.swap(temp);
        }

        return (PTRACE_EVENT_INFO)(&buffer[0]);
    }

The access to the unordered_map is not protected by synchronization mechanism at auto& buffer = cache_[key]; an insertion can occur and that can result in a rehashing of the container, then all iterators will be invalidated.
If another thread is reading the container at the same exact moment, the behavior will be undefined. I think that using a "singleton double checked locking pattern" for insertion is not enough, a full locking is required to completely eliminate the threat...
Mathieu

enable subscription to Security Event Log (was "Question about subscribing to the Security Log")

Hi again Zac.
I just would like to know if its possible for lobster to subscribe to the "Security" Log under "Windows Logs" in event viewer.

I saw this old and unaswered question on stack overflow and i started to wonder if it can be done.

Thanks again.

how to get the stack info when receive evnet?

It seems need set EVENT_ENABLE_PROPERTY_STACK_TRACE when call EnabeTraceEx2.
but i can't see any api to enbale this.
Thanks.

krabsetw performance

Any ideas on handling memory/performance while performing long running traces. Wrote an app utilizing a few providers (works great btw, thanks!) and memory just keeps increasing, similar to memory leak. Not a coder so may be just generally bad code, but how best to manage memory, cpu? After 10hrs are to near 1gb of memory used, steadily increasing....

Null terminator being returned in null terminated fields

Fixed in #14

WPP trace message capture

Hi,

I'm trying to create an application that captures events produced by WPP in a kernel-mode driver and I found krabsetw quite useful for this task. Basically creating and starting a session using krabs::user_trace works as expected.

The issue is that ut::forward_events() checks the record.EventHeader.ProviderId in each event and forwards it only if this field matches the provider GUID. I found that in WPP messages the EventHeader.ProviderId field is not set to the provider GUID. It is set to some local GUID generated by WPP instead. So WPP messages cannot pass the check in ut::forward_events().

The documentation states that the EVENT_HEADER_FLAG_TRACE_MESSAGE flag indicates a WPP message. So I fixed that by modifying the check as follows:

if (0 != (record.EventHeader.Flags & EVENT_HEADER_FLAG_TRACE_MESSAGE) ||
record.EventHeader.ProviderId == provider.get().guid_) {
provider.get().on_event(record);
}

It would be great if this check could be made configurable.
Or could it be removed completely?
Maybe there should be a separate trace type for WPP, e.g. krabs::wpp_trace.

Thanks
Udo

Enabling a trace when there are no more sessions available fails silently

1 Run the test app creating a new GUID session each time.
2 Verify the trace is receiving events.
3 Kill the test app (instead of allowing dtors tor run)
4 Repeat until eventually step 2 fails because the session hasn't actually started/registered properly.

Workaround: delete the stale trace sessions.

We should fail with an error or something that will let the caller know the session isn't set up properly.

Remove references to the retired Microsoft Message Analyzer in the documentation

Microsoft Message Analyzer (MMA) was retired and its download packages removed from microsoft.com sites on November 25 2019. There is currently no Microsoft replacement for Microsoft Message Analyzer in development at this time.

-- https://docs.microsoft.com/en-us/openspecs/blog/ms-winintbloglp/dd98b93c-0a75-4eb0-b92e-e760c502394f

Question about IEventRecord

What is the magic number on this line?
https://github.com/Microsoft/krabsetw/blob/6240e000ec1b52d49425616eeae3060ac12a4564/examples/ManagedExamples/UserTrace001.cs#L38

I assume this is some number signifying the event type, but where is the reference?
I have been mucking around with krabstew to build a tool to extract some information from ETW. I think i'm getting what I'm after, but the last piece that I can't figure out is how to translate the IEventRecord to actual data. I guess i need the provider's schema or MOF? But I cannot for the life of me find any reference for these anywhere in MSDN or etc. Anything to point me in the right direction would be appreciated.

Thanks!

KERNEL_LOGGER_NAME define issue

Hi! I'm using the NuGet 1.0.12 package, wrote a simple script just to import <krabs.hpp>, and get the following error:
no suitable constructor exists to convert from "const char [17]" to "std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t>>"

in kt.hpp:100

Am I doing anything wrong?

Thanks!

Native version of kernel trace examples seem to be busted?

The native versions of the kernel trace examples don't seem to process data correctly. Investigating.

Parsing TDH_INTYPE_WBEMSID

I created a tiny wrapper for a number of ETW events, however there are some TDH_IN_TYPE types which I am unable to parse. For example:

Here type 310 is TDH_INTYPE_WBEMSID which should be a TOKEN_USER struct followed by a SID but I can't find a way to access the value of this property. I have a switch(prop.Type)->case statement to handle the different types, eg:

case (int)TDH_IN_TYPE.TDH_INTYPE_UINT32:
    propertyValue = record.GetUInt32(prop.Name);
    break;

If I can cast the result as an IntPtr I can make it work: Increment the pointer by the size of TOKEN_USER and then call LookupAccountSid etc. However I can't make it work, if I try something like this I get an exception saying the requested size does not match the size of the parameter.

case (int)TDH_IN_TYPE.TDH_INTYPE_WBEMSID:
    IntPtr pRecod = (IntPtr)record.GetUInt64(prop.Name );
    break;

Thanks for the great wrapper!

Better performance in event parser property_iterator

Hi,
I am using ETW (and krabs) very intensively in my code, I am parsing different events at a very high rate (filesystem, registry...)
To increase the performance of the parsing I am using the helpful propery_iterator instead of calls to find_property to eliminate the parser.propertyCache_ population. (to not use the find_property function, I am caching the result of size_provider::get_property_size too, then I am using pointer arithmetic to access the data in UserData)
I changed the member name_ in the class property in property.hpp used by the class property_iterator to be a const wchar_t* instead of a std::wstring. That permits me to reduce drastically the allocation rate of my code.
I think this low-level infrastructure should postpone such conversion and give the choice to the consumer to transform it (in my case with c++17 string_view give me a way to modernly work without reallocation by example)
What do you think?
Mathieu