microsoft/kqltools

A command line tool to explore real-time streams of events.

Real-Time KQL

To process data with Kusto Query Language (KQL) queries today, users generally have to upload their data to storage first and then query it. The Kql Tools eliminate this need by processing event streams with KQL queries as events arrive, in real-time.

Contents

• List of tools
• Supported event sources
• Supported event destinations
• Contributing

List of Tools

Command Line Tool	Python Module	PowerShell Module
Documentation	Documentation	Documentation
Downloads	Downloads	Downloads
Demo	Demo	Demo

Supported Event Sources

In addition to processing CSV files, the KQL tools support the following input sources:

	Windows	Linux
OS Logs	WinLog - logs seen in EventVwr or log file(s) on disk	Syslog - the OS log
High-Volume Tracing	Etw - Event Tracing for Windows	EBPF - dynamic interception of kernel and user mode functions (Coming soon)

Supported Event Destinations

Real-Time Output	File Output	Upload Output
json- Results printed to standard output in JSON format	json file - Results written to file in JSON format	adx - Upload to Kusto (Azure Data Explorer)
table - Results printed to standard output in table format		blob - Upload as JSON objects to BlobStorage

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

kqltools's Issues

Any plans for a release ?

Are there any plans for a release to just download rather than compile and build ?

Exceptions using the adx argument to export ETL into Azure Data Explorer

I have the following scenario: we have a tool dumping ETW events into an ETL file. I want to ingest this into Kusto. I was experimenting with the adx option, but am facing some issues (and didn't find the documentation to unblock me):

1. What should be my table schema in Kusto? I used a hack, using the Azure Data Explorer one-click ingestion to upload an ETL file to extract the schema. I tried the following schema: EventName: string, Time: real, ProcessName: string, Rest: string, data: string. When I try ingestion using the following command line:

RealTimeKql.exe etl "D:\rEvents.etl" adx --adxcluster=<cluster> --adxdatabase=<db> --adxtable=<tableName>

I see some progress, and then an index out of bounds exception:

Welcome to Real-Time KQL!
Current Batch Count: 5
Current Batch Count: 7
[0]Kusto.Ingest.Exceptions.IngestClientException: An error occurred for source: 'DataReader'. Error: 'Index was outside the bounds of the array.'

2. When I try to debug by providing the exact command line in Visual Studio, I get the following error:

Welcome to Real-Time KQL!
System.Exception: Unexpected TDH status 1168
   at Tx.Windows.EtwTdhEventInfo.ReadTdhMetadata(EtwNativeEvent& e)
   at Tx.Windows.EtwTdhEventInfo..ctor(EtwNativeEvent& e)
   at Tx.Windows.EtwTdhDeserializer.Deserialize(EtwNativeEvent& e)
   at Tx.Windows.EtwTdhEvent.Materialize()
   at Tx.Windows.EtwTdhEvent.get_Keys()
   at RealTimeKqlLibrary.AdxOutput.OutputAction(IDictionary`2 obj) in D:\repos\KqlTools\Source\RealTimeKqlLibrary\Output\AdxOutput.cs:line 111
Stopping RealTimeKql...

Could you provide some pointers on the Kusto ingestion of ETL files?

Web UI

Where can I find a demo for a web ui flavor? In particular I would like to mimic the log analytics query capability but then against one or more cosmosdb containers.

• resulting columns laid out horizontally
• expandable json contents in a column