sysmon latest version does not work with the Sysmonparser

[azure]

sysmon latest version does not work with the Sysmonparser

[ssi0202]

Will do :-)

[ssi0202]

so after some experimentation this now yields data for event id 3, i can not really see why this works ? ill make a compare tomorrow of the eventID3 section of the parser and this.

let EventData = Event
| where Source contains "Sysmon"
| extend RenderedDescription = tostring(split(RenderedDescription, ":")[0])
| project TimeGenerated, Source, EventID, Computer, UserName, EventData, RenderedDescription
| extend EvData = parse_xml(EventData)
| extend EventDetail = EvData.DataItem.EventData.Data
| project-away EventData, EvData
;
let SysmonEvent1_ProcessCreate=() {
let processEvents = EventData
| where EventID == 3
| extend RuleName = EventDetail.[0].["#text"], UtcTime = EventDetail.[1].["#text"], ProcessGuid = EventDetail.[2].["#text"], ProcessId = EventDetail.[3].["#text"], Image = EventDetail.[4].["#text"],
User = EventDetail.[5].["#text"], Protocol = EventDetail.[6].["#text"], Initiated = EventDetail.[7].["#text"], SourceIsIpv6 = EventDetail.[8].["#text"], SourceIp = EventDetail.[9].["#text"], 
SourceHostname = EventDetail.[10].["#text"], SourcePort = EventDetail.[11].["#text"], SourcePortName = EventDetail.[12].["#text"], DestinationIsIpv6 = EventDetail.[13].["#text"], 
DestinationIp = EventDetail.[14].["#text"], DestinationHostname = EventDetail.[15].["#text"], DestinationPort = EventDetail.[16].["#text"], DestinationPortName = EventDetail.[17].["#text"]
| parse RuleName with * 'technique_id=' TechniqueId ',' * 'technique_name=' TechniqueName  
| project-away EventDetail, RuleName
;
processEvents;
};
(union isfuzzy=true
SysmonEvent1_ProcessCreate,SysmonEvent2_FileCreateTime,SysmonEvent3_NetworkConnect,SysmonEvent4_ServiceStateChange,SysmonEvent5_ProcessTerminate,
SysmonEvent6_DriverLoad,SysmonEvent7_ImageLoad,SysmonEvent8_CreateRemoteThread,SysmonEvent9_RawAccessRead,SysmonEvent10_ProcessAccess,
SysmonEvent11_FileCreate,SysmonEvent12_RegistryObjectAddDel,SysmonEvent13_RegistrySetValue,SysmonEvent14_RegistryObjectRename,
SysmonEvent15_FileCreateStreamHash,SysmonEvent16_ConfigChange,SysmonEvent17_CreateNamedPipe,SysmonEvent18_ConnectNamedPipe,
SysmonEvent19_WMIEventFilter,SysmonEvent20_WMIEventConsumer,SysmonEvent21_WMIEventConsumerToFilter)
| project TimeGenerated,Source,EventID,RenderedDescription,UtcTime,ProcessGuid,ProcessId,Image,User,Protocol,SourceIsIpv6,SourceHostname,Initiated,SourceIp,SourcePort,DestinationHostname,DestinationIp,DestinationPort,DestinationPortName,TechniqueId,TechniqueName

[ssi0202]

there is also an issue with EventID 7, here the hashes are not being parsed / handled correctly

reguarding eventID3

this yield the data i would expect

let SysmonEvent3_NetworkConnect=() {
let processEvents = EventData
| where EventID == 3
| extend rule_name = EventDetail.[0].["#text"], event_creation_time = EventDetail.[1].["#text"], process_guid = EventDetail.[2].["#text"], process_id = EventDetail.[3].["#text"], process_path = EventDetail.[4].["#text"],
user_name = EventDetail.[5].["#text"], network_protocol = EventDetail.[6].["#text"], network_connection_initiated = EventDetail.[7].["#text"], src_is_ipv6 = EventDetail.[8].["#text"], src_ip = EventDetail.[9].["#text"], 
src_host_name = EventDetail.[10].["#text"], src_port = EventDetail.[11].["#text"], src_port_name = EventDetail.[12].["#text"], dst_is_ipv6 = EventDetail.[13].["#text"], 
dst_ip = EventDetail.[14].["#text"], dst_host_name = EventDetail.[15].["#text"], dst_port = EventDetail.[16].["#text"], dst_port_name = EventDetail.[17].["#text"]
| parse rule_name with * 'technique_id=' technique_id ',' * 'technique_name=' technique_name ',' * 'phase_name=' phase_name  
| project-away EventDetail, rule_name
;

[ashwin-patil]

Thanks for reporting the bug for sysmon parser. It seems parser had an extra statement for parsing Hashes but since EventID 3 do not have it , it was failing to parse. I have submitted bugfix with the change in Pull Request #389.

Regarding Event ID 7:
Can you paste sample records (EventData column) for Event Id 7. You can remove any sensitive details such as IP, computer name etc.
Screenshot you provided have different schema than in parser file so i can not verify how it is failing. Having sample source data will help us in testing the parser logic and fix as required.

[ssi0202]

hi here is the data

EventData | <DataItem type="System.XmlData" time="2019-11-26T09:57:57.9094107+01:00" sourceHealthServiceId="215A6DEA-9F44-63F3-7A0A-5A10B1A18F16"><EventData xmlns="http://schemas.microsoft.com/win/2004/08/events/event"><Data Name="RuleName">Defense Evasion - Unmanaged PowerShell Detected</Data><Data Name="UtcTime">2019-11-26 08:57:57.890</Data><Data Name="ProcessGuid">{75F25E05-E915-5DDC-0000-00106814313B}</Data><Data Name="ProcessId">3784</Data><Data Name="Image">C:\Program Files\Microsoft Monitoring Agent\Agent\MonitoringHost.exe</Data><Data Name="ImageLoaded">C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Manaa57fc8cc#\f9fd1c4197933e588a190dfc61a257f6\System.Management.Automation.ni.dll</Data><Data Name="FileVersion">10.0.14393.3053</Data><Data Name="Description">System.Management.Automation</Data><Data Name="Product">Microsoft (R) Windows (R) Operating System</Data><Data Name="Company">Microsoft Corporation</Data><Data Name="OriginalFileName">System.Management.Automation.dll</Data><Data Name="Hashes">SHA1=0AA76356A7A419D85FC3BA33361EC76D97598159,MD5=E5F2E0D35CF2289BF3F6E67ED1FA20CA,SHA256=D294C1685BC9BE4F49C877884D453420A10A0AFD8ABBF40EDE968866EB45EFEB,IMPHASH=00000000000000000000000000000000</Data><Data Name="Signed">false</Data><Data Name="Signature"></Data><Data Name="SignatureStatus">Unavailable</Data></EventData></DataItem>
-- | -
```-


here is what it looks like (currently) from going through the parser

TimeGenerated: [UTC] 2019-11-26T08:57:57.91Z
Source: Microsoft-Windows-Sysmon
EventID: 7
Computer: XXXXXX.XX.local
UserName: NT AUTHORITY\SYSTEM
RenderedDescription: Image loaded
event_creation_time: 2019-11-26T08:57:57.8900000Z
process_guid: {75F25E05-E915-5DDC-0000-00106814313B}
process_id: 3784
process_path: C:\Program Files\Microsoft Monitoring Agent\Agent\MonitoringHost.exe
file_version: 10.0.14393.3053
file_description: System.Management.Automation
file_product: Microsoft (R) Windows (R) Operating System
file_company: Microsoft Corporation
Hashes: System.Management.Automation.dll <- WRONG
module_loaded:C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Manaa57fc8cc#\f9fd1c4197933e588a190dfc61a257f6\System.Management.Automation.ni.dll <- WRONG - i think should be privious data field (ie the name of the dll)
module_is_signed: SHA1=0AA76356A7A419D85FC3BA33361EC76D97598159,MD5=E5F2E0D35CF2289BF3F6E67ED1FA20CA,
SHA256=D294C1685BC9BE4F49C877884D453420A10A0AFD8ABBF40EDE968866EB45EFEB,
IMPHASH=00000000000000000000000000000000 WRONG should be split into seperate fields ie, hashes = SHA256 field name and then the hash value of each hash
module_signature: false WRONG should be in the "module_is_signed" field

i hope this helps.

once we get it working i i'll get started on amending it to fix OSSEM standard to work with the blue-team sentinel attack setup.

as a suggestion going forward i suggest you have the sysmon version that works with the parser available for download on this github or ensure coordination with the sysmon people have an archive available for older version to download ;)

[ashwin-patil]

Thanks for providing the sample data. I will update PR to include original_file_name field which was missing in the parser.

Also, we have already connected with sysmon team and they have plans to generate kql parser via sysmon commandline option to make it convenient. This is on the roadmap until then we can use this parser file.

You can also test below query directly in Sentinel with the sample data you provided to see events post parsing.

let EventID7 = datatable(EventData:string)
[
@'<DataItem type="System.XmlData" time="2019-11-26T09:57:57.9094107+01:00" sourceHealthServiceId="215A6DEA-9F44-63F3-7A0A-5A10B1A18F16"><EventData xmlns="http://schemas.microsoft.com/win/2004/08/events/event"><Data Name="RuleName">Defense Evasion - Unmanaged PowerShell Detected</Data><Data Name="UtcTime">2019-11-26 08:57:57.890</Data><Data Name="ProcessGuid">{75F25E05-E915-5DDC-0000-00106814313B}</Data><Data Name="ProcessId">3784</Data><Data Name="Image">C:\Program Files\Microsoft Monitoring Agent\Agent\MonitoringHost.exe</Data><Data Name="ImageLoaded">C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Manaa57fc8cc#\f9fd1c4197933e588a190dfc61a257f6\System.Management.Automation.ni.dll</Data><Data Name="FileVersion">10.0.14393.3053</Data><Data Name="Description">System.Management.Automation</Data><Data Name="Product">Microsoft (R) Windows (R) Operating System</Data><Data Name="Company">Microsoft Corporation</Data><Data Name="OriginalFileName">System.Management.Automation.dll</Data><Data Name="Hashes">SHA1=0AA76356A7A419D85FC3BA33361EC76D97598159,MD5=E5F2E0D35CF2289BF3F6E67ED1FA20CA,SHA256=D294C1685BC9BE4F49C877884D453420A10A0AFD8ABBF40EDE968866EB45EFEB,IMPHASH=00000000000000000000000000000000</Data><Data Name="Signed">false</Data><Data Name="Signature"></Data><Data Name="SignatureStatus">Unavailable</Data></EventData></DataItem>'
];
EventID7
| extend EvData = parse_xml(EventData)
| extend EventDetail = EvData.DataItem.EventData.Data
| extend RuleName = EventDetail.[0].["#text"], UtcTime = EventDetail.[1].["#text"], ProcessGuid = EventDetail.[2].["#text"], ProcessId = EventDetail.[3].["#text"], Image = EventDetail.[4].["#text"], 
ImageLoaded = EventDetail.[5].["#text"], FileVersion = EventDetail.[6].["#text"], Description = EventDetail.[7].["#text"], Product = EventDetail.[8].["#text"], Company = EventDetail.[9].["#text"], OriginalFileName = EventDetail.[10].["#text"], 
Hashes = EventDetail.[11].["#text"], Signed = EventDetail.[12].["#text"], Signature = EventDetail.[13].["#text"], SignatureStatus = EventDetail.[14].["#text"]
| parse RuleName with * 'technique_id=' TechniqueId ',' * 'technique_name=' TechniqueName
| parse Hashes with * 'SHA1=' SHA1 ',' * 'MD5=' MD5 ',' * 'SHA256=' SHA256 ',' * 'IMPHASH=' IMPHASH
| project-away EventDetail, RuleName

[ssi0202]

perfectly

[ssi0202]

this opens the door to MISP implementation etc thanks for the quick response, really looking forward to the sysmon integration.

[ssi0202]

hate say it but i think you need to take a look at eventID1 too.

TimeGenerated [UTC] | 2019-11-26T19:02:40.233Z
-- | --
  | Source | Microsoft-Windows-Sysmon
  | EventID | 1
  | Computer | XXXXXXXX
  | UserName | NT AUTHORITY\SYSTEM
  | RenderedDescription | Process Create
  | event_creation_time | 2019-11-26T19:02:40.2110000Z
  | process_guid | {29477D06-76D0-5DDD-0200-00107E8526A1}
  | process_id | 60620
  | process_path | C:\Program Files\Microsoft Monitoring Agent\Agent\MonitoringHost.exe
  | file_version | 8.0.13053.0
  | file_description | System Center Management Service Host Process
  | file_product | Microsoft Monitoring Agent
  | file_company | Microsoft Corp.
  | process_commandline | MonitoringHost.exe
  | file_directory | "C:\Program Files\Microsoft Monitoring Agent\Agent\MonitoringHost.exe" -Embedding
  | user_name | C:\windows\system32\
  | user_logon_guid | NT AUTHORITY\SYSTEM
  | user_logon_id | {29477D06-3031-5DB1-0000-0020E7030000}
  | user_session_id | 0x3e7
  | process_integrity_level | 0
  | process_parent_guid | SHA1=6D455AAD1244C2D38B5257912756A1570D927516,MD5=2F45C5F52413FC1582D6BC6A81325A28,SHA256=29A34A3E57DB191A35DAA5A9003127EA2328077B5AFEFB2398C330CBDC484B09,IMPHASH=A7E16841131DA3DC3562014BB1AE5F68
  | process_parent_id | {29477D06-3031-5DB1-0000-0010EC810000}
  | process_parent_path | 940
  | process_parent_command_line | C:\Windows\System32\svchost.exe

seems the hashes does not get handled correctly here too

[ashwin-patil]

I am in-fact currently generating samples for each eventId and checking against the parser event field schema in test lab. I will do review for all and submit changes accordingly to parser by early next week.

[ssi0202]

ok thanks just redid the test you send for eventID 1 to confirm - cool let me know if there is anything i can do on my end and thanks for the quick response. I suppose the blue-team attacking the sentinel will have to update the OSSEM conform paser as well once this is fixed. ;o)

[ashwin-patil]

Yeah. i will submit same changes to sentinel-attack repo as well post testing and review.
Thank you and feel free to continue reporting any issues you see with the parser.

[shainw]

Just to close on this, we will be producing versioned parsers so that as sysmon changes with different event schemas we will still have the parser available. Ashwin will be working on completing that this week.

[shainw]

This is now fixed and commited to master via this PR - #389