azure/azure-sentinel

Cloud-native SIEM for intelligent security analytics for your entire enterprise.

Microsoft Sentinel and Microsoft 365 Defender

Welcome to the unified Microsoft Sentinel and Microsoft 365 Defender repository! This repository contains out of the box detections, exploration queries, hunting queries, workbooks, playbooks and much more to help you get ramped up with Microsoft Sentinel and provide you security content to secure your environment and hunt for threats. The hunting queries also include Microsoft 365 Defender hunting queries for advanced hunting scenarios in both Microsoft 365 Defender and Microsoft Sentinel. You can also submit to issues for any samples or resources you would like to see here as you onboard to Microsoft Sentinel. This repository welcomes contributions and refer to this repository's wiki to get started. For questions and feedback, please contact [email protected]

Resources

• Microsoft Sentinel documentation
• Microsoft 365 Defender documentation
• Security Community Webinars
• Getting started with GitHub

We value your feedback. Here are some channels to help surface your questions or feedback:

1. General product specific Q&A for SIEM and SOAR - Join in the Microsoft Sentinel Tech Community conversations
2. General product specific Q&A for XDR - Join in the Microsoft 365 Defender Tech Community conversations
3. Product specific feature requests - Upvote or post new on Microsoft Sentinel feedback forums
4. Report product or contribution bugs - File a GitHub Issue using Bug template
5. General feedback on community and contribution process - File a GitHub Issue using Feature Request template

Contribution guidelines

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

Add in your new or updated contributions to GitHub

Note: If you are a first time contributor to this repository, General GitHub Fork the repo guidance before cloning or Specific steps for the Sentinel repo.

General Steps

Brand new or update to a contribution via these methods:

• Submit for review directly on GitHub website
  • Browse to the folder you want to upload your file to
  • Choose Upload Files and browse to your file.
  • You will be required to create your own branch and then submit the Pull Request for review.
• Use GitHub Desktop or Visual Studio or VSCode
  • Fork the repo
  • Clone the repo
  • Create your own branch
  • Do your additions/updates in GitHub Desktop
  • Be sure to merge master back to your branch before you push.
  • Push your changes to GitHub

Pull Request

• After you push your changes, you will need to submit the Pull Request (PR)
• Details about the Proposed Changes are required, be sure to include a minimal level of detail so a review can clearly understand the reason for the change and what he change is related to in the code.
• After submission, check the Pull Request for comments
• Make changes as suggested and update your branch or explain why no change is needed. Resolve the comment when done.

Pull Request Detection Template Structure Validation Check

As part of the PR checks we run a structure validation to make sure all required parts of the YAML structure are included. For Detections, there is a new section that must be included. See the contribution guidelines for more information. If this section or any other required section is not included, then a validation error will occur similar to the below. The example is specifically if the YAML is missing the entityMappings section:

A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.95]     Kqlvalidations.Tests.DetectionTemplateStructureValidationTests.Validate_DetectionTemplates_HaveValidTemplateStructure(detectionsYamlFileName: "ExcessiveBlockedTrafficGeneratedbyUser.yaml") [FAIL]
  X Kqlvalidations.Tests.DetectionTemplateStructureValidationTests.Validate_DetectionTemplates_HaveValidTemplateStructure(detectionsYamlFileName: "ExcessiveBlockedTrafficGeneratedbyUser.yaml") [104ms]
  Error Message:
   Expected object to be <null>, but found System.ComponentModel.DataAnnotations.ValidationException with message "An old mapping for entity 'AccountCustomEntity' does not have a matching new mapping entry."

Pull Request KQL Validation Check

As part of the PR checks we run a syntax validation of the KQL queries defined in the template. If this check fails go to Azure Pipeline (by pressing on the errors link on the checks tab in your PR) In the pipeline you can see which test failed and what is the cause:

Example error message:

A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:01.81]     Kqlvalidations.Tests.KqlValidationTests.Validate_DetectionQueries_HaveValidKql(detectionsYamlFileName: "ExcessiveBlockedTrafficGeneratedbyUser.yaml") [FAIL]
  X Kqlvalidations.Tests.KqlValidationTests.Validate_DetectionQueries_HaveValidKql(detectionsYamlFileName: "ExcessiveBlockedTrafficGeneratedbyUser.yaml") [21ms]
  Error Message:
   Template Id:fa0ab69c-7124-4f62-acdd-61017cf6ce89 is not valid Errors:The name 'SymantecEndpointProtection' does not refer to any known table, tabular variable or function., Code: 'KS204', Severity: 'Error', Location: '67..93',The name 'SymantecEndpointProtection' does not refer to any known table, tabular variable or function., Code: 'KS204', Severity: 'Error', Location: '289..315'

If you are using custom logs table (a table which is not defined on all workspaces by default) you should verify your table schema is defined in json file in the folder Azure-Sentinel\.script\tests\KqlvalidationsTests\CustomTables

Example for table tablexyz.json

{
  "Name": "tablexyz",
  "Properties": [
    {
      "Name": "SomeDateTimeColumn",
      "Type": "DateTime"
    },
    {
      "Name": "SomeStringColumn",
      "Type": "String"
    },
    {
      "Name": "SomeDynamicColumn",
      "Type": "Dynamic"
    }
  ]
}

Run KQL Validation Locally

In order to run the KQL validation before submitting Pull Request in you local machine:

• You need to have .Net Core 3.1 SDK installed How to download .Net (Supports all platforms)
• Open Shell and navigate to Azure-Sentinel\\.script\tests\KqlvalidationsTests\
• Execute dotnet test

Example of output (in Ubuntu):

Welcome to .NET Core 3.1!
---------------------
SDK Version: 3.1.403

Telemetry
---------
The .NET Core tools collect usage data in order to help us improve your experience. The data is anonymous. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.

Read more about .NET Core CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry

----------------
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Find out what's new: https://aka.ms/dotnet-whats-new
Learn about the installed HTTPS developer cert: https://aka.ms/aspnet-core-https
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli-docs
Write your first app: https://aka.ms/first-net-core-app
--------------------------------------------------------------------------------------
Test run for /mnt/c/git/Azure-Sentinel/.script/tests/KqlvalidationsTests/bin/Debug/netcoreapp3.1/Kqlvalidations.Tests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.7.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 171
     Passed: 171
 Total time: 25.7973 Seconds

Detection schema validation tests

Similarly to KQL Validation, there is an automatic validation of the schema of a detection. The schema validation includes the detection's frequency and period, the detection's trigger type and threshold, validity of connectors Ids (valid connectors Ids list), etc. A wrong format or missing attributes will result with an informative check failure, which should guide you through the resolution of the issue, but make sure to look into the format of already approved detection.

Run Detection Schema Validation Locally

In order to run the KQL validation before submitting Pull Request in you local machine:

• You need to have .Net Core 3.1 SDK installed How to download .Net (Supports all platforms)
• Open Shell and navigate to Azure-Sentinel\\.script\tests\DetectionTemplateSchemaValidation\
• Execute dotnet test

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

For information on what you can contribute and further details, refer to the "get started" section on the project's wiki.

azure-sentinel's Issues

Azure Information Protection workbook queries broken

It looks like the table for the Azure Information Protection workbook changed from "InformationProtectionEvents to a custom log source called "InformationProtectionLogs_CL". The workbook queries the old table name still so all the queries are broken. No biggie, but hopefully the workbook can get updated soon.

Playbooks cannot be saved. The request content is not valid and could not be deserialized: 'Could not find member 'defaultValue' on object of type 'FlowTemplateParameter'.

Logic Apps cannot be created from the json files available in the Playbooks folder.

To Reproduce

1. Go to 'https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Playbooks/Block-AADUser.json' and copy the content.
2. Paste into a new logic app.
3. Click save.
4. See error:
   Failed to save logic app 'APPNAMEHERE'. The request content is not valid and could not be deserialized: 'Could not find member 'defaultValue' on object of type 'FlowTemplateParameter'. Path 'properties.parameters.PlaybookName.defaultValue', line 1, position 968.'.

Expected behavior
Successful save.

Desktop (please complete the following information):

• OS: Win10
• Browser: Chrome
• Version: Up to date.

Windows Firewall Logging

Been through all the issue reports and guides, had the Windows Firewall Logging working until this morning, but it died. I've been through removing the data connector and putting it back in, reset the MMA agents, and so on.... has something else changed here?

Issue injesting data from Cisco FTD

Describe the bug
When sending syslog messages from Cisco FTD they are now collected correctly in Sentinel. This causes the data not to be translated correctly, so events (incidents + playbooks) don't work correctly

Additional context
https://techcommunity.microsoft.com/t5/Azure-Sentinel/Azure-Sentinel-The-Syslog-and-CEF-source-configuration-grand/ba-p/803891

It appears that CEF will structure the data correctly, but wanted to confirm if there is a resolution to importing the syslog messages so they can be used in Sentinel?

Thank you

Playbook - Isolate an Azure VM

Requirements

• Create an Azure Logic Apps playbook for Azure Sentinel that includes the following workflow:
  • Triggers on an Azure Sentinel alert.
  • If the alert has host(s), isolate the respective machine(s)/host(s)
  • Update the incident associated with the alert with the isolated VMs and outcome of each action.
• Validate the playbook works as expected per the functional requirement mentioned above.
• Submit the playbook as a GitHub Pull Request per the playbook contribution guidance.
• Attach a screenshot of the working playbook with your submission.
• Provide a readme with the steps with your submission to document any prerequisite steps required before running this playbook.

The first submission that meets the requirements gets the reward. Rewards are subject to Microsoft terms and conditions.

Hint: Feel free to leverage HTTP Connector for Azure Logic Apps as needed.

Helpful resources

• Azure Sentinel Playbook documentation
• Playbook contribution guidance
• Create Azure Logic Apps Playbook

Events not populating in Incident/Alert

Good Day

My alarms are appearing in this manner this morning, is there anything i can do to fix it?

CEF connector dropping Cisco ASA message ID

I noticed that the CEF connector pulls in Cisco ASA logs to the Cisco ASA connector, but the OMS Agent is dropping the message ID at the front of the log. The workspace in Sentinel uses these Message IDs to parse the logs. For example:

When fed into the syslog connector the logs have the message ID:
%ASA-3-722035: Group <> User <> IP <> Received large packet ().

When fed into the Cisco ASA connector it gets dropped:
%ASA: end configuration: OK

If the workspace queries are built on parsing logs based on message ID, how can we get this ID into the Cisco ASA connector?

Alerts keep getting triggered even when outside the analysis period

I created an alert to be triggered when a specific event id (4728 - it indicates when a user is added to Domain Admins) is recorded in the SecurityEvent table. The frequency and the period were both set to 5 minutes. The alert suppression was also set for 5 min. The query was:

SecurityEvent
| where EventID == 4728
| project TimeGenerated, Computer, Admin=SubjectAccount,MemberName, TargetAccount, Activity
| extend AccountCustomEntity = TargetAccount
| extend HostCustomEntity = Computer

I added a user to Domain Admins and while the alert was triggered, it kept being sent every 5 minutes, even though there were no further events with id 4728. The alert simulation graph shows the correct data, one single event within the last couple of hours, however the alert (and the corresponding emails and SMS text) kept being raised every 5 minutes.

My expectation was to receive just one alert, triggered within the 5 minutes of the event 4728 recording.

Sentinel Detection Analytics

Describe the bug
There are many different detection methods that are generating a lot of alerts, even though they are being blocked by our firewall. Are these alerts supposed to be changed or just a "example" so we can build out own queries off them?
Example query is: Senntinel/Detections/CommonSecurityLog/HighConfTIDetection.txt

We can write new queries and change them, but just wanted to confirm the "best use case" of these queries included on GitHub?

Thank you
@shainw

[Feature Request] Support hex encoded conversion to ASCII

PROBLEM STATEMENT

Linux audit log is fed to Log Analytics workspace following standard format. The PROCTILE field records full command-line of the command that was used to invoke the process (with syscall normally). This field is encoded in hexadeximal notation.

Currently when doing analysis against this type of log, we normally need to convert hex to ASCII. There is not any built-in function to help analyst to convert from hex to ascii using Kusto Query Language.

SOLUTION PROPOSAL

There are the following options:

• Provide built-in function to support converting hex to ascii via KQL. A boilerplate code would be:

LinuxAuditLog_CL
| where RecordType == "PROCTITLE"
| extend decodedCmd == toascii(proctitle)

Function name would be fromHex() too.

• Provide a function in msticpy to help analyst as well as to group records by Audit ID.

# This command is used to extract set of events that match audit ID number 
and display decoded command line.
.\sectools\unhexlify.py --audit-id 123455

or

# This command is used to find a keyword from encoded command line
.\sectools\unhexlify.py --search '/usr/bin/'

WORKAROUND

A workaround would be to build a utility to invoke ausearch with --interpret and feed logs to Log Analytics workspace. However this way would lead to a lot of efforts, and would be counter to built-in agent's function.

Workbook – Monitor data using Azure Active Directory audit, sign in logs and Azure Activity logs

Requirements

• Create an Azure Monitor workbook for Azure Sentinel that includes the following:
  • Mash up of Azure Active Directory audit, sign in logs and Azure Activity logs data in Azure Sentinel
  • Workbook includes at least three charts / visuals that includes one trend chart
  • Workbook description should include a note on how the workbook demonstrates usefulness to an enterprise, for example by streamlining security operations, reducing threat detection and response times, or improving the effectiveness of existing security tools.
• Validate the workbook works as expected per the functional requirement mentioned above.
• Submit the workbook as a GitHub Pull Request per the workbook contribution guidance.

The first submission that meets the requirements gets the reward. Rewards are subject to Microsoft terms and conditions.

Helpful resources

• Azure Sentinel Workbook documentation
• Workbook contribution guidance
• Create Azure Monitor Workbooks

new_processes.txt Hunting Query fails

Describe the bug
New Processes Hunting Query fails to run when copied into Azure Sentinel/Log Analytics

To Reproduce

1. Copy Hunting Query
2. Paste into query window
3. Run query
4. See error

Expected behavior
Query should run without error. I was able to modify the query which appears ti return the proper results.

let ProcessCreationEvents=() {
let processEvents=SecurityEvent
| where EventID==4688
| where TimeGenerated >= ago(30d)
| project TimeGenerated, ComputerName=Computer,AccountName=SubjectUserName, AccountDomain=SubjectDomainName, FileName=tostring(split(NewProcessName, @'')[(-1)]), ProcessCommandLine = CommandLine, InitiatingProcessFileName=ParentProcessName,InitiatingProcessCommandLine="",InitiatingProcessParentFileName="";
processEvents };
ProcessCreationEvents
| where TimeGenerated < ago(1d)
| summarize HostCount=dcount(ComputerName) by tostring(FileName)
| join kind=rightanti (
ProcessCreationEvents
| where TimeGenerated >= ago(1d)
| summarize HostCount=dcount(ComputerName) by tostring(FileName)
) on FileName
| project HostCount, FileName

sysmon latest version does not work with the Sysmonparser

Describe the bug
eventID 3 is not being parsed via the parser. I have noticed that the current version of sysmon with schema version 4.22 does not work.

To Reproduce

search for sysmon event id 3 without using the parser
Event
| where Source contains "sysmon"
| where EventID == 3

the events shows up as expected, so we know the data is being captured by sysmon and send to azure sentinel workspace correctly.

doing the same search with the parser function

Expected behavior
eventID 3 data should show up

using the parser with other eventID's does show data ex below

Retrieve data from data warehouse

I am implementing a azure playbook(using logic apps) for incident response. For that, i need to retrieve data from internal data warehouse at some-point of playbook work flow.
For an instance, retrieve info such as asset owner, hostname, regards to specific IP from internal data warehouse
any-idea on how to connect data-warehouse and query info from there to achieve this in logic apps ?

Get-MDATPInvestigationPackage - issue with retrieving package download URI due to POST instead of GET

Hello,
I am having issues with running the Get-investigation package download URI step in the Get-MDATPInvestigationPackage playbook and I am pretty sure that the issue is related to that the logic app step submits a POST instead of a GET, i was able to reproduce the issue in the MDATP API explorer.

The following error appears:
{
"error": {
"code": "NotFound",
"message": "No HTTP resource was found that matches the request URI 'https://api.securitycenter.windows.com/api/machineactions/ea098113-c6ed-44ed-bb04-f1e3f160eff4/getPackageUri'.",
"target": "693228c8-f7d5-463a-8945-872d217253a2"
}
}

When running the same command with GET, the URL is correctly returned.

however when i change the logic app code to
"method": "get",
"path": "/api/machineactions/@{encodeURIComponent(body('Actions_-_Get_single_machine_action')?['id'])}/getPackageUri"

i get completely other errors.

This resource was not found, it may have been deleted.

Hello,

I've onboarded several of my clients to the Azure Sentinel preview as a POC. For any of these customers, when I go to one of my Azure Sentinel solution dashboards (Identity & Access, in this example), I get the following error when trying to click on the LogAnalytics button at the top of the visualization.

"This resource was not found, it may have been deleted. /subscriptions/mySub/resourceGroups/myRG/providers/microsoft.insights/components/workspaceName"

I've also noticed the workspace resource ID is incorrect - it's looking for it at ./providers/microsoft.insights/etc. but it's actually at /providers/microsoft.operationalInsights/etc.

Expected behaviour would be to bring me to the Log Analytics workspace with the specific query loaded.

Send logs to CommonSecurityLog with the HTTP Data Collector API ?

It's possible to send log data to Azure with the HTTP Data Collector API where Log-Type is the name of log which can be seen later under Custom Logs with _CL suffix.

But how to send log to CommonSecurityLog (seen under SecurityInsights) So it can be used in Sentinel Dashboards (without oms agent) ?

Nsg Resource_id display

Describe the bug
Hi i want to make one of my query result the Resource Id to automate the lonk to my Network Security Group resource. I have this output "/subscriptions//ressourceGroups//providers/Microsoft.Network/networkSecurityGroups/", that is the resource id definition of NSG. But it's not detected as resource id and it's not automatically create the link to the ressource. I have define properly the column settings but nothing change.

Br,

Nicolas

Screenshots
If applicable, add screenshots to help explain your problem.
The column Settings

My query Result

Desktop (please complete the following information):

• OS: [Mac]
• Browser [Mozilla]
• Version [ 69.0.3]

Cannot find "Validate the connector UX" - build your own connector

Hello,
I am trying to build my own data connector according to the guidelines mentioned in https://github.com/Azure/Azure-Sentinel/blob/master/DataConnectors/ReadMe.md#build-the-connector. In the 3rd step i.e. "Validate the Connector UX", I am supposed to upload my connector JSON file using the "Import" option, but I cannot find such option on my portal.

As mentioned in the document I tried to access the utility using the https://ms.portal.azure.com/?feature.BringYourOwnConnector=true but I was redirected to https://portal.azure.com/?feature.BringYourOwnConnector=true#home.
Can anyone let me know if this feature is still in beta version or do I need to update my subscription or any other issue?

Playbook – Create a list of discovered vulnerabilities pertaining to a machine/host

Requirements

• Create an Azure Logic Apps playbook for Azure Sentinel that includes the following workflow:
  • Triggers on an Azure Sentinel alert.
  • If the alert has hosts, find list of vulnerabilities pertaining to each host/machine. Include name, description, severity, CVE at a minimum.
  • Feed that into a document and post the document onto a SharePoint site.
  • Update the Azure Sentinel incident with the link to the vulnerability report.
• Validate the playbook works as expected per the functional requirement mentioned above.
• Submit the playbook as a GitHub Pull Request per the playbook contribution guidance.
• Attach a screenshot of the working playbook with your submission.
• Attach a screenshot of the final output with the vulnerability data.
• Provide a readme with the steps with your submission to document any prerequisite steps required before running this playbook.

The first submission that meets the requirements gets the reward. Rewards are subject to Microsoft terms and conditions.

Hints

• Connect with any product (API) of your choice that provides vulnerability list. Few examples are Qualys, Microsoft Defender ATP, etc.
• Feel free to leverage HTTP Connector for Azure Logic Apps as needed.

Helpful resources

• Azure Sentinel Playbook documentation
• Playbook contribution guidance
• Create Azure Logic Apps Playbook
• SharePoint Logic Apps connector

Windows Firewall Data Connector

Describe the bug
Windows Firewall Logs are "not" being sent through to Sentinel after Windows Firewall connector has been configured.

To Reproduce
Steps to reproduce the behavior:

1. Go to "Azure Sentinel"
2. Click on "Data Connectors"
3. Scroll down to 'Windows Firewall"

Expected behavior
Windows Firewall Events to be available under the log options:

1. Logs
2. Windows Firewall
3. Run the default command > expecting to see results

Please request additional content if required.

Include Alert Generating Platform Alert Link in Sentinel Alerts

Is your feature request related to a problem? Please describe.
I would like similar functionality to the Microsoft365 Security Portal in Sentinel where you can directly pivot into the appropriate security product portal from a specific alert. I have to manually navigate into for example the MDATP portal and find the corresponding alert to action on it.

Describe the solution you'd like
For a MDATP alert in Sentinel, include a link to the alert in Sentinel that directs you to appropriate the MDATP portal alert.

Describe alternatives you've considered
N/A

Additional context
N/A

execute_base64_decodedpayload.txt Query Fails

Describe the bug
Query fails to run.

To Reproduce

1. Copy query into Kusto
2. Run Query
3. Query fails to execute and syntax errors are shown

Expected behavior
Query should execute without syntax errors

Fix:
Modify Line 28
FileName=tostring(split(NewProcessName, '\')[-1], // convert SecurityEvents raw schema to get FileName & CommandLine

To:

FileName=tostring(split(NewProcessName, '\')[-1]), // convert SecurityEvents raw schema to get FileName & CommandLine

Issues with installing & running CEF collector on Linux machine

Describe the bug
Installation is semi-successful, in that many packages are installed, but receive several errors indicating "no such file" or "permission denied". Installation completes with "Omsagent restarted successfully" and "Installation completed", and returns a successful return code of 0 (zero).

To Reproduce
Steps to reproduce the behavior:

1. execute installation command:
   # wget https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_installer.py&&sudo python cef_installer.py <workspace id> <secret key>

2. errors produced
   

3. yet those directories are created with appropriate ownership and permissions:
   

4. services do not start successfully following install
   

Expected behavior
Log agent installed successfully & sending syslog messages to Azure

Additional context
Red Hat Enterprise Linux Server release 7.7
have applied CIS benchmark: CIS_Red_Hat_Enterprise_Linux_7_v2.1.0 (level 1)

*See attached text file containing full installation output and errors
CEF Installer errors.txt

Connecting ITSM with Sentinel Incidents

We are standardizing on Azure Monitor Alerts that call ITSM workflows. Is there a way to automatically create Azure Monitor Alerts from Sentinel Incidents? That is, are Sentinel Incidents exposed to Log Analytics? If that is not the case, I am seeing that Playbooks are the Sentinel solution to export Incident data to ITSM workflows, is that correct? What if we don't use one of the listed ITSM products at https://github.com/Azure/Azure-Sentinel/tree/master/Playbooks (JIRA, SNOW, or ZenDesk?)...(we use AutoTask) what development track should we take to create our own playbook, that is, one that creates AutoTask tickets when Sentinel Incidents are created. (FYI if I can surface Incidents in Log Analytics I already have an AutoTask connection solution there.) Thanks, John

DNS lookup requested

Describe the bug
DNS logs are being received into Sentinel although DNS lookup requests from malicious sources are not being sent.

Expected behavior
All DNS logs to be uploaded to Sentinel for consolidation

Additional context
Please let us know if you require any further information

Office 365 Connector Fails

Describe the bug
Can't connect our Office 365 tenant to Azure Sentinel

To Reproduce
Steps to reproduce the behavior:

1. Go to "Azure Sentinel - Data collection"
2. Click "Configure" under "Office 365"
3. Ensure solution is installed.
4. Select "Connect tenants to Azure Sentinel" then "Add tenant"
5. Enter auth info into popup window. (MFA and Conditional Access is enabled for our admin accounts)
6. Error: "Consent process failed - internal error. Try again later."

Expected behavior
Tenant appears in Azure Sentinel, etc.

Desktop (please complete the following information):

• OS: Windows 10 Enterprise
• Browser: Chrome and Edge

IpRep and GeoIP Playbooks giving null value error

After importing both Get-IPReputation and Get-GeoFromIpAndTagIncident from the git and running against an incident both playbooks report the following error at the "For each 2" loop:

ExpressionEvaluationFailed. The execution of template action 'For_each_2' failed: the result of the evaluation of 'foreach' expression '@Body('Alert_-_Get_incidents')?['value']' is of type 'Null'. The result must be a valid array.

screen shot from Get-IPReputation

screen shot from Get-GeoFromIpAndTagIncident

Get-IPReputation contains JSON syntax error

Describe the bug

Get-IPReputation.json files contains a JSON syntax error at line 294

Azure Sentinel Suppression alert issue

Good Day

I am having an issue where a certain user's MFA has been disabled, and it should be that way for a few days. I have the analytics rule alert enabled but it keeps querying and opening a new alert. (I want it to query for at least 1 hour) . I have set the suppression to 5 hours after an alert is generated, although when testing no new alarms are coming through, e.g. another user disabling MFA.

Is there anyway to set the suppression for only one specific alert?

Azure-Sentinel .yaml format description?

Is your feature request related to a problem? Please describe.

We are maintaining MISP and we have seen people sharing specific rule-sets like Sigma. Is the format .yaml format in Azure-Sentinel related to Sigma? or is it a specific format? If yes, do you have a schema description available. We would like to make a specific parser to extract the data and feed it into MISP format.

Analytics query never triggers

The example analytics Kusto query doesn't ever seem to fire.

AzureDiagnostics | where Type != 'AzureMetric' and OperationName == 'NetworkSecurityGroupCounters' and type_s == 'block' and direction_s == 'In' and Resource == 'WEBTRAFFICONLY'

I don't think this query is doing what we want in the HOL. We need to check for any results where a rule counter is triggers higher than usual. The above query will always return the same number of results regardless of scanning activity.

Example query that does work:

AzureDiagnostics | where Type != 'AzureMetric' and OperationName == 'NetworkSecurityGroupCounters' and type_s == 'block' and direction_s == 'In' and Resource == 'WEBTRAFFICONLY' and matchedConnections_d > 500

With a threshold of 2 or greater results, this seems to work.

Disabled Account Detection won't fire

Describe the bug
For this detection: https://github.com/Azure/Azure-Sentinel/blob/master/Detections/SigninLogs/DisabledAccountSigninsAcrossManyApplications.yaml

I think line 31 should be
| where count_ >= threshold

Screenshots
No results:

Success with modified code:

Setting up a new custom connector for security solution

I am developing a new connector on top of a built-in connector(Common Event Format(CEF)) just like Checkpoint,CISCO ASA, for which a security solution is yet to develop.I am not getting proper guidelines/documentation on how to add setup/add a new vendor (as a security solution) to configure with my custom connector(in-built CEF connector in my case).We can create workbook and PR it but if we want to create new data connector for new security solution which has CEF format data how can we create data connector for that?

cef_troubleshoot.py error with Python3

Describe the bug
In the cef_troubleshoot.py we saw errors with variable 'line' from line 277. Result of tcp_dump.stdout.readline() is a byte type, not a str. Cannot compare with str "CEF" or "ASA" later in script. Was able to modify my copy of the script to add:
line = line.decode("utf-8")
this fixed the error.

Steps to reproduce the behavior:

1. New VM with Fedora, RHEL, or CentOS
2. Update Python.
3. Install cef_installer.py
4. Run cef_troubleshoot.py
5. errors:
   Validating CEF\ASA into rsyslog daemon - port 514
   This will take 60 seconds.
   sudo tcpdump -A -ni any port 514 -vv
   Traceback (most recent call last):
   File "cef_troubleshoot2.py", line 633, in
   main()
   File "cef_troubleshoot2.py", line 619, in main
   handle_rsyslog(workspace_id)
   File "cef_troubleshoot2.py", line 589, in handle_rsyslog
   "Received CEF\ASA message in daemon incoming port.[" + daemon_port + "]", mock_message=False)
   File "cef_troubleshoot2.py", line 279, in incoming_logs_validations
   if handle_tcpdump_line(line, incoming_port, ok_message):
   File "cef_troubleshoot2.py", line 238, in handle_tcpdump_line
   if "CEF" in line or "ASA" in line:
   TypeError: a bytes-like object is required, not 'str'

• OS: Fedora 31

Adding new field to sentinel record

Check CONTRIBUTING guideline first and here is the list to help us investigate the problem.

Describe the bug
I am trying to add new field to Fluentd record,with two filter sections for the same source section, but the new record doesn't seem to be added.

To Reproduce

Expected behavior

Your Environment

• Fluentd0.12.40

• NAME="CentOS Linux"
  VERSION="7 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="7"

• Kernel version: 3.10.0-1062.el7.x86_64

type syslog port 25226 bind 127.0.0.1 protocol_type udp tag oms.syslog

<filter oms.syslog.**>
type record_transformer

myownfield "ABCDE"

<filter oms.syslog.**>
type filter_syslog

No related errors.

Playbook – Add IP to Exchange blocklist

Requirements

• Create an Azure Logic Apps playbook for Azure Sentinel that includes the following workflow:
  • Triggers on an Azure Sentinel alert.
  • If the alert has an IP, add the IP to the block list
• Validate the playbook works as expected per the functional requirement mentioned above.
• Submit the playbook as a GitHub Pull Request per the playbook contribution guidance.

The first submission that meets the requirements gets the reward. Rewards are subject to Microsoft terms and conditions.

Hint: Check out Azure Automation connector for Azure Logic Apps.

Helpful resources

• Azure Sentinel Playbook documentation
• Playbook contribution guidance
• Create Azure Logic Apps Playbook
• Adding IP to Exchange blocklist
• Authoring Azure Automation runbooks

Workbook – Monitor data using any firewall logs with other data logs of your choice

Requirements

• Create an Azure Monitor workbook for Azure Sentinel that includes the following:
  • Mash up of Windows Firewall data in Azure Sentinel and at least 3 data types
  • Workbook includes at least four charts / visuals that includes one trend chart
  • Correlate entities across the different data types
  • Workbook description should include a note on how the workbook demonstrates usefulness to an enterprise, for example by streamlining security operations, reducing threat detection and response times, or improving the effectiveness of existing security tools.
• Validate the workbook works as expected per the functional requirement mentioned above.
• Submit the workbook as a GitHub Pull Request per the workbook contribution guidance.

The first submission that meets the requirements gets the reward. Rewards are subject to Microsoft terms and conditions.

Helpful resources

• Azure Sentinel Workbook documentation
• Workbook contribution guidance
• Create Azure Monitor Workbooks

Security Center Connector permission error

I am having issues connecting Azure Security Center to the Sentinel solution. I have the correct permission Security Reader set on all my organisations subscriptions, My account has Global Administrator set in Azure AD and Security Admin and Reader set for Azure. In the attached image the only subscription I have permission to add the integration to is the one I am a owner of.

Unable to install CEF collector on Linux machine using web proxy

Describe the bug
When installing the Linux Syslog agent to collect Common Event Format (CEF) Syslog messages, the install errors out when the system using a web proxy (such as Squid) to forward requests to the internet.

To Reproduce
Steps to reproduce the behavior:

1. Login to Linux machine to install the CEF collector
2. Configure proxy settings

cat << EOF > /etc/profile.d/proxy.sh
export http_proxy=http://myproxy:3128/
export https_proxy=https://myproxy:3128/
export no_proxy="127.0.0.1, localhost"
EOF

2. Re-login to system to ensure proxy settings persist
3. Run the following command to install and apply the CEF collector

sudo wget https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_installer.py&&sudo python cef_installer.py <hash value>

4. See error

--2020-03-16 14:50:45--  https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_installer.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... failed: Name or service not known.
wget: unable to resolve host address ‘raw.githubusercontent.com’

Expected behavior
Log agent installed successfully

Server Info
Red Hat Enterprise Linux 7.7

Combining two related incidents into one

We are trying to combine a correlated incidents from different connectors (WDATP And Azure ATP) into one incident with an interval of lets say 30 mins.

Is it possible to do so and what would be the criteria to get such related incidents?

Thank you

Typo in Palo Alto Threat dashboard

Describe the bug
There is a typo in the "Threat subtypes sver sime" panel on the Palo Alto Threat dashboard.

To Reproduce
Steps to reproduce the behavior:

1. View the Palo Alto Threat dashboard
2. See typo

Azure Sentinel Workbook Displays Old Data Maybe Due To Caching

I have created a KQL and using that plotted a Tile view in the workbook. The data in the Tile is supposed to change based on the Timerange filter I have added. This is working fine unless new events are received. So let's say I have selected "Last 24 hours" at 12 PM and the count in tile is "2". Now I keep the session idle for a few minutes and at 12:10 PM, a new event arrives. Thereafter when I change the Timerange filter to "Last 5 minutes", I see the new event, but when I again change the Timerange filter to "Last 24 hours", the count still will be "2" instead of "3".

However, when I refresh the entire web page, the count will be updated. Is there any way to overcome this? I think the results are being cached by Azure and reused, but is there a way to disable this?

security log trough WEF

forward security event will not include the whole fields that are originally withing the security event.
regular security event log fields:

TenantId	ff632c0c-8a76-4ec2-a668-e155d2babcd1
	TimeGenerated [UTC]
	SourceSystem
	Account
	AccountType
	Computer
	EventSourceName
	Channel
	Task
	Level
	EventID
	Activity
	IpAddress
	IpPort
	LogonGuid
	Process
	ProcessId
	ProcessName
	SubjectAccount
	SubjectDomainName
	SubjectLogonId
	SubjectUserName
	SubjectUserSid
	TargetAccount
	TargetDomainName
	TargetInfo
	TargetLogonGuid
	TargetServerName
	TargetUserName
	SourceComputerId
	EventOriginId
	MG
	TimeCollected [UTC]
	ManagementGroupName
	Type

event that are coming from WEF:

AuthenticationPackageName	Negotiate
	FailureReason
	IpAddress
	IpPort
	KeyLength
	LmPackageName
	LogonProcessName
	LogonType
	ProcessId
	ProcessName
	Status
	SubStatus
	SubjectDomainName
	SubjectLogonId
	SubjectUserName
	SubjectUserSid
	TargetDomainName
	TargetUserName
	TargetUserSid
	TransmittedServices
	WorkstationName

pl

Incidents from Potential malicious events and Data source anomalies

We are envisioning managing Sentinel mainly from Incidents, rather than manually watching the Sentinel console in the Azure portal. We would like incidents auto-created for serious Potential malicious events (those of the "large orange dot" and higher, not the 'small orange dots' however). Also when there are significant spikes in any data source anomaly. How can we get visibility into these two Sentinel Overview page controls without watching the portal?
Thanks, John

Detection rule 'Failed login attempts to Azure Portal' incorrect IPCustomEntity

Describe the bug
The IPCustomEntity in the detection rule 'Failed login attempts to Azure Portal' is a list of IP adresses which causes incorrect view of entities in the Sentinel incident.

To Reproduce
Steps to reproduce the behavior:

1. Trigger a failed logon attempt incident
2. Compare the entities list to the query results

Expected behavior
I believe the query should result in one row per failed login, with one IP per row, instead of aggregating all failures to one row per account.

Screenshots

Desktop (please complete the following information):

• OS: N/A
• Browser N/A
• Version N/A

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Playbook – Snapshot an Azure VM

Requirements

• Create an Azure Logic Apps playbook for Azure Sentinel that includes the following workflow:
  • Triggers on an Azure Sentinel alert.
  • Get the incident associated with the alert.
  • Create a new snapshot of the Azure VM
  • Update the incident with the snapshot location.
• Validate the playbook works as expected per the functional requirement mentioned above.
• Submit the playbook as a GitHub Pull Request per the playbook contribution guidance.

The first submission that meets the requirements gets the reward. Rewards are subject to Microsoft terms and conditions.

Hint: HTTP connector for Azure Logic Apps might be helpful here
Helpful resources

• Azure Sentinel Playbook documentation
• Playbook contribution guidance
• Create Azure Logic Apps Playbook
• Create / Update Azure VM snapshots

cef_troubleshoot.py error

https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_troubleshoot.py exiting with errors on Ubuntu 18.04.3 LTS with Russian locale

Traceback (most recent call last):
File "cef_troubleshoot.py", line 632, in
main()
File "cef_troubleshoot.py", line 610, in main
security_config_omsagent_test(workspace_id=workspace_id)
File "cef_troubleshoot.py", line 451, in security_config_omsagent_test
is_security_config_omsagent_dir_exists = check_file_in_directory("security_events.conf", path)
File "cef_troubleshoot.py", line 309, in check_file_in_directory
output = o.decode('ascii')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd1 in position 35: ordinal not in range(128)

Problem is in 299 line
def check_file_in_directory(file_name, path):
'''
Check if the given file is found in the current directory.
:param path:
:param file_name:
:return: return True if it is found elsewhere False
'''
current_dir = subprocess.Popen(["ls", "-ltrh", path], stdout=subprocess.PIPE)
grep = subprocess.Popen(["grep", "-i", file_name], stdin=current_dir.stdout, stdout=subprocess.PIPE)
o, e = grep.communicate()
output = o.decode('ascii')
if e is None and file_name in output:
return True
return False

if we change output = o.decode('ascii') to output = o.decode(' 'utf8'') - everething is fine

Some went wrong deploying CEF-VMSS

Describe the bug
I clicked the deploy to Azure button at the page described below and the deployment wizard deployed a website...

To Reproduce
Steps to reproduce the behavior:

1. Go to 'https://github.com/Azure/Azure-Sentinel/tree/master/DataConnectors/CEF-VMSS#deploy-unbuntu-vmss', (Deploy Unbuntu VMSS)
2. Click on 'Deploy to Azure'
3. Page 1 will appear, accept whatever default or change what you like.
4. Page 2 appears to show what it will deploy:
   
5. Only Website???
6. After deployment:
   

Expected behavior
It should deploy other resources than this, like as described in
https://github.com/Azure/Azure-Sentinel/tree/master/DataConnectors/CEF-VMSS

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: macOs Mojave
• Browser safari
• Version 13.0

[Question] Detection Mitre ATT&CK Technique Tagging

Currently the detections are mapped to Mitre ATT&CK Tactics, would you be open to adding technique tagging? Either based on technique id for example with the detection SecurityEvent/SecurityEventLogCleared.txt:

// Techniques: #T1070

or by name:

// Techniques: #Indicator Removal on Host

I'm happy to go through and add them to the current detections and make a pull request.