{"id":13777406,"url":"https://github.com/ashwin-patil/blue-teaming-with-kql","last_synced_at":"2025-03-26T02:30:47.150Z","repository":{"id":40299708,"uuid":"306551201","full_name":"ashwin-patil/blue-teaming-with-kql","owner":"ashwin-patil","description":"Repository with Sample KQL Query examples for Threat Hunting","archived":false,"fork":false,"pushed_at":"2022-09-01T15:20:55.000Z","size":22662,"stargazers_count":208,"open_issues_count":0,"forks_count":36,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-03-21T01:34:29.706Z","etag":null,"topics":["azure","azure-data-explorer","azure-sentinel","blueteaming","kql","loganalytics","security","siem","threat-hunting"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ashwin-patil.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-23T06:43:28.000Z","updated_at":"2025-03-05T00:50:52.000Z","dependencies_parsed_at":"2022-08-31T00:21:47.716Z","dependency_job_id":null,"html_url":"https://github.com/ashwin-patil/blue-teaming-with-kql","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwin-patil%2Fblue-teaming-with-kql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwin-patil%2Fblue-teaming-with-kql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwin-patil%2Fblue-teaming-with-kql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwin-patil%2Fblue-teaming-with-kql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ashwin-patil","download_url":"https://codeload.github.com/ashwin-patil/blue-teaming-with-kql/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245576522,"owners_count":20638123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","azure-data-explorer","azure-sentinel","blueteaming","kql","loganalytics","security","siem","threat-hunting"],"created_at":"2024-08-03T18:00:42.855Z","updated_at":"2025-03-26T02:30:45.746Z","avatar_url":"https://github.com/ashwin-patil.png","language":null,"funding_links":[],"categories":["Community"],"sub_categories":["Community Repositories"],"readme":"# blue-teaming-with-kql\nRepository with Sample KQL Query examples for Threat Hunting\n\nThis folder has various KQL examples related to Threat Hunting/Blue Teaming presented at [Blue Team Village at GrayHat 2020] (https://grayhat.co/event/blue-teaming-with-kusto-query-language-kql/) and a presentation given at [KQLCafe](https://kqlcafe.github.io/website/) Podcast- Aug 2022.\n\n## Presentation:\n\n[GrayHat-BlueTeamingwithKQL](https://github.com/ashwin-patil/blue-teaming-with-kql/blob/main/GrayHat-%20BlueTeamingwithKQL.pdf)\n\n[Blue Teaming with KQL – 2022 KQL|Café Edition](https://github.com/ashwin-patil/blue-teaming-with-kql/blob/main/KQLCafe-BlueTeamingwithKQL-2022.pdf)\n\n## GrayHat Talk Recorded Video\n[![IMAGE ALT TEXT HERE](./images/youtube_thumbnail.jpg)](https://www.youtube.com/watch?v=IMZkqTEBFeA)\n\n## KQLCafe-Aug2022 Recorded Video\n[PendingforPublishing]\n\n **Jupyter Notebook** : \n \n Click on nbviewer Badge :: [![nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.jupyter.org/github/microsoft/msticpy/blob/master/docs/notebooks/Data_Queries.ipynb)\n\n\n- [Structure of Basic KQL Query](#structure-of-basic-kql-query)\n- [KQL Basic Searches](#kql-basic-searches)\n- [Exploring Tables and Schemas](#explore-tables-schemas)\n- [Asset/Device Details](#asset-details)\n- [Query Parameterization](#query-parameterization)\n- [Dynamic DataTypes](#dynamic-datatypes)\n- [Datetime](#datetime)\n- [Regex Extraction](#regex-extraction)\n- [Functions](#functions)\n    - [User Defined](#user-defined)\n    - [Built-in Functions](#built-in-functions)\n- [Time Series Analysis](#time-series-analysis)\n- [Network Beaconing](#network-beaconing)\n- [KQL Programmatic Interfaces](#kql-programmatic-interfaces)\n    - [QueryProvider Demo](#queryprovider-demo)\n - [KQL Gallery](#kql-gallery)\n ## KQLCafe-2022\n- [Practical Detection Engineering/Hunting with KQL](#practical-detection-engineeringhunting-with-kql)\n  - [Simple aggregation and threshold-based query](#simple-aggregation-and-threshold-based-query)\n  - [Bringing context from other data sources](#bringing-context-from-other-data-sources)\n  - [Rare events – Not historically seen events](#rare-events-–-not-historically-seen-events)\n  - [Pivot- To create heatmap like data structure to identify hourly spikes](#pivot--to-create-heatmap-like-data-structure-to-identify-hourly-spikes)\n- [Extending KQL](#extending-kql)\n  - [Github Action for dynamic TI Feeds](#github-action-for-dynamic-ti-feeds)\n  - [ADX/LA Interoperability](#adxla-interoperability)\n## Structure of Basic KQL Query\n1. Variable Declaration\n2. Table Name\n3. Datetime Filtering\n4. Event Type Filtering\n5. Output Formatting/ Display selected Fields\n6. Limit Results\n\n![Image](./images/01-kql-query-structure.png)\n\n## KQL Basic Searches\n### Search for presence of keyword and output tables where it is present\n\n```python\n  search \"badaccount\" \n  | where TimeGenerated \u003e ago(4h) \n  | summarize count() by $tableName\n```\n\n### Search for IP in multiple tables - irrespective of field names\n```python\n  search \"8.8.8.8\" in (\"AzureNetworkAnalytics_CL\", \"CommonSecurityLog\") \n  | where TimeGenerated \u003e ago(1h)\n  | limit 100\n```\n\n### Sort by time\n```python\n  AzureActivity \n  | where TimeGenerated \u003e ago(1h) \n  | sort by TimeGenerated desc\n```\n\n### Filter by value\n```python\n  SecurityEvent\n  | where TimeGenerated \u003e ago(1h)\n  | where EventID == 4688\n  | limit 100\n```\n\n### Aggregation by Field name\n```python\n  OfficeActivity\n  | where TimeGenerated \u003e ago(1h)\n  | summarize count() by OperationName\n```\n\n## Exploring Tables and Schemas\n\n### DataTypes ingested along with the Sizes\n```python\n  Usage\n  | where TimeGenerated \u003e ago(1d)\n  | summarize DataSizeinMB = sum(Quantity) by DataType\n  | sort by DataSizeinMB desc \n```\n\n### Schema and datatypes for each field of Table\n```python\n  AzureActivity\n  | getschema \n```\n\n### Tables across Workspace Queries\n```python\n  union workspace('WorkSpace01').Heartbeat, workspace('WorkSpace02').Heartbeat\n  | where TimeGenerated \u003e ago(1d)\n  | where Computer == \"CH-UBNTVM\"\n  | limit 100\n```\n\n## Asset/Device Details\n### Asset Details\n```python\n  Heartbeat\n  | where ComputerIP == \"40.71.227.249\"\n  | summarize LastReported = max(TimeGenerated) by Computer, ComputerIP, RemoteIPCountry, \n  ComputerEnvironment, OSType, OSMajorVersion, OSMinorVersion, SubscriptionId, TenantId\n```\n\n### Microsoft 365 Defender - Device Information\n```python\n  DeviceInfo\n  | where DeviceName == \"contosohost\" and isnotempty(OSPlatform)\n  | project TenantId, DeviceName, PublicIP, IsAzureADJoined, OSPlatform, OSBuild, OSArchitecture, LoggedOnUsers\n```\n\n### Microsoft 365 Defender - Hostname based on Private IP addresses\n```python\n  DeviceNetworkInfo\n  | mv-expand IPAddresses\n  | extend IPAddress = tostring(parse_json(IPAddresses).IPAddress)\n  | where IPAddress== '10.0.0.100' \n  | project DeviceName, NetworkAdapterType, TunnelType, MacAddress\n```\n\n## Query Parameterization\n\n### Query Parameterization - Dynamic List  - in~ operator \n\n```python\n  let timeframe = 1d;\n  let EventNameList = dynamic([\"UpdateTrail\",\"DeleteTrail\",\"StopLogging\",\"DeleteFlowLogs\",\"DeleteEventBus\"]);\n  AWSCloudTrail\n  | where TimeGenerated \u003e ago(timeframe)\n  | where EventName in~ (EventNameList)\n  | limit 100\n```\n\n### Query Parameterization - Dynamic list - has_any operator\n\n```python\nExpensive Computes\n  let timeframe = 1d;\n  let tokens = dynamic([\"416\",\"208\",\"128\",\"120\",\"96\",\"80\",\"72\",\"64\",\"48\",\"44\",\"40\",\"g5\",\"gs5\",\"g4\",\"gs4\",\"nc12\",\"nc24\",\"nv12\"]);\n  let operationList = dynamic([\"Create or Update Virtual Machine\", \"Create Deployment\"]);\n  AzureActivity\n  | where TimeGenerated \u003e= ago(timeframe)\n  | where OperationName in (operationList)\n  | where ActivityStatus == \"Accepted\" \n  | where isnotempty(Properties)\n  | extend vmSize = tolower(tostring(parse_json(tostring(parse_json(tostring(parse_json(tostring\n  (parse_json(Properties).responseBody)).properties)).hardwareProfile)).vmSize))\n  | where isnotempty(vmSize)\n  | where vmSize has_any (tokens) \n  | limit 100\n```\n\n### Contains vs has demo\n```python\n  let CustomLogs = datatable(Username:string)\n  [\n      \"abcadmin123\",\n      \"admin123\",\n      \"admin\",\n      \"samadmin\"\n  ];\n  CustomLogs \n  //| where Username has \"admin\"\n  | where Username contains \"admin\"\n```\n\n## Dynamic DataTypes\n![](./images/dynamic-datatypes.PNG)\n\n\n## Datetime\n\n### Todatetime demo \n```python\n  let CustomLogs = datatable(TimeGenerated:string)\n  [\n      \"2020-10-23 01:00:00\",\n      \"2020-10-24 02:00:00\"\n  ];\n  CustomLogs \n  | extend TimeGenerated1 = todatetime(TimeGenerated)\n  | getschema\n```\n\n### Datetime conversion demo\n```python\n  let CustomLogs = datatable(TimeGenerated:string)\n  [\n      \"2020-10-23 01:00:00\",\n      \"2020-10-24 02:00:00\"\n  ];\n  CustomLogs \n  | extend TimeGenerated1 = todatetime(TimeGenerated)\n  | extend Day = format_datetime(TimeGenerated1, \"yyyy-MM-dd\")\n```\n\n## Regex Extraction\n### Matches regex demo\n\n```python\n  let PrivateIPregex = @'^127\\.|^10\\.|^172\\.1[6-9]\\.|^172\\.2[0-9]\\.|^172\\.3[0-1]\\.|^192\\.168\\.';\n  let endtime = 1d;\n  CommonSecurityLog\n  | where TimeGenerated \u003e= ago(endtime) \n  | where DeviceVendor =~ \"Cisco\"\n  | where DeviceAction =~ \"denied\"\n  | extend SourceIPType = iff(SourceIP matches regex PrivateIPregex,\"private\" ,\"public\" )\n  | where SourceIPType == \"public\"\n  | summarize count() by SourceIP\n  | join (\n    // Successful signins from IPs blocked by the firewall solution are suspect\n    // Include fully successful sign-ins, but also ones that failed only at MFA stage\n    // as that supposes the password was sucessfully guessed.\n      SigninLogs\n      | where ResultType in (\"0\", \"50074\", \"50076\") \n      ) on $left.SourceIP == $right.IPAddress\n  | limit 100\n```\n\n### Extract Key value pair from AdditionalExtension field in CommonSecurityLog\n\n```python\n  let CommonSecurityLog = datatable (DeviceVendor: string, AdditionalExtensions: string)\n  [\n  \"ZScaler\", \"country=United States;sourceAddress=10.10.10.10;sourcehostname=http://abc.ac.com;deviceTranslatedPort=60095;tunnelType=IPSEC;dnat=No;stateful=Yes;reason=Allow DNS;cs6label=threatname;destCountry=Italy;avgduration=143\",\n  \"Fortinet\", \"FortinetFortiGatelogid=1059028704;cat=utm:app-ctrl;FortinetFortiGatesubtype=app-ctrl;FortinetFortiGateeventtype=signature;FortinetFortiGatevd=root\",\n  \"Palo Alto Networks\", \"cat=general;PanOSDGl1=0;PanOSDGl2=0;PanOSDGl3=0;PanOSDGl4=0;PanOSVsysName=;PanOSActionFlags=0x0\"\n  ];\n  CommonSecurityLog\n  | extend AdditionalExtensions = extract_all(@\"(?P\u003ckey\u003e\\w+)=(?P\u003cvalue\u003e[a-zA-Z0-9-_:/@. ]+)\", dynamic([\"key\",\"value\"]), AdditionalExtensions)\n  | mv-apply AdditionalExtensions on (\n  summarize AdditionalExtensionsParsed = make_bag(pack(tostring(AdditionalExtensions[0]), AdditionalExtensions[1]))\n  )\n```\n\n## Functions\n\n### User Defined\nFunction Demo - GetAllAlertsOnHost\n\nSource - https://github.com/Azure/Azure-Sentinel/blob/master/Exploration%20Queries/InputEntity_Host/AlertsOnHost.txt\n\n```python\n  let GetAllAlertsOnHost = (suspiciousEventTime:datetime, v_Host:string){\n  //-3d and +6h as some alerts fire after accumulation of events\n  let v_StartTime = suspiciousEventTime-3d;\n  let v_EndTime = suspiciousEventTime+6h;\n  SecurityAlert\n  | where TimeGenerated between (v_StartTime .. v_EndTime)\n  // expand JSON properties\n  | extend Extprop = parsejson(ExtendedProperties)\n  | extend Computer = toupper(tostring(Extprop[\"Compromised Host\"]))\n  | where Computer contains v_Host\n  | project TimeGenerated, AlertName, Computer, ExtendedProperties\n  };\n  // change datetime value and hostname value below\n  GetAllAlertsOnHost(datetime('2020-10-23T00:00:00.000'), toupper(\"VICTIM00\"))\n```\n\n### Built-in Functions\n\n### Parse_path demo\n```python\n  let SecurityEvent = datatable (EventID: string, ​ShareLocalPath: string)\n  [\n  \"5145\",@\"\\\\shared\\users\\temp\\file.txt.gz\",\n  \"5145\",@\"\\\\shared\\users\\temp\\bad.exe\",\n  \"5145\",@\"\\\\shared\\users\\temp\\script.ps1\"\n  ];\n  SecurityEvent\n  | where EventID == 5145\n  | extend ShareLocalPathParsed = parse_path(ShareLocalPath)\n  | extend extension = tostring(parse_json(ShareLocalPathParsed).Extension),\n   FileName = tostring(parse_json(ShareLocalPathParsed).Filename), \n   DirName = tostring(parse_json(ShareLocalPathParsed).DirectoryName)\n```\n\n### ip4_is_match with lookup demo\n```python\n  let lookup = dynamic ([\"13.66.60.119/32\",\"13.66.143.220/30\",\"13.66.202.14/32\"]);\n  let AzureSubnetMatchedIPs=materialize(\n  CommonSecurityLog\n  | where TimeGenerated \u003e ago(4h)\n  | mv-apply l=lookup to typeof(string) on\n  (\n  where ipv4_is_match (DestinationIP, l)\n  )\n  | project-away l);\n  AzureSubnetMatchedIPs\n  | limit 100 \n```\n\n### Windows XML Parsing of Dynamic Field - EventData\n```python\n   Event\n   | where TimeGenerated \u003e ago(4h)\n   | extend EventData = parse_xml(EventData).DataItem.EventData.Data\n   | mv-expand bagexpansion=array EventData\n   | evaluate bag_unpack(EventData)\n   | extend Key=tostring(['@Name']), Value=['#text']\n   | evaluate pivot(Key, any(Value), TimeGenerated, EventLog, Computer, EventID)\n```\n\n## Externaldata Demo\n\nKQL Blog - [Using External data sources to enrich network logs using Azure storage and KQL](https://techcommunity.microsoft.com/t5/azure-sentinel/using-external-data-sources-to-enrich-network-logs-using-azure/ba-p/1450345)\n```python\n  let covidIndicators = (externaldata(TimeGenerated:datetime, FileHashValue:string, FileHashType: string )\n  [@\"https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Sample%20Data/Feeds/Microsoft.Covid19.Indicators.csv\"]\n  with (format=\"csv\"));\n  covidIndicators\n```\n\n### Externaldata - Azure IP ranges feed. \nLink is not static and gets expired as new content arrives\n\n```python\n  let AzureIPRangesPublicCloud = (externaldata(changeNumber:string, cloud:string, values: dynamic)\n  [@\"https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63/ServiceTags_Public_20201019.json\"] \n  with (format=\"multijson\"));\n  let AzureSubnetRangeAllowlist = AzureIPRangesPublicCloud \n  | mv-expand values \n  | extend addressPrefixes = parse_json(parse_json(values).properties).addressPrefixes; \n  AzureSubnetRangeAllowlist\n```\n\n## Time Series Analysis\n KQL Blog - [Time Series Analysis and  it`s applications in Security](https://techcommunity.microsoft.com/t5/azure-sentinel/looking-for-unknown-anomalies-what-is-normal-time-series/ba-p/555052)\n\nTime Series Analysis - Process Execution Anomaly\n```python\n  let starttime = 14d;\n  let endtime = 1d;\n  let timeframe = 1h;\n  let TotalEventsThreshold = 5;\n  let ExeList = dynamic([\"powershell.exe\",\"cmd.exe\",\"wmic.exe\",\"psexec.exe\",\"cacls.exe\",\"rundll.exe\"]);\n  let TimeSeriesData = \n  SecurityEvent\n  | where EventID == 4688 | extend Process = tolower(Process)\n  | where TimeGenerated between (startofday(ago(starttime))..startofday(ago(endtime)))\n  | where Process in (ExeList)\n  | project TimeGenerated, Computer, AccountType, Account, Process\n  | make-series Total=count() on TimeGenerated from ago(starttime) to ago(endtime) step timeframe by Process;\n  let TimeSeriesAlerts = TimeSeriesData\n  | extend (anomalies, score, baseline) = series_decompose_anomalies(Total, 1.5, -1, 'linefit')\n  | mv-expand Total to typeof(double), TimeGenerated to typeof(datetime), anomalies to typeof(double), score to typeof(double), baseline to typeof(long)\n  | where anomalies \u003e 0\n  | project Process, TimeGenerated, Total, baseline, anomalies, score\n  | where Total \u003e TotalEventsThreshold;\n  TimeSeriesAlerts\n  | join (\n  SecurityEvent\n  | where EventID == 4688 | extend Process = tolower(Process)\n  | summarize CommandlineCount = count() by bin(TimeGenerated, 1h), Process, CommandLine, Computer, Account\n  ) on Process, TimeGenerated \n  | project AnomalyHour = TimeGenerated, Computer, Account, Process, CommandLine, CommandlineCount, Total, baseline, anomalies, score \n  | extend timestamp = AnomalyHour, AccountCustomEntity = Account, HostCustomEntity = Computer\n```\n\n## Network Beaconing\nReference Work:\n- [Threat Hunting Project](https://github.com/ThreatHuntingProject/ThreatHunting/blob/master/hunts/beacon_detection_via_intra_request_time_deltas.md)\n\n- [Flare](http://www.austintaylor.io/detect/beaconing/intrusion/detection/system/command/control/flare/elastic/stack/2017/06/10/detect-beaconing-with-flare-elasticsearch-and-intrusion-detection-systems/) by [Austin Taylor](https://twitter.com/HuntOperator)\n\n KQL Blog - [Detect Network Beaconing via Intr-Request time delta patterns in Azure Sentinel](https://techcommunity.microsoft.com/t5/azure-sentinel/detect-network-beaconing-via-intra-request-time-delta-patterns/ba-p/779586)\n\n```python\n  let starttime = 2d;\n  let endtime = 1d;\n  let TimeDeltaThreshold = 10;\n  let TotalEventsThreshold = 15;\n  let PercentBeaconThreshold = 80;\n  let PrivateIPregex = @'^127\\.|^10\\.|^172\\.1[6-9]\\.|^172\\.2[0-9]\\.|^172\\.3[0-1]\\.|^192\\.168\\.';\n  let DestIPList = CommonSecurityLog\n  | where DeviceVendor == \"Palo Alto Networks\" and Activity == \"TRAFFIC\"\n  | where TimeGenerated between (ago(starttime)..ago(endtime))\n  | extend DestinationIPType = iff(DestinationIP matches regex PrivateIPregex,\"private\" ,\"public\" )\n  | where DestinationIPType == \"public\"\n  | summarize dcount(SourceIP) by DestinationIP\n  | where dcount_SourceIP \u003c 5\n  | distinct DestinationIP;\n  CommonSecurityLog\n  | where DeviceVendor == \"Palo Alto Networks\" and Activity == \"TRAFFIC\"\n  | where TimeGenerated between (ago(starttime)..ago(endtime))\n  | where DestinationIP in ((DestIPList))\n  | project TimeGenerated, DeviceName, SourceUserID, SourceIP, SourcePort, DestinationIP, DestinationPort, ReceivedBytes, SentBytes\n  | sort by SourceIP asc,TimeGenerated asc, DestinationIP asc, DestinationPort asc\n  | serialize\n  | extend nextTimeGenerated = next(TimeGenerated, 1), nextSourceIP = next(SourceIP, 1)\n  | extend TimeDeltainSeconds = datetime_diff('second',nextTimeGenerated,TimeGenerated)\n  | where SourceIP == nextSourceIP\n  //Whitelisting criteria/ threshold criteria\n  | where TimeDeltainSeconds \u003e TimeDeltaThreshold \n  | project TimeGenerated, TimeDeltainSeconds, DeviceName, SourceUserID, SourceIP, SourcePort, DestinationIP, DestinationPort, ReceivedBytes, SentBytes\n  | summarize count(), sum(ReceivedBytes), sum(SentBytes), make_list(TimeDeltainSeconds) \n  by TimeDeltainSeconds, bin(TimeGenerated, 1h), DeviceName, SourceUserID, SourceIP, DestinationIP, DestinationPort\n  | summarize (MostFrequentTimeDeltaCount, MostFrequentTimeDeltainSeconds) = arg_max(count_, TimeDeltainSeconds), TotalEvents=sum(count_), TotalSentBytes = sum(sum_SentBytes), TotalReceivedBytes = sum(sum_ReceivedBytes) \n  by bin(TimeGenerated, 1h), DeviceName, SourceUserID, SourceIP, DestinationIP, DestinationPort\n  | where TotalEvents \u003e TotalEventsThreshold \n  | extend BeaconPercent = MostFrequentTimeDeltaCount/toreal(TotalEvents) * 100\n  | where BeaconPercent \u003e PercentBeaconThreshold\n  | extend timestamp = TimeGenerated, IPCustomEntity = DestinationIP, AccountCustomEntity = SourceUserID, HostCustomEntity = DeviceName\n\n```\n\n## KQL Programmatic Interfaces\n- [msticpy]()\n    - [list of Data queries](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataQueries.html)\n    - List Pre-defined Queries :[list_queries()](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#listing-available-queries)\n    - Execute a Query \n        - [Running pre-defined query](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#running-an-pre-defined-query) \n        - Running Ad-Hoc Query: [exec_query()](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#running-an-ad-hoc-query)\n    - Import from query file : [import_query_file()](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#adding-a-new-set-of-queries-and-running-them)\n    - [Splitting Query Execution into Chunks](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#splitting-query-execution-into-chunks)\n\n### QueryProvider Demo\n![Demo](./images/nbdemo.gif)\n\n\n## KQL Gallery\n\n\u003e Curated list of KQL queries worth highlighting\n\n\nOne-stop KQL query for most regex (IP, URL, API Access tokens, Crypto Wallets) needs. \n\u003cbr\u003eRegex Credits - [bee-san/PyWhat Github](https://github.com/bee-san/pyWhat) \n\n```python\nlet pywhatregex = (externaldata(Name:string, Regex:string, plural_name: string, Description: string, Rarity:string, URL: string, Tags: dynamic )\n[@\"https://raw.githubusercontent.com/bee-san/pyWhat/main/pywhat/Data/regex.json\"] with (format=\"multijson\"));\npywhatregex\n```\nsubquery to select a specific regex\n```python\nlet pywhatregex = (externaldata(Name:string, Regex:string, plural_name: string, Description: string, Rarity:string, URL: string, Tags: dynamic )\n[@\"https://raw.githubusercontent.com/bee-san/pyWhat/main/pywhat/Data/regex.json\"] with (format=\"multijson\"));\nlet githubaccesstokenregex = pywhatregex | where Name == \"GitHub Access Token\" | project Regex;\ngithubaccesstokenregex\n```\n\n ## KQLCafe-2022\n## Practical Detection Engineering/Hunting with KQL\n\n\n### Simple aggregation and threshold-based query\n[Suspicious enumeration using Adfind tool](https://github.com/Azure/Azure-Sentinel/blob/master/Hunting%20Queries/SecurityEvent/Suspicious_enumeration_using_adfind.yaml)\n\n```python\nlet lookupwindow = 2m;\n  let threshold = 3; //number of commandlines in the set below\n  let DCADFSServersList = dynamic ([\"DCServer01\", \"DCServer02\", \"ADFSServer01\"]); // Enter a reference list of hostnames for your DC/ADFS servers\n  let tokens = dynamic([\"objectcategory\",\"domainlist\",\"dcmodes\",\"adinfo\",\"trustdmp\",\"computers_pwdnotreqd\",\"Domain Admins\", \"objectcategory=person\", \"objectcategory=computer\", \"objectcategory=*\"]);\n  SecurityEvent\n  //| where Computer in (DCADFSServersList) // Uncomment to limit it to your DC/ADFS servers list if specified above or any pattern in hostnames (startswith, matches regex, etc).\n  | where EventID == 4688\n  | where CommandLine has_any (tokens)\n  | where CommandLine matches regex \"(.*)\u003e(.*)\"\n  | summarize Commandlines = make_set(CommandLine), LastObserved=max(TimeGenerated) by bin(TimeGenerated, lookupwindow), Account, Computer, ParentProcessName, NewProcessName\n  | extend Count = array_length(Commandlines)\n  | where Count \u003e threshold\n```\n### Bringing context from other data sources\n[Privileged Accounts - Failed MFA](https://github.com/Azure/Azure-Sentinel/blob/master/Hunting%20Queries/MultipleDataSources/AADPrivilegedAccountsFailedMFA.yaml)\nPopulating Privileged accounts dynamically via IdentityInfo table.\n```python\nlet starttime = 2d;\n  let endtime = 1d;\n  let aadFunc = (tableName:string){\n  IdentityInfo\n  | where AssignedRoles contains \"Admin\"\n  | mv-expand AssignedRoles\n  | extend Roles = tostring(AssignedRoles), AccountUPN = tolower(AccountUPN)\n  | where Roles contains \"Admin\"\n  | distinct Roles, AccountUPN\n  | join kind=inner (\n    // Failed Signins attempts with reasoning related to MFA.\n    table(tableName)\n    | where TimeGenerated between(ago(starttime)..ago(endtime))\n    | where ResultDescription has_any (\"MFA\", \"second factor\", \"multi-factor\", \"second factor\") or ResultType in (50074, 50076, 50079, 50072, 53004, 500121)\n  ) on $left.AccountUPN == $right.UserPrincipalName\n  | extend timestamp = TimeGenerated, IPCustomEntity = IPAddress, AccountCustomEntity = UserPrincipalName\n  };\n  let aadSignin = aadFunc(\"SigninLogs\");\n  let aadNonInt = aadFunc(\"AADNonInteractiveUserSignInLogs\");\n  union isfuzzy=true aadSignin, aadNonInt\n```\n### Rare events – Not historically seen events\n[Palo Alto Threat signatures from Unusual IP addresses](https://github.com/Azure/Azure-Sentinel/blob/master/Solutions/PaloAlto-PAN-OS/Analytic%20Rules/PaloAlto-UnusualThreatSignatures.yaml)\n\n```python\nlet starttime = 7d;\n  let endtime = 1d;\n  let timeframe = 1h;\n  let HistThreshold = 25; \n  let CurrThreshold = 10; \n  let HistoricalThreats = CommonSecurityLog\n  | where isnotempty(SourceIP)\n  | where TimeGenerated between (startofday(ago(starttime))..startofday(ago(endtime)))\n  | where DeviceVendor =~ \"Palo Alto Networks\"\n  | where Activity =~ \"THREAT\" and SimplifiedDeviceAction =~ \"alert\" \n  | where DeviceEventClassID in ('spyware', 'scan', 'file', 'vulnerability', 'flood', 'packet', 'virus','wildfire', 'wildfire-virus')\n  | summarize TotalEvents = count(), ThreatTypes = make_set(DeviceEventClassID), DestinationIpList = make_set(DestinationIP), FirstSeen = min(TimeGenerated) , LastSeen = max(TimeGenerated) by SourceIP, DeviceAction, DeviceVendor;\n  let CurrentHourThreats =  CommonSecurityLog\n  | where isnotempty(SourceIP)\n  | where TimeGenerated \u003e ago(timeframe)\n  | where DeviceVendor =~ \"Palo Alto Networks\"\n  | where Activity =~ \"THREAT\" and SimplifiedDeviceAction =~ \"alert\" \n  | where DeviceEventClassID in ('spyware', 'scan', 'file', 'vulnerability', 'flood', 'packet', 'virus','wildfire', 'wildfire-virus')\n  | summarize TotalEvents = count(), ThreatTypes = make_set(DeviceEventClassID), DestinationIpList = make_set(DestinationIP), FirstSeen = min(TimeGenerated) , LastSeen = max(TimeGenerated) by SourceIP, DeviceAction, DeviceProduct, DeviceVendor;\n  CurrentHourThreats \n  | where TotalEvents \u003c CurrThreshold\n  | join kind = leftanti (HistoricalThreats \n  | where TotalEvents \u003e HistThreshold) on SourceIP\n```\n### Pivot- To create heatmap like data structure to identify hourly spikes\nUse `granny-asc` option with `project-reorder` to sort columns with numbers as name. \n\n```python\nlet end = now();\nlet start = end - 7d;\nSecurityEvent\n| where EventID == 4625\n| where TimeGenerated \u003e= startofday(start)\n| where TimeGenerated \u003c= startofday(end)\n| extend\n    HourOfLogin = toint(hourofday(TimeGenerated)),\n    DayNumberofWeek = dayofweek(TimeGenerated),\n    Date = format_datetime(TimeGenerated, \"yyyy-MM-dd\")\n| extend DayofWeek = case(DayNumberofWeek == \"00:00:00\", \"Sunday\", DayNumberofWeek == \"1.00:00:00\", \"Monday\", DayNumberofWeek == \"2.00:00:00\", \"Tuesday\", DayNumberofWeek == \"3.00:00:00\", \"Wednesday\", DayNumberofWeek == \"4.00:00:00\", \"Thursday\", DayNumberofWeek == \"5.00:00:00\", \"Friday\", DayNumberofWeek == \"6.00:00:00\", \"Saturday\", \"InvalidTimeStamp\")\n| evaluate pivot(HourOfLogin, count(), DayofWeek, Date)\n| project-reorder Date, DayofWeek, * granny-asc \n| sort by Date asc\n```\n\n\n## Extending KQL\n\n### Github Action for dynamic TI Feeds\n- Allows to connect external data sources. \n- Limited to static sites or blob storage data sources.\n\nUse case - Nord VPN API not accessible via `externaldata`\n\n[Threat Essentials - Signins from Nord VPN Providers](https://github.com/Azure/Azure-Sentinel/blob/master/Solutions/SecurityThreatEssentialSolution/Hunting%20Queries/Signins-from-NordVPN-Providers.yaml)\n\nGithub Actions:\n- Workflow: [Daily Nord VPN Servers Feed](https://github.com/microsoft/mstic/blob/master/.github/workflows/nordvpn-feed.yml) \n- Python Script : [get-nordvpnservers.py](https://github.com/microsoft/mstic/blob/master/.script/get-nordvpnservers.py)\n- Monitor Github Action: Github Action Workflow (https://github.com/microsoft/mstic/actions/workflows/nordvpn-feed.yml)\n\n### ADX/LA Interoperability\n- KQL has varying support in Azure Data Explorer (ADX) and Azure Log Analytics(LA)/Sentinel.\n- You can connect both products from each other and can run native KQL against it.\n- Connect additional data sources without duplicating data.\n- Use Kusto explorer client with rich features on LA data. \n- Extend support of missing KQL operators in LA/Sentinel.\n\nConnect ADX via LA: [Cross-resource query Azure Data Explorer by using Azure Monitor - Azure Monitor | Microsoft Docs](https://docs.microsoft.com/en-us/azure/azure-monitor/logs/azure-monitor-data-explorer-proxy#cross-query-your-log-analytics-or-application-insights-resources-and-azure-data-explorer)\n\nConnect LA via ADX: [Query data in Azure Monitor with Azure Data Explorer | Microsoft Docs](https://docs.microsoft.com/en-us/azure/data-explorer/query-monitor-data#add-a-log-analyticsapplication-insights-workspace-to-azure-data-explorer-client-tools)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashwin-patil%2Fblue-teaming-with-kql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fashwin-patil%2Fblue-teaming-with-kql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashwin-patil%2Fblue-teaming-with-kql/lists"}