User Email Submissions accuracy vs Admin review verdict

Hunt Hypothesis

User email submissions with high accuracy scores but no corresponding admin review verdict may indicate automated or staged submissions designed to bypass manual verification. SOC teams should proactively hunt for this behavior in Azure Sentinel to identify potential adversarial activity that evades standard review processes.

KQL Query

let ReviewResults = CloudAppEvents | where ActionType == "SubmissionNotification" 
| extend SubmissionId = tostring(parse_json(RawEventData).SubmissionId)
| extend Properties = parse_json(RawEventData.ExtendedProperties)
| mv-expand element = Properties
| where element.Name == "AdminReviewResult"
| project SubmissionId, AdminReviewResult = element.Value;
CloudAppEvents
| where ActionType == "UserSubmission"
| extend SubmissionId = tostring(parse_json(RawEventData).SubmissionId), SubmissionType = parse_json(RawEventData).SubmissionType
| join kind=leftouter ReviewResults on SubmissionId
| extend UserReportedAs=iif(SubmissionType == 1, "Phish",iif(SubmissionType == 2, "Junk",iif(SubmissionType == 3, "NotJunk","")))
| extend ReviewedAccuracy=iif(AdminReviewResult==UserReportedAs, "Correct", iif(AdminReviewResult=="Phish" and UserReportedAs == "Junk", "Phish reported as junk",iif(AdminReviewResult=="Junk" and UserReportedAs == "Phish","Junk reported as Phish",iif(AdminReviewResult=="NotJunk","Reported but not malicious or spam",iif(isempty(AdminReviewResult),"Not Reviewed","Not correct")))))
| extend Reviewed=iif(isempty(AdminReviewResult),"Not Reviewed","Reviewed")
| project SubmissionId,UserReportedAs,Reviewed,AdminReviewResult, ReviewedAccuracy
| where Reviewed=="Reviewed"
| summarize count() by ReviewedAccuracy
| render piechart

Analytic Rule Definition

id: 0d03314a-ecbe-47f0-b411-2a9471f26b46
name: User Email Submissions accuracy vs Admin review verdict
description: |
  This query visualises user submissions type compared to admin review verdict
description-detailed: |
  This query visualises user submissions type compared to admin review verdict, such as a User reporting a message as Phish but admin review is 'No threat found'
  Query is also included as part of the Defender for Office 365 solution in Sentinel: https://techcommunity.microsoft.com/blog/microsoftdefenderforoffice365blog/part-2-build-custom-email-security-reports-and-dashboards-with-workbooks-in-micr/4411303
requiredDataConnectors:
- connectorId: MicrosoftThreatProtection
  dataTypes:
  - CloudAppEvents
tactics:
  - InitialAccess
relevantTechniques:
  - T1566
query: |
  let ReviewResults = CloudAppEvents | where ActionType == "SubmissionNotification" 
  | extend SubmissionId = tostring(parse_json(RawEventData).SubmissionId)
  | extend Properties = parse_json(RawEventData.ExtendedProperties)
  | mv-expand element = Properties
  | where element.Name == "AdminReviewResult"
  | project SubmissionId, AdminReviewResult = element.Value;
  CloudAppEvents
  | where ActionType == "UserSubmission"
  | extend SubmissionId = tostring(parse_json(RawEventData).SubmissionId), SubmissionType = parse_json(RawEventData).SubmissionType
  | join kind=leftouter ReviewResults on SubmissionId
  | extend UserReportedAs=iif(SubmissionType == 1, "Phish",iif(SubmissionType == 2, "Junk",iif(SubmissionType == 3, "NotJunk","")))
  | extend ReviewedAccuracy=iif(AdminReviewResult==UserReportedAs, "Correct", iif(AdminReviewResult=="Phish" and UserReportedAs == "Junk", "Phish reported as junk",iif(AdminReviewResult=="Junk" and UserReportedAs == "Phish","Junk reported as Phish",iif(AdminReviewResult=="NotJunk","Reported but not malicious or spam",iif(isempty(AdminReviewResult),"Not Reviewed","Not correct")))))
  | extend Reviewed=iif(isempty(AdminReviewResult),"Not Reviewed","Reviewed")
  | project SubmissionId,UserReportedAs,Reviewed,AdminReviewResult, ReviewedAccuracy
  | where Reviewed=="Reviewed"
  | summarize count() by ReviewedAccuracy
  | render piechart
version: 1.0.0

Required Data Sources

MITRE ATT&CK Context

• Tactic: InitialAccess
• Technique: T1566 — T1566

References

• [Source Query](https://github.com/Azure/Azure-Sentinel/blob/main/Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Submissions/User Submission Accuracy versus Admin Verdicts.yaml)

False Positive Guidance

• Scenario: Scheduled Email Backup Job
  Description: A scheduled job (e.g., rsync, cron, or Veeam Backup) automatically submits email files to a central repository, which is flagged as a user submission.
  Filter/Exclusion: Exclude submissions from known backup tools or processes (e.g., backup_job_id = 'veeam_backup_123' or process_name = 'rsync').

• Scenario: Admin Review of User Submissions
  Description: An admin manually reviews user-submitted emails (e.g., via a ticketing system like Zendesk or ServiceNow) and updates the verdict, which may trigger the rule due to the review action.
  Filter/Exclusion: Exclude actions where the user is an admin (e.g., user_role = 'admin' or user_id IN ('admin_123', 'admin_456')).

• Scenario: Automated Email Archiving Tool
  Description: An email archiving tool (e.g., Microsoft Exchange Archiving, Mimecast, or Proofpoint) submits emails to a central archive, which is mistakenly flagged as a user submission.
  Filter/Exclusion: Exclude submissions from known archiving tools (e.g., tool_name = 'Mimecast' or source_system = 'Exchange_Archive').

• Scenario: Email Submission via API for Compliance
  Description: A compliance tool (e.g., Cisco SecureX, IBM QRadar, or Splunk) submits emails to a central system for compliance purposes, which is flagged as a user submission.
  Filter/Exclusion: Exclude submissions from known compliance tools (e.g., submission_type = 'compliance_api' or `tool_id = ‘splunk_compliance_78