Azure Event Hub Error Investigation Guide¶

A practical, systematic approach for diagnosing Event Hub issues in production — from initial triage through root cause.

1. Triage: Where Is the Error Coming From?¶

Before diving into logs, classify the error by where it's surfacing:

Start by pulling the exception type and Event Hubs error code from the client SDK — almost every failure maps to a specific EventHubsException reason code, and that code should drive the rest of your investigation rather than guessing.

2. Core Diagnostic Surfaces¶

2.1 Azure Monitor Metrics (fastest first look)¶

In the Event Hubs namespace → Metrics blade, check these first:

• Incoming/Outgoing Requests vs Incoming/Outgoing Messages — divergence suggests batching or throttling
• Throttled Requests — non-zero means you're hitting Throughput Unit (TU) or Processing Unit (PU) limits
• Server Errors — internal service issues (rare, but check Azure Status if present)
• User Errors — client-caused (bad auth, malformed requests, invalid partition key, expired SAS)
• Captured Messages/Bytes (if Capture enabled) — confirms whether Capture is silently failing
• Active Connections — spikes/drops correlate with connection churn issues

Split metrics by Entity Name (per Event Hub) and by partition where possible — aggregate namespace metrics hide per-hub hot spots.

2.2 Diagnostic Logs (the real detail)¶

Enable and route these log categories to a Log Analytics workspace (or your ELK stack, given your Vector pipeline):

• ArchiveLogs — Capture-specific failures
• OperationalLogs — management-plane operations (entity create/update, auth changes)
• AutoScaleLogs — TU/PU autoscale decisions (useful for confirming scale wasn't the bottleneck)
• KafkaCoordinatorLogs / KafkaUserErrorLogs — if using the Kafka-compatible endpoint
• EventHubVNetConnectionEvent — private endpoint / VNet connectivity failures
• CustomerManagedKeyUserLogs — if using CMK encryption

KQL starting point:

AzureDiagnostics
| where ResourceProvider == "[ hostname ]"
| where TimeGenerated > ago(1h)
| project TimeGenerated, Category, OperationName, ResultType, ResultDescription, Level
| order by TimeGenerated desc

Filter to errors only:

AzureDiagnostics
| where ResourceProvider == "[ hostname ]"
| where Level == "Error" or ResultType != "Success"
| project TimeGenerated, Category, OperationName, ResultDescription, CorrelationId
| order by TimeGenerated desc

2.3 Application Insights (client-side correlation)¶

If your producers/consumers are instrumented (and given your ai-audit work, you likely have App Insights everywhere already), correlate:

exceptions
| where cloud_RoleName has "your-service-name"
| where type has "EventHubs"
| project timestamp, type, outerMessage, innerMessage, operation_Id
| order by timestamp desc

Join operation_Id back to requests/dependencies to see what the app was doing at the moment of failure — this is usually faster than reasoning from the SDK exception alone.

3. Common Error Codes and What They Actually Mean¶

4. Deep-Dive Playbooks by Symptom¶

4.1 "We're getting throttled" (ServerBusyException)¶

1. Check Throttled Requests metric — confirm it's actually throttling and not a downstream timeout masquerading as one.
2. Check Incoming/Outgoing Throughput against namespace TU/PU capacity: 1 TU ≈ 1 MB/s in or 1000 events/s in, 2 MB/s out.
3. Check if Auto-inflate is enabled and whether it hit maximumThroughputUnits.
4. Look for hot partitions — if a partition key is unevenly distributed, one partition can throttle while overall namespace usage looks fine.
5. If using Standard tier, consider whether Premium/Dedicated (PU-based, isolated capacity) is warranted for the workload.

4.2 "Consumer is lagging or stuck"¶

1. Confirm consumer group and partition ownership — are all partitions actually claimed, or is one orphaned?
2. Check the checkpoint store (Blob Storage container, typically) for stale checkpoints — compare lastModified timestamps per partition blob against current time.
3. Pull Outgoing Messages metric per partition — a partition with zero outgoing messages while others are flowing indicates a stuck consumer instance or an unbalanced EventProcessorClient load-balancing state.
4. Check for LeaseLostException in App Insights — frequent lease loss usually means too many consumer instances competing, or checkpoint store latency.
5. If message ordering/duplication issues appear downstream (e.g., in your Logstash/Vector pipeline), verify whether the consumer is checkpointing before or after processing — checkpoint-after-process is safer but can cause reprocessing on restart.

4.3 "Messages are silently disappearing"¶

1. Confirm there's no Capture misconfiguration diverting/duplicating data.
2. Check message retention — default is 1 or 7 days depending on tier; if a consumer was down longer than retention, data is genuinely gone (expected behavior, not a bug).
3. Verify producer isn't silently swallowing exceptions — check for try/catch blocks around SendAsync/EventHubProducerClient calls that log-and-continue without alerting.
4. Confirm downstream isn't filtering messages on ingest (e.g., a Logstash filter or Vector transform silently dropping malformed events — check DLQ if one exists).

4.4 "Auth failures" (401/403)¶

1. If using SAS tokens: check expiry and whether the Shared Access Policy has the right claim (Send, Listen, Manage).
2. If using Azure AD / RBAC (likely your setup given IAM work): confirm the identity has Azure Event Hubs Data Sender and/or Azure Event Hubs Data Receiver role assigned at the correct scope (namespace vs. individual Event Hub).
3. Check for role assignment propagation delay — RBAC changes can take a few minutes to propagate.
4. If using Managed Identity, confirm the identity is actually attached to the compute resource (easy to miss after a redeploy) — worth a quick az identity show / az role assignment list --assignee <principalId> check.

4.5 "Connectivity / VNet / Private Endpoint issues"¶

1. Check EventHubVNetConnectionEvent diagnostic log category.
2. Confirm DNS resolution — private endpoint FQDNs must resolve to the private IP, not the public one (a common issue after DNS zone misconfiguration).
3. Confirm NSG rules allow outbound on 5671/5672 (AMQP) or 443 (AMQP over WebSockets) as applicable.
4. If firewall rules restrict the namespace to specific VNets/IPs, confirm the client's egress IP matches an allowed rule.

5. CLI / PowerShell Quick Reference¶

# Check namespace throughput/capacity config
az eventhubs namespace show --name <ns> --resource-group <rg> \
  --query "{sku:sku, autoInflate:isAutoInflateEnabled, maxTU:maximumThroughputUnits}"

# List consumer groups on an Event Hub
az eventhubs eventhub consumer-group list \
  --namespace-name <ns> --eventhub-name <hub> --resource-group <rg>

# Check RBAC role assignments for a principal
az role assignment list --assignee <principalId> --scope <namespaceResourceId>

# Pull recent diagnostic settings config (confirm logs are actually being routed)
az monitor diagnostic-settings list --resource <namespaceResourceId>

6. Terraform / IaC Angle¶

Given your ILM/Terraform-heavy workflow, a few things worth checking in code before assuming it's a runtime issue:

• Confirm partition_count and message_retention haven't drifted from what the app expects (partition count is immutable post-creation on Standard tier — a mismatch usually means the app config is stale, not the infra).
• Confirm auto_inflate_enabled / maximum_throughput_units are actually applied (state drift here is common after manual portal changes).
• Confirm diagnostic settings resources are still attached — these get silently orphaned if the Log Analytics workspace ID changes and Terraform isn't re-applied.
• If RBAC was recently migrated to a dedicated module (as you've been doing for other resources), confirm Event Hub data-plane roles weren't dropped in the migration — this is a common regression source for 401s right after an RBAC refactor.

7. Escalation Checklist (if it's genuinely Azure-side)¶

Before opening a support ticket, gather:

• Namespace name, region, tier (Basic/Standard/Premium/Dedicated)
• Time window of the issue (UTC)
• Correlation ID from a failed operation (from diagnostic logs or SDK exception)
• Whether the issue is namespace-wide or scoped to specific Event Hub/partition/consumer group
• Relevant metric screenshots (Throttled Requests, Server Errors) for the time window

This turns a multi-day back-and-forth into a much faster resolution.

Quick Decision Tree¶

Error observed
├── Client SDK exception thrown?
│   ├── ServerBusy/Timeout → Section 4.1 (throttling)
│   ├── Unauthorized/403 → Section 4.4 (auth)
│   ├── LeaseLost/ReceiverDisconnected → Section 4.2 (consumer)
│   └── CommunicationException → Section 4.5 (network)
├── No exception, but data missing → Section 4.3
└── Portal alert only, no app-side symptom → Start at Section 2.1 (Metrics)