Resolving PagerDuty Integration Failures and Routing Key Errors in Wazuh

Modern security operations rely on fast, automated alert delivery to reduce response times and minimize the impact of cyber threats.

Wazuh’s integration framework allows security events detected by the platform to be forwarded to third-party services, including PagerDuty, enabling security teams to receive real-time incident notifications without manually monitoring dashboards.

PagerDuty plays a critical role in incident management by automatically routing alerts to the appropriate responders based on escalation policies, schedules, and on-call rotations.

When the integration between Wazuh and PagerDuty works correctly, high-priority security events can immediately trigger notifications, ensuring incidents are acknowledged and investigated as quickly as possible.

However, one of the most common issues administrators encounter is the “webhook failed status code 400” error, often accompanied by problems involving an invalid or missing routing_key.

Since HTTP 400 indicates that PagerDuty rejected the request, the problem usually lies within the request payload, authentication configuration, or Events API formatting rather than network connectivity.

Typical symptoms include:

• Wazuh alerts never appearing in PagerDuty
• HTTP 400 responses in the Wazuh integrator logs
• PagerDuty rejecting Events API requests
• No incidents being created despite Wazuh detecting security events
• Errors related to missing, invalid, or malformed routing_key values

In this guide, you’ll learn how the Wazuh-PagerDuty integration works, why routing key errors occur, how to interpret common log messages, and the exact steps required to diagnose and resolve webhook failures.

You’ll also learn several best practices that help prevent these issues from returning after upgrades or configuration changes.

Understanding How Wazuh Integrates with PagerDuty

Successful troubleshooting begins with understanding how alerts travel from Wazuh to PagerDuty.

Knowing each stage of the integration makes it much easier to identify where failures occur and why PagerDuty may reject incoming requests.

Wazuh Alerting Workflow

When Wazuh detects a security event, it generates an alert after analyzing logs, rules, decoders, and threat intelligence data.

Depending on your configuration, alerts meeting specific severity thresholds are forwarded to external integrations through the Integrator component.

The simplified workflow looks like this:

Endpoint or Log Source
        ↓
Wazuh Agent
        ↓
Wazuh Manager
        ↓
Rules & Decoders
        ↓
Generated Alert
        ↓
Integrator Module
        ↓
PagerDuty Events API
        ↓
PagerDuty Incident

Because the Integrator sends alerts only after Wazuh has successfully processed them, failures occurring after this stage generally involve API communication rather than detection problems.

If Wazuh isn’t generating alerts correctly in the first place, review your rule configuration first.

Related Guide: How to Test Wazuh Rules

Role of Integrations in Wazuh

The Wazuh Integrator is responsible for forwarding alerts to external security platforms and notification services.

It supports numerous integrations, including:

• PagerDuty
• Slack
• VirusTotal
• Shuffle
• Maltiverse
• Custom webhooks

Rather than simply forwarding raw logs, the Integrator formats alert data into the structure expected by the destination platform.

This formatting becomes particularly important with PagerDuty because its Events API requires specific JSON fields before accepting requests.

Configuration mistakes inside the integration definition often lead directly to HTTP 400 responses.

For similar webhook troubleshooting techniques, see Fix Wazuh Slack Webhook Errors: Curl 52 Empty Reply.

How PagerDuty Events API Receives Alerts

PagerDuty receives alerts through its Events API v2, which accepts HTTPS POST requests containing JSON payloads.

Every request must include several required fields, including:

• routing_key
• event_action
• payload
• summary
• source
• severity

If any required field is missing, malformed, or contains invalid values, PagerDuty immediately returns an HTTP 400 Bad Request response instead of creating an incident.

The official PagerDuty Events API documentation provides the complete payload specification.

What the routing_key Does

The routing_key is arguably the most important field in the PagerDuty payload.

This key uniquely identifies which PagerDuty Event Orchestration or Integration should receive the event.

It effectively acts as an authentication token that tells PagerDuty where to route incoming alerts.

If the routing key is:

• Missing
• Incorrect
• Expired
• Copied from the wrong integration
• Contains extra whitespace
• Truncated during configuration

PagerDuty rejects the request before processing the remainder of the payload.

Unlike traditional API keys, the routing key is specific to a PagerDuty integration endpoint, meaning copying one from another service or environment (such as staging instead of production) commonly results in rejected requests.

Typical Data Flow Between Wazuh and PagerDuty

Under normal conditions, the complete integration process follows these steps:

1. Wazuh detects a security event.
2. A matching rule generates an alert.
3. The Integrator formats the alert into a PagerDuty Events API payload.
4. The payload includes the configured routing key.
5. Wazuh sends the HTTPS POST request.
6. PagerDuty validates the request.
7. If validation succeeds, PagerDuty creates or updates an incident.
8. The appropriate on-call responder receives a notification.

If validation fails at Step 6, PagerDuty responds with HTTP 400, and no incident is created.

According to PagerDuty’s developer documentation, the Events API performs validation before incident creation, making malformed payloads one of the most common causes of rejected events.

Industry guidance also emphasizes validating webhook payloads before deployment.

The experts at the OWASP Web Security Testing Guide recommend verifying API request structure, required fields, and authentication during integration testing to prevent failures caused by malformed requests.

Common Symptoms of PagerDuty Integration Failures

PagerDuty integration failures rarely occur silently.

Wazuh, the operating system, and PagerDuty itself usually provide valuable clues that point toward the underlying issue.

Recognizing these symptoms can significantly reduce troubleshooting time.

Webhook Failed Status Code 400 Errors

The most obvious symptom appears in the Wazuh integrator logs:

Webhook failed. Status code: 400

An HTTP 400 response indicates that PagerDuty received the request but rejected it because something about the payload was invalid.

Common causes include:

• Missing routing key
• Invalid JSON formatting
• Unsupported field values
• Missing required payload attributes
• Incorrect Events API endpoint
• Corrupted request body

Unlike network or firewall issues, a 400 response confirms that communication with PagerDuty succeeded, the request simply failed validation.

Alerts Not Appearing in PagerDuty

Another common symptom is that Wazuh continues generating alerts normally while PagerDuty never creates incidents.

Administrators often notice:

• Alerts visible in the Wazuh Dashboard
• Active Wazuh rules firing correctly
• No corresponding PagerDuty notifications
• Empty incident timeline

This usually indicates that detection is functioning properly, while the integration layer is failing.

If alerts are missing entirely inside Wazuh, first verify your detection pipeline.

PagerDuty Events Rejected

PagerDuty may explicitly reject Events API requests because of validation failures.

Common rejection reasons include:

• Invalid routing key
• Missing payload fields
• Invalid severity values
• Unsupported event_action
• Malformed JSON
• Authentication errors

The PagerDuty API response often contains useful diagnostic information that identifies the exact validation failure.

Wazuh Integrator Log Errors

The Wazuh integrator logs frequently contain additional context beyond the HTTP status code.

You may encounter errors referencing:

• Failed POST requests
• Invalid responses
• Connection retries
• Payload formatting issues
• JSON serialization errors
• API response bodies

Reviewing these logs should always be one of the first troubleshooting steps, as they often reveal whether the issue is related to configuration, connectivity, or payload validation.

Missing Incident Creation Despite Triggered Alerts

Perhaps the most confusing symptom occurs when everything appears operational:

• Wazuh detects threats.
• Rules execute successfully.
• Alerts are generated.
• The integration runs.
• No PagerDuty incident is ever created.

In many cases, this points to a payload validation problem rather than an issue with Wazuh itself.

Because PagerDuty rejects invalid requests before creating incidents, administrators may mistakenly believe alerts were never sent.

For broader troubleshooting involving external integrations, you may also find these guides useful:

• Fixing Microsoft Graph API Authentication Failures in Wazuh
• How to Integrate Wazuh with VirusTotal for Threat Intelligence

What Causes the “Webhook Failed Status Code 400” Error?

A 400 Bad Request response from PagerDuty means the request reached the Events API, but the API rejected it because it failed validation.

Unlike connection timeouts or authentication failures, this error indicates that the request itself is malformed or contains invalid information.

Below are the most common causes of this error when integrating Wazuh with PagerDuty.

Invalid routing_key

The most frequent cause of HTTP 400 responses is an invalid routing_key.

The routing key identifies the PagerDuty service that should receive the event.

If the value doesn’t match an active integration, PagerDuty cannot determine where to route the alert and rejects the request.

Common reasons include:

• Using a routing key from the wrong PagerDuty service
• Accidentally deleting or recreating the PagerDuty integration
• Copying only part of the routing key
• Including leading or trailing spaces
• Using an old routing key after rotating integrations

If you’ve recently modified your PagerDuty services or Event Orchestration configuration, verify that the routing key configured in Wazuh still matches the active integration.

Incorrect JSON Payload Structure

PagerDuty’s Events API expects requests to follow a strict JSON schema.

Even a small formatting mistake can cause the API to reject the request before processing it.

Examples include:

• Missing opening or closing braces
• Incorrect nesting of objects
• Missing quotation marks
• Invalid JSON syntax
• Improper escaping of special characters

For example, omitting the required payload object or placing required fields at the wrong level will trigger a validation error.

The PagerDuty Events API documentation outlines the required request structure and field hierarchy.

Expired or Deleted PagerDuty Integration

A routing key remains valid only while its associated integration exists.

If an administrator:

• Deletes the PagerDuty service
• Removes the Events API integration
• Recreates the integration
• Migrates to Event Orchestration

PagerDuty generates a new routing key, and the previous one immediately becomes invalid.

This situation commonly occurs after infrastructure migrations or service reorganization, where the Wazuh configuration is overlooked.

Whenever PagerDuty integrations are modified, update the corresponding configuration in Wazuh to prevent failed webhook requests.

Typographical Errors in Configuration Files

Manual edits to configuration files are another common source of integration failures.

Simple mistakes such as:

• Misspelled parameter names
• Missing XML closing tags
• Incorrect quotation marks
• Extra whitespace
• Incorrect capitalization

can prevent Wazuh from generating a valid API request.

For example, a single missing character in the routing key is enough to invalidate the request.

After making configuration changes, always validate the XML syntax before restarting the Wazuh Manager.

If you’re troubleshooting configuration issues, this guide may also help:

Related Guide: How to Fix ossec.conf Syntax Errors in Wazuh Agents

Unsupported Event Payload Fields

PagerDuty only accepts supported fields defined by the Events API.

Including unsupported attributes or assigning invalid values may cause the request to fail validation.

Examples include:

• Invalid severity values
• Unsupported event_action values
• Incorrect timestamp formats
• Custom fields placed outside supported objects

Whenever you customize the integration or modify the payload format, verify that every field conforms to the current Events API specification.

Corrupted Integration Parameters

Configuration corruption can occur during:

• Manual editing
• Configuration management deployments
• File synchronization
• Version upgrades
• Copying configuration between environments

Common examples include:

• Truncated routing keys
• Hidden control characters
• Broken XML encoding
• Incorrect line endings
• Partial configuration merges

If the configuration appears correct but errors persist, recreate the integration configuration manually instead of copying existing values.

This eliminates hidden formatting problems that are difficult to detect visually.

Understanding the PagerDuty routing_key Requirement

The routing_key is one of the most important components of the Wazuh-PagerDuty integration.

Without a valid routing key, PagerDuty cannot authenticate or route incoming events, resulting in rejected webhook requests and HTTP 400 responses.

Understanding how this key works makes troubleshooting significantly easier.

What Is a routing_key?

A routing_key is a unique identifier assigned to a PagerDuty Events API integration.

Every event submitted to the Events API must include this value.

When PagerDuty receives the request, it uses the routing key to determine:

• Which service should receive the alert
• Which escalation policy applies
• Which responders should be notified
• Which Event Orchestration rules should execute

Unlike a general API token, the routing key is tied to a specific integration rather than an entire PagerDuty account.

Difference Between Service IDs and Routing Keys

New administrators sometimes confuse PagerDuty Service IDs with routing keys.

Although both identify resources within PagerDuty, they serve completely different purposes.

Using a Service ID where a routing key is expected always results in rejected API requests.

Where to Find the Routing Key in PagerDuty

You can locate the routing key by navigating to your PagerDuty service and opening the Events API integration.

The exact navigation path may vary slightly depending on your PagerDuty plan and whether you’re using traditional integrations or Event Orchestration, but generally follows this process:

1. Open the PagerDuty console.
2. Select the appropriate service.
3. Open Integrations.
4. Locate the Events API v2 integration.
5. Copy the displayed routing key.

Always copy the complete value without adding spaces or extra characters.

PagerDuty’s official documentation explains how integrations generate routing keys and how they are used.

How Routing Keys Are Used by the Events API

Every event submitted through the Events API begins with routing key validation.

PagerDuty performs several checks before creating an incident:

1. Does the routing key exist?
2. Is the integration active?
3. Is the request properly formatted?
4. Are all required fields present?
5. Is the payload valid?

Only after these checks succeed does PagerDuty create or update an incident.

This validation process explains why many integration problems produce an immediate HTTP 400 response instead of partially processing the alert.

Common Mistakes When Copying Routing Keys

Routing keys are long strings of characters, making them prone to copying mistakes.

The most common errors include:

• Copying only part of the key
• Including trailing spaces
• Adding quotation marks from documentation
• Using a routing key from another environment
• Mixing production and testing integrations
• Copying a Service ID instead of the routing key
• Forgetting to update Wazuh after regenerating the integration

To reduce human error, consider storing routing keys in a secure secrets management solution rather than repeatedly copying them during deployments.

How to Verify Your PagerDuty Integration Configuration

Before modifying firewall rules or debugging network traffic, verify that the Wazuh integration itself is configured correctly.

Many PagerDuty webhook failures originate from small configuration mistakes that can be identified in just a few minutes.

Inspecting the Wazuh Integration Configuration

Begin by reviewing the integration configuration used by the Wazuh Manager.

Depending on your deployment, this configuration typically specifies:

• The integration name
• PagerDuty endpoint
• Routing key
• Alert level
• Alert formatting
• Integration options

Carefully compare every configured value against the official PagerDuty documentation.

Pay particular attention to copied values, as hidden whitespace and truncated strings are surprisingly common causes of failures.

Reviewing ossec.conf Settings

The primary configuration file for Wazuh is:

/var/ossec/etc/ossec.conf

Review the integration section and verify:

• The XML is well formed.
• The integration name is correct.
• Required parameters are present.
• The routing key matches the PagerDuty integration.
• There are no duplicate integration entries.
• XML tags are properly nested.

After making changes, validate the configuration before restarting the Wazuh Manager.

If you’re troubleshooting XML-related issues, see How to Fix ossec.conf Syntax Errors in Wazuh Agents.

Confirming Integration Type and Parameters

Ensure the integration configuration matches the PagerDuty Events API version you’re using.

Verify that:

• The integration targets the correct PagerDuty service.
• Required parameters are populated.
• Event formatting matches the API specification.
• Severity mappings are valid.
• Trigger conditions are configured correctly.

Configuration copied from older tutorials may reference deprecated parameters or older API formats that are no longer accepted.

Checking Integration Scripts

If you’ve customized the PagerDuty integration or use wrapper scripts, inspect them carefully.

Review for:

• Incorrect JSON generation
• Missing required fields
• Invalid character encoding
• Script execution failures
• Improper variable substitution
• Payload manipulation that breaks the Events API schema

Even minor modifications to integration scripts can inadvertently remove required fields or alter the request structure.

Validating Configuration Syntax

Before restarting Wazuh, perform a complete configuration review.

Confirm that:

• XML syntax is valid.
• JSON payload generation produces valid output.
• Routing keys exactly match PagerDuty.
• Required fields are populated.
• File permissions allow Wazuh to read the configuration.
• No duplicate or conflicting integration definitions exist.

The Wazuh documentation recommends validating configuration changes and testing integrations after every modification to avoid introducing syntax errors or broken alert workflows.

How to Fix Invalid routing_key Errors

An invalid routing_key is the leading cause of PagerDuty integration failures in Wazuh.

Fortunately, it’s also one of the easiest problems to resolve once you’ve identified that the routing key is incorrect, outdated, or no longer associated with an active PagerDuty integration.

Follow the steps below to restore successful alert delivery.

Generate a New Events API Integration

If you suspect the existing routing key has been deleted, regenerated, or compromised, create a new Events API v2 integration within PagerDuty.

A new integration generates a fresh routing key that can be used immediately.

Creating a new integration is often the fastest solution when:

• The original integration was deleted.
• The routing key is unknown.
• The key may have been exposed.
• The PagerDuty service was recreated.
• You’re migrating to a new environment.

Avoid reusing routing keys from other services or environments, as each key is tied to a specific integration.

For detailed instructions, refer to the official PagerDuty documentation.

Copy the Correct Integration Key

Once the integration has been created, copy the routing key directly from PagerDuty.

When copying the value:

• Copy the entire string.
• Do not include quotation marks.
• Remove any leading or trailing whitespace.
• Verify that the key belongs to the intended service.
• Double-check that you’re using the production key for production systems.

A good practice is to paste the key into a plain text editor before adding it to your configuration.

This helps eliminate hidden formatting characters that may be introduced when copying from a browser or password manager.

Update Wazuh Configuration Files

After obtaining the correct routing key, update the Wazuh integration configuration.

Typically, this involves editing the appropriate integration settings in:

/var/ossec/etc/ossec.conf

Verify that:

• The routing key matches the PagerDuty integration exactly.
• There are no duplicate PagerDuty integration entries.
• XML formatting remains valid.
• No hidden characters were introduced while editing.

After saving your changes, validate the configuration before restarting Wazuh.

Restart Wazuh Services

Configuration changes do not take effect until the relevant Wazuh services reload the updated settings.

After updating the configuration:

1. Save the configuration file.
2. Restart the Wazuh Manager.
3. Confirm that the Integrator starts successfully.
4. Verify there are no startup errors in the logs.

Immediately after restarting, review the manager and integrator logs for configuration errors that may prevent the integration from loading.

Test Event Delivery

The final step is confirming that PagerDuty can successfully receive events.

Generate a test alert by:

• Triggering a known Wazuh rule.
• Running a safe test event.
• Using an existing alert generation method within your environment.

Then verify:

• The request reaches PagerDuty.
• No HTTP 400 errors appear in the logs.
• PagerDuty creates an incident.
• Notifications are delivered according to the escalation policy.

Experts recommend validating every third-party integration after configuration changes rather than waiting for a real security incident to reveal a problem.

Regular integration testing is also encouraged in the NIST Cybersecurity Framework (CSF 2.0) as part of maintaining effective incident response capabilities.

Fixing PagerDuty Webhook Status Code 400 Errors

Once you’ve verified the routing key, the next step is ensuring that the request being sent to PagerDuty conforms to the Events API specification.

Even a valid routing key cannot compensate for malformed or incomplete requests.

Validate JSON Payload Formatting

PagerDuty accepts only properly formatted JSON.

Before sending requests, validate the payload using a JSON validator or your preferred development tools.

Check for:

• Missing braces
• Missing commas
• Improper quotation marks
• Incorrect nesting
• Invalid escape characters
• Duplicate fields

Malformed JSON prevents PagerDuty from parsing the request, resulting in an immediate HTTP 400 response.

The official Events API documentation provides examples of valid request bodies.

Ensure Required Fields Are Present

The Events API requires several mandatory fields.

At a minimum, verify that the payload contains:

• routing_key
• event_action
• payload
• summary
• source
• severity

Depending on your implementation, additional fields may also be required.

If any mandatory field is missing or placed outside the expected JSON structure, PagerDuty rejects the request before processing it.

Verify Trigger Action Values

The event_action field tells PagerDuty what operation should be performed.

Supported values include:

• trigger
• acknowledge
• resolve

Using unsupported values such as:

create
alert
incident
notify

results in request validation failures.

Ensure that every request uses one of the officially supported actions documented by PagerDuty.

Check Character Encoding Issues

Character encoding problems are less common but can still produce malformed requests.

Potential issues include:

• Non-UTF-8 encoding
• Hidden Unicode characters
• Smart quotes inserted by text editors
• Invalid escape sequences
• Corrupted line endings

These problems frequently occur when configuration files are edited using different operating systems or copied from formatted documents.

When possible, edit configuration files with a plain text editor that preserves UTF-8 encoding.

Remove Unsupported Payload Elements

Custom integrations sometimes include additional JSON fields that PagerDuty does not recognize.

While some custom fields are supported within designated objects, placing arbitrary attributes at the wrong level can cause validation failures.

Review the payload for:

• Deprecated parameters
• Unsupported custom fields
• Incorrect nesting
• Legacy Events API v1 attributes
• Duplicate properties

If you’ve customized the integration, compare the generated payload with the current PagerDuty Events API specification to ensure compatibility.

Reviewing Wazuh Logs for Integration Errors

The Wazuh logs are often the fastest way to determine why PagerDuty rejected a request.

Rather than relying solely on the HTTP status code, administrators should examine the surrounding log entries to identify configuration mistakes, payload validation failures, or communication problems.

Wazuh Integrator Logs

The Wazuh Integrator logs provide the most direct insight into third-party notification failures.

These logs typically record:

• Outgoing webhook requests
• HTTP response codes
• Connection failures
• Retry attempts
• Integration execution errors

When troubleshooting PagerDuty, this should be your first stop.

Look for entries similar to:

Webhook failed. Status code: 400

or

Failed sending event to PagerDuty

The timestamps associated with these entries can be correlated with PagerDuty logs to identify exactly when requests were rejected.

Manager Logs

The Wazuh Manager logs provide additional context before alerts reach the Integrator.

These logs help determine whether:

• The alert was generated successfully.
• Rules executed correctly.
• Integrations were invoked.
• Configuration loaded without errors.
• Startup validation completed successfully.

If no alert appears in the manager logs, the issue may lie with rule generation rather than the PagerDuty integration itself.

API-Related Error Messages

Many integration failures include detailed API response messages beyond the HTTP status code.

Examples include:

• Invalid routing key
• Missing required field
• Invalid event_action
• Invalid payload
• JSON parsing error
• Authentication failure

These responses often identify the exact validation rule that failed, making them far more useful than the status code alone.

Whenever possible, capture the full response body rather than only logging the HTTP code.

Identifying Failed PagerDuty Requests

When reviewing logs, identify patterns rather than isolated failures.

Pay attention to:

• Repeated HTTP 400 responses
• Errors occurring after configuration changes
• Failures affecting only one PagerDuty service
• Alerts from specific Wazuh rules
• Timestamp correlations with PagerDuty activity

A consistent pattern can quickly narrow the investigation to a particular configuration file, integration, or payload.

Extracting Detailed Error Information

If the default logs don’t provide enough information, increase the logging verbosity temporarily and reproduce the issue.

Collect as much diagnostic information as possible, including:

• Complete HTTP response body
• Request timestamps
• Payload samples (with sensitive values removed)
• Integration configuration
• Relevant manager and integrator log entries

Providing this information greatly accelerates troubleshooting and makes it easier to compare your requests against the official PagerDuty Events API specification or seek assistance from the Wazuh or PagerDuty communities.

For additional guidance on troubleshooting Wazuh integrations, you may also find these articles useful:

• Fix Wazuh Slack Webhook Errors: Curl 52 Empty Reply
• Troubleshooting Wazuh GitHub Module: Fixing 400 Bad Request Payloads

curl -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "<YOUR_ROUTING_KEY>",
    "event_action": "trigger",
    "payload": {
      "summary": "Wazuh PagerDuty integration test",
      "source": "Wazuh Manager",
      "severity": "warning"
    }
  }'

{
  "status": "success",
  "message": "Event processed",
  "dedup_key": "..."
}

nslookup events.pagerduty.com

dig events.pagerduty.com

curl -I https://events.pagerduty.com

openssl s_client -connect events.pagerduty.com:443

https://events.pagerduty.com/v2/enqueue

/var/ossec/logs/ossec.log

curl -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "YOUR_ROUTING_KEY",
    "event_action": "trigger",
    "payload": {
      "summary": "PagerDuty integration test",
      "source": "Wazuh",
      "severity": "warning"
    }
  }'

https://events.pagerduty.com/v2/enqueue

systemctl restart wazuh-manager