How to Fix Wazuh All Shards Failed Error

The Wazuh All Shards Failed error is one of the most disruptive problems you can encounter in a Wazuh deployment. When this error appears, the Wazuh Dashboard is unable to retrieve data from the Wazuh Indexer, causing searches, visualizations, dashboards, and security alerts to stop working correctly. Instead of returning log data, the dashboard displays an “all shards failed” message because every shard involved in the search request has become unavailable.

Unlike isolated index issues that affect only specific datasets, this error typically indicates a broader problem with the underlying OpenSearch cluster.

The root cause may range from insufficient disk space and unhealthy cluster status to failed nodes, corrupted indexes, excessive JVM memory usage, or incorrect shard allocation.

In many environments, administrators first notice the problem after a node outage, storage exhaustion, or an unsuccessful upgrade.

The good news is that the error is usually recoverable once the underlying cluster health problem is identified.

Most cases can be resolved by restoring healthy shard allocation, recovering missing indexes, or fixing resource bottlenecks before permanent data loss occurs.

In this guide, you’ll learn:

• What the Wazuh All Shards Failed error actually means
• Why it occurs in Wazuh Indexer (OpenSearch)
• The most common causes behind failed shards
• Step-by-step troubleshooting techniques to restore cluster health
• Best practices to prevent the problem from happening again

If your cluster is already reporting unhealthy status, you may also find this guide helpful: How to Fix a Yellow Cluster Status in Wazuh Indexer.

For readers who want to better understand how Wazuh stores and distributes data across an OpenSearch cluster, see the official OpenSearch documentation on shards and indexes.

What Does the Wazuh All Shards Failed Error Mean?

The All Shards Failed error indicates that OpenSearch was unable to execute a search because every shard involved in the request failed to respond successfully.

Since Wazuh stores alerts, archives, vulnerability data, and monitoring information inside OpenSearch indexes, the dashboard depends entirely on successful shard operations to display data.

Understanding Shards in Wazuh Indexer (OpenSearch)

A shard is simply a partition of an OpenSearch index.

Rather than storing an entire index on a single machine, OpenSearch divides the index into multiple primary shards and optionally creates replica shards for redundancy and high availability.

For example, a Wazuh alerts index might consist of:

• 3 primary shards
• 1 replica per primary shard
• Data distributed across multiple Indexer nodes

This architecture allows Wazuh to:

• Scale horizontally
• Balance workloads across nodes
• Improve search performance
• Continue operating if individual nodes fail

The official OpenSearch documentation provides an excellent overview of shard architecture.

Understanding How Search Requests Work

Whenever you search in Wazuh Dashboard, OpenSearch performs the following process:

1. Identifies every index involved in the search.
2. Locates the shards containing the requested data.
3. Sends the query to every required shard.
4. Combines the results.
5. Returns the final response to the dashboard.

Normally, if one replica becomes unavailable, OpenSearch can still retrieve data from another healthy copy.

However, if every shard required for the query fails, OpenSearch cannot produce a valid response and returns the all shards failed exception.

Why Searches Fail When All Shards Become Unavailable

Several conditions can prevent shards from serving search requests:

• Primary shards are unassigned
• Indexer nodes are offline
• Disk usage exceeds safety thresholds
• Shards are corrupted
• JVM memory is exhausted
• Cluster recovery is incomplete
• Search execution times out

When enough failures occur simultaneously, OpenSearch has nowhere to execute the query, causing every shard request to fail.

This doesn’t always mean data has been permanently lost.

In many cases, the shards still exist but cannot be allocated or recovered until the underlying infrastructure issue is resolved.

Single Failed Shard vs. All Shards Failed

It’s important to distinguish between these two situations.

If your cluster has recently experienced allocation problems, you should also review How to Design a Wazuh OpenSearch Shard Allocation Strategy and How to Configure Wazuh Index State Management (ISM), as both topics help reduce the likelihood of widespread shard failures.

Common Causes of the Wazuh All Shards Failed Error

Although the error message appears simple, the underlying cause can vary significantly.

In most environments, the failure originates from cluster health issues, storage limitations, hardware failures, or resource exhaustion.

Red Cluster Status

A Red cluster status is one of the most common causes of the All Shards Failed error.

A Red status means one or more primary shards are unavailable. Since primary shards contain the authoritative copy of the data, searches cannot complete successfully.

Typical reasons include:

• Node failures
• Corrupted indexes
• Failed shard allocation
• Missing storage devices
• Interrupted cluster recovery

When primary shards remain unavailable, every search targeting those indexes will fail.

If your deployment is already reporting cluster health issues, see How to Fix a Yellow Cluster Status in Wazuh Indexer, as many of the same troubleshooting techniques also apply to Red clusters.

Unassigned Primary Shards

Primary shards can become UNASSIGNED after:

• Unexpected server shutdowns
• Storage failures
• Cluster configuration changes
• Failed upgrades
• Lost Indexer nodes

You can identify unassigned shards using the OpenSearch Cluster Allocation API.

According to OpenSearch documentation, the Cluster Allocation Explain API is the primary tool for determining why shards cannot be allocated.

Cluster Unable to Recover Indexes

Sometimes OpenSearch cannot successfully recover indexes during startup.

Recovery may fail because of:

• Missing shard copies
• Incompatible node versions
• Corrupted metadata
• Storage failures
• Interrupted recoveries

Until recovery finishes successfully, affected shards remain unavailable.

Disk Watermarks Reached

Disk space problems are among the most frequent causes of shard failures.

OpenSearch continuously monitors available disk capacity using configurable watermarks.

When these thresholds are exceeded, OpenSearch intentionally restricts shard allocation to protect cluster stability.

Low Disk Space

When storage begins running out:

• New shards cannot be allocated.
• Existing shards may relocate.
• Recovery operations slow dramatically.
• Search performance declines.

Large Wazuh environments often accumulate hundreds of gigabytes, or even terabytes, of historical security data if retention policies are not managed properly.

For long-term storage planning, see How to Manage Wazuh Storage.

Flood-Stage Watermark

Once disk usage reaches the flood-stage watermark, OpenSearch automatically places affected indexes into read-only mode.

This safety mechanism prevents the filesystem from becoming completely full, which could otherwise result in index corruption.

Read-Only Indexes

Read-only indexes may prevent normal indexing and recovery operations.

Administrators sometimes clear disk space without removing the read-only block afterward, leaving the cluster unable to return to normal operation.

Wazuh Indexer Node Failures

If one or more Indexer nodes become unavailable, shards stored on those nodes disappear until the nodes recover or replicas take over.

Common causes include:

Offline Nodes

• Power failures
• System maintenance
• Virtual machine shutdowns
• Cloud instance termination

Network Communication Issues

Cluster communication may fail because of:

• Firewall changes
• DNS failures
• TLS certificate problems
• Network partitions
• Switch failures

Nodes that cannot communicate with the cluster may appear healthy individually but remain unavailable for shard allocation.

Node Crashes

Unexpected crashes caused by software bugs, kernel panics, or resource exhaustion may leave multiple primary shards unavailable until the node restarts successfully.

Corrupted or Missing Indexes

Physical index corruption is less common but considerably more serious.

Potential causes include:

Failed Recovery

Recovery operations interrupted during restart or node migration can leave indexes in an inconsistent state.

Damaged Shard Data

Hardware failures, filesystem corruption, or storage controller problems can damage shard files.

Although modern filesystems help reduce corruption risks, they cannot eliminate them entirely.

Snapshot Restoration Issues

If snapshot restoration fails or is interrupted, some shards may never finish restoring, leaving indexes incomplete.

Organizations following a regular backup strategy typically recover much faster than those relying solely on live cluster recovery.

JVM Memory Pressure

Wazuh Indexer depends on the Java Virtual Machine (JVM).

If available heap memory becomes exhausted, OpenSearch may struggle to execute searches or allocate shards.

Heap Exhaustion

High memory usage can occur because of:

• Large aggregations
• Excessive shard counts
• Heavy indexing activity
• Large field caches
• Memory leaks

Garbage Collection Delays

Long garbage collection pauses temporarily stop JVM execution.

During these pauses:

• Search requests time out.
• Cluster communication slows.
• Nodes may temporarily leave the cluster.
• Shards become unavailable.

Search Failures

When JVM pressure becomes severe, OpenSearch may reject search requests altogether rather than risking node instability.

For environments frequently experiencing memory issues, see How to Tune OpenSearch Heap Size to Stop Wazuh High Memory Crashes.

Google’s Site Reliability Engineering guidance emphasizes that resource exhaustion, especially memory pressure and cascading failures, is a common contributor to distributed system outages, reinforcing the importance of proactive capacity management.

Cluster Configuration Problems

Improper cluster configuration can prevent shards from being allocated correctly even when the hardware is functioning normally.

Examples include:

Incorrect Shard Allocation

Allocation filters may accidentally prevent shards from being assigned to eligible nodes.

Discovery Configuration Issues

Incorrect cluster discovery settings may stop nodes from joining the cluster after restarts or upgrades.

Split-Brain Recovery

Although modern OpenSearch versions include protections against split-brain scenarios, improperly recovered clusters may require manual intervention before shard allocation returns to normal.

Hardware or Storage Failures

Finally, the underlying infrastructure may simply be failing.

Disk Errors

Bad sectors, SSD wear, or filesystem corruption can prevent shard data from being read successfully.

RAID Failures

RAID controller failures or degraded arrays may make portions of shard storage temporarily inaccessible.

Storage Latency

Even without complete hardware failure, high storage latency can significantly delay shard recovery and search execution.

Slow disks, overloaded SAN environments, or congested network-attached storage can cause nodes to miss cluster timeouts, resulting in unavailable shards and failed search requests.

As noted by experienced OpenSearch practitioners, maintaining fast, reliable storage, preferably SSD-backed volumes for hot data, and continuously monitoring disk latency is critical to stable shard allocation and cluster health.

Symptoms of the Error

The Wazuh All Shards Failed error is rarely an isolated issue.

In most cases, it is a symptom of an unhealthy OpenSearch cluster that can affect nearly every component of your Wazuh deployment.

Recognizing these symptoms early can help you diagnose the underlying problem before it results in prolonged downtime or data loss.

Wazuh Dashboard Displays “All Shards Failed”

The most obvious symptom is the error message itself.

When users attempt to view dashboards, run searches, or access security data, the Wazuh Dashboard displays an error similar to:

all shards failed

or

search_phase_execution_exception
all shards failed

Because every search request depends on healthy shards, the dashboard cannot retrieve the requested information.

Searches Return No Results

Search queries that previously worked may suddenly fail.

Users might experience:

• Empty search results
• HTTP 500 errors
• Search execution exceptions
• Long-running queries that eventually time out

This often affects searches across multiple indexes rather than a single dataset.

Dashboards Fail to Load

Most Wazuh dashboards rely on aggregating data from several indexes simultaneously.

If those indexes become unavailable, visualizations cannot be generated.

Common symptoms include:

• Endless loading indicators
• Blank dashboards
• Missing charts
• Visualization errors
• Timeout messages

Because dashboards aggregate data from many shards, they are often one of the first components to fail.

Indexer APIs Return Errors

Direct API requests to the Wazuh Indexer may also begin returning errors such as:

• search_phase_execution_exception
• all shards failed
• index_not_found_exception
• no_shard_available_action_exception

These responses indicate that the problem exists within the Indexer itself rather than the Dashboard.

Alerts Stop Appearing

In many environments, security alerts stop appearing even though monitored systems continue generating events.

This happens because:

• New events cannot be indexed.
• Existing indexes are unavailable.
• Dashboard searches cannot retrieve recent alerts.

Administrators may mistakenly assume agents have stopped reporting when the actual issue is a failing Indexer cluster.

If alerts are missing but the cluster remains healthy, also review Wazuh Email Alerts Not Working? Complete Fix Guide.

Cluster Health Becomes Yellow or Red

One of the strongest indicators is degraded cluster health.

• Green — All primary and replica shards are assigned.
• Yellow — Primary shards are healthy, but one or more replica shards are unassigned.
• Red — One or more primary shards are unavailable.

Although a Yellow cluster can still serve most searches, a Red cluster frequently produces the All Shards Failed error because required primary shards cannot be accessed.

How to Diagnose the Problem

Before attempting repairs, identify why the shards are failing.

OpenSearch provides several APIs and system-level tools that reveal the health of your cluster, storage, and Indexer nodes.

A systematic diagnosis helps avoid making unnecessary changes that could worsen the situation.

Check Cluster Health

The first step is checking overall cluster health.

Run:

curl -k -u admin:password \
https://INDEXER_IP:9200/_cluster/health?pretty

Example output:

{
  "cluster_name": "wazuh-cluster",
  "status": "yellow",
  "number_of_nodes": 3,
  "active_primary_shards": 95,
  "active_shards": 180,
  "unassigned_shards": 10
}

How to Interpret the Results

Green

A Green cluster indicates:

• Every primary shard is assigned.
• Every replica shard is assigned.
• Searches should complete successfully.

If you’re still seeing All Shards Failed with a Green cluster, the issue may involve corrupted indexes, query-specific failures, or JVM memory pressure.

Yellow

A Yellow cluster means:

• Primary shards are healthy.
• Replica shards are missing.
• Searches usually continue working.
• Hardware redundancy has been reduced.

Yellow status is often a warning that a more serious problem is developing.

Red

A Red cluster requires immediate attention.

It means:

• One or more primary shards are unavailable.
• Some indexes cannot be searched.
• Data may be temporarily inaccessible.
• Dashboard functionality may be severely affected.

The official OpenSearch cluster health API documentation explains every returned field in detail.

List Unassigned Shards

Next, determine which shards are unavailable.

Run:

curl -k -u admin:password \
https://INDEXER_IP:9200/_cat/shards?v

Example output:

index                 shard prirep state
wazuh-alerts-4.x      0     p      UNASSIGNED
wazuh-alerts-4.x      0     r      UNASSIGNED

What to Look For

Pay particular attention to:

• UNASSIGNED shards
• Missing primary shards (p)
• Large numbers of relocating shards
• Repeated failures involving the same index

If primary shards remain unassigned, searches against those indexes will often fail completely.

View Cluster Allocation

Once you’ve identified unassigned shards, determine why OpenSearch cannot allocate them.

Run:

curl -k -u admin:password \
https://INDEXER_IP:9200/_cluster/allocation/explain?pretty

The response explains the allocation decision for a shard and may include reasons such as:

• Insufficient disk space
• Missing nodes
• Allocation filtering
• Existing shard copies
• Read-only indexes
• Recovery failures

This API is one of the most valuable troubleshooting tools because it tells you exactly why OpenSearch refuses to start a shard instead of leaving you to guess.

Check Disk Usage

Storage problems account for a large percentage of shard failures.

Begin by checking filesystem usage:

df -h

Then determine how much space the Indexer data directory consumes:

du -sh /var/lib/wazuh-indexer

Why Disk Usage Matters

When available storage falls below OpenSearch’s configured watermarks, the cluster may:

• Stop allocating new shards.
• Block writes.
• Mark indexes as read-only.
• Prevent recovery operations.

For long-term storage optimization, see How to Manage Wazuh Storage.

Review Wazuh Indexer Logs

Indexer logs often identify the root cause directly.

Typical locations include:

/var/log/wazuh-indexer/
/var/log/opensearch/

Look for messages containing:

• all shards failed
• disk watermark exceeded
• OutOfMemoryError
• failed shard
• corrupt index
• allocation failed
• master not discovered
• recovery failed

These messages frequently pinpoint the failing component before other monitoring tools do.

Check JVM Heap Usage

Memory exhaustion is another common cause of search failures.

Useful APIs include:

curl -k -u admin:password \
https://INDEXER_IP:9200/_nodes/stats/jvm?pretty

or

curl -k -u admin:password \
https://INDEXER_IP:9200/_cat/nodes?v

Monitoring Heap Utilization

Watch for:

• Heap usage consistently above 85%
• Frequent garbage collection
• Long GC pauses
• Memory circuit breaker exceptions
• OutOfMemoryError events

Sustained heap pressure often causes nodes to become unstable before they actually crash.

If heap usage is consistently high, see How to Tune OpenSearch Heap Size to Stop Wazuh High Memory Crashes.

Verify Node Status

Finally, ensure every Indexer node is participating in the cluster.

Run:

curl -k -u admin:password \
https://INDEXER_IP:9200/_cat/nodes?v

Verify that:

• Every expected node appears.
• Nodes have the correct roles.
• No nodes repeatedly leave and rejoin the cluster.
• Cluster manager elections are stable.

Missing nodes frequently explain why primary shards cannot be allocated.

How to Fix the Wazuh All Shards Failed Error

After identifying the root cause, you can begin restoring cluster health.

Always start with the least invasive solution, and avoid deleting indexes or forcing shard allocation unless you fully understand the consequences.

Whenever possible, ensure recent snapshots are available before making changes to shard allocation or index recovery.

Fix 1: Free Disk Space

If disk watermarks have been exceeded, freeing storage is usually the fastest way to restore normal shard allocation.

Consider the following actions:

• Remove unnecessary log files.
• Delete obsolete indexes that are no longer required.
• Delete old snapshots that are no longer needed.
• Expand storage volumes or add additional disks.
• Move historical data to lower-cost storage.

After freeing sufficient space, OpenSearch may automatically resume shard allocation.

If it does not, reroute or retry allocation once the cluster recognizes the additional capacity.

Fix 2: Remove Read-Only Blocks

When the flood-stage watermark is reached, OpenSearch automatically marks affected indexes as read-only to protect the filesystem from becoming completely full.

After resolving the storage issue, remove the block with:

curl -k -u admin:password -X PUT \
"https://INDEXER_IP:9200/_all/_settings" \
-H "Content-Type: application/json" \
-d '{
  "index.blocks.read_only_allow_delete": null
}'

Only remove the block after disk usage has fallen below the configured watermarks. Otherwise, OpenSearch may immediately reapply the protection.

Fix 3: Restart Wazuh Indexer

If nodes became unstable because of temporary resource exhaustion or failed services, restarting the Indexer can restore cluster functionality.

On most Linux systems:

sudo systemctl restart wazuh-indexer

After restarting, verify that:

• The service is running.
• All expected nodes rejoin the cluster.
• Cluster health improves.
• Previously unassigned shards begin recovering.

Do not restart multiple production nodes simultaneously unless your cluster architecture is designed to tolerate it.

See How to Set Up a Multi-Node Wazuh Cluster for high-availability deployment guidance.

Fix 4: Recover Unassigned Shards

If shards remain unassigned, use the allocation APIs to determine whether they can be recovered automatically or require manual intervention.

Depending on the root cause, OpenSearch may allow:

• Retrying failed allocations
• Allocating stale primary shards
• Allocating empty primary shards

Manual allocation should only be performed after confirming that healthy shard copies are unavailable.

Forcing allocation from stale or empty shards can result in permanent data loss if newer copies still exist elsewhere in the cluster.

Always review the allocation explanation before issuing allocation commands.

Fix 5: Restore Missing Indexes

If indexes cannot be recovered because shard data has been lost or corrupted, restore them from a recent snapshot.

A typical recovery process involves:

1. Identifying the latest healthy snapshot.
2. Restoring the affected indexes.
3. Verifying successful shard recovery.
4. Confirming searches return expected results.

If no usable snapshot exists, you may need to rebuild the affected indexes by reingesting data from the original log sources.

This may not be possible for historical events that are no longer available.

Fix 6: Increase JVM Heap

Persistent memory pressure can prevent shards from recovering and cause search requests to fail.

If your servers have sufficient RAM, increase the JVM heap size according to OpenSearch best practices, then restart the Indexer service.

After the restart, verify that:

• Heap utilization remains below critical levels.
• Garbage collection pauses decrease.
• Search latency improves.
• Shard recovery completes successfully.

Avoid assigning more than approximately 50% of system memory to the JVM, as the operating system also requires memory for filesystem caching and other processes.

Refer to the official OpenSearch performance guidance for sizing recommendations.

Also see How to Tune OpenSearch Heap Size to Stop Wazuh High Memory Crashes.

Fix 7: Repair Cluster Communication

If nodes cannot communicate reliably, shard allocation and recovery will fail.

Verify the following:

• All Indexer nodes can reach one another over the required ports.
• TLS certificates are valid and trusted.
• Hostnames and DNS resolution are correct.
• Discovery and cluster settings match across all nodes.
• Firewalls and security groups allow cluster traffic.
• Network latency and packet loss remain within acceptable limits.

Unstable cluster communication often causes nodes to repeatedly disconnect and rejoin, delaying recovery and leaving primary shards unassigned.

Fix 8: Resolve Corrupted Shards

If diagnostics indicate that specific shard data is corrupted, identify the affected indexes before taking corrective action.

Recommended recovery sequence:

1. Restore healthy shard copies if replicas are available.
2. Recover affected indexes from snapshots.
3. Validate index integrity after restoration.
4. Delete unrecoverable indexes only when all recovery options have been exhausted and the data is no longer recoverable.

Deleting damaged indexes may restore cluster health, but it also permanently removes the associated data.

Before proceeding, ensure you understand the impact on historical alerts, compliance requirements, and forensic investigations.

curl -k -u admin:password \
https://INDEXER_IP:9200/_cat/shards?v

UNASSIGNED

curl -k -u admin:password \
https://INDEXER_IP:9200/_cluster/allocation/explain?pretty