Fixing Wazuh Cluster Desync Caused by Master-Worker Time Drift

A healthy Wazuh cluster relies on every node maintaining a consistent view of the environment. Whether you’re running a small deployment with a single master and a few workers or a large enterprise cluster spanning multiple data centers, synchronization between nodes is critical.

When that synchronization breaks down, administrators may notice missing agent data, inconsistent configurations, delayed alerts, or cluster health warnings.

One of the most overlooked causes of these issues is master-worker time drift. Even a relatively small difference in system clocks can interfere with how cluster nodes validate synchronization events, exchange files, and coordinate scheduled tasks.

Because many cluster operations depend on timestamps, inaccurate system time can gradually cause nodes to fall out of sync until the cluster reports errors or stops functioning correctly.

If left unresolved, Wazuh Cluster Desync can reduce detection accuracy, delay security monitoring, and complicate troubleshooting efforts.

Fortunately, identifying and correcting time drift is usually straightforward once you understand how cluster synchronization works.

In this guide, you’ll learn:

  • What Wazuh Cluster Desync means
  • How Wazuh master and worker nodes synchronize
  • Why accurate system time is essential for cluster stability
  • How master-worker time drift occurs
  • The most common causes of clock drift in Wazuh deployments
  • How to diagnose, fix, and prevent synchronization problems

Understanding Wazuh Cluster Synchronization

 

What Is the Wazuh Cluster?

A Wazuh cluster is a group of Wazuh manager nodes working together as a single logical deployment.

Instead of relying on one manager to handle every connected agent, clustering distributes workloads across multiple systems while maintaining a centralized configuration.

This architecture provides several important benefits:

  • High availability
  • Improved scalability
  • Load balancing
  • Fault tolerance
  • Easier management of large environments

The cluster consists of one master node and one or more worker nodes, each performing different responsibilities while continuously exchanging information.

Purpose of Clustering in Wazuh

Clustering allows organizations to monitor thousands, or even hundreds of thousands, of endpoints without overwhelming a single server.

The master node serves as the authoritative source for cluster configuration, while worker nodes process agent communications and forward synchronized information throughout the cluster.

This architecture helps organizations:

  • Scale endpoint monitoring horizontally
  • Reduce single points of failure
  • Maintain centralized management
  • Improve overall platform resilience

Organizations running distributed infrastructure or geographically separated environments especially benefit from clustering because workloads can be shared across multiple servers while maintaining a unified management interface.

Master and Worker Node Responsibilities

The master node acts as the central coordinator.

Its responsibilities include:

  • Maintaining cluster metadata
  • Synchronizing configuration files
  • Managing agent registration information
  • Coordinating synchronization tasks
  • Validating cluster integrity

Worker nodes primarily:

  • Receive agent events
  • Process log data
  • Execute local analysis
  • Synchronize changes back to the master
  • Keep local copies of required configuration files

Workers depend on regular communication with the master to ensure that every node is operating with identical configuration and current cluster information.

How Synchronization Keeps Nodes Consistent

Synchronization ensures that every node shares the same operational state.

Examples include:

  • Shared agent information
  • Decoder updates
  • Ruleset changes
  • Configuration files
  • Internal metadata
  • Cluster health information

Without continuous synchronization, worker nodes gradually diverge from the master, leading to inconsistent monitoring results and unexpected behavior across the deployment.

How Cluster Synchronization Works

 

File Synchronization

The master continuously synchronizes critical files with worker nodes.

These commonly include:

  • Rules
  • Decoders
  • Configuration files
  • Shared data
  • Internal cluster metadata

Rather than copying everything constantly, Wazuh compares file states and transfers only required changes, reducing unnecessary network traffic.

Agent Information Replication

Agent registration information must remain identical across every node.

This includes:

  • Agent IDs
  • Authentication keys
  • Connection status
  • Metadata
  • Inventory information

Accurate replication prevents workers from maintaining conflicting records for the same endpoint.

Related Guide: Why Is client.keys File Empty? Restoring Lost Wazuh Agents

Configuration Updates

Whenever administrators modify cluster configuration, new settings must propagate to worker nodes.

Examples include:

  • New decoders
  • Updated rules
  • Active Response settings
  • Monitoring configuration
  • Cluster parameters

Synchronization ensures every worker begins using identical configurations.

Related Guide: How to Configure Wazuh Active Response

API Communication Between Nodes

Internal cluster communication also occurs through authenticated API exchanges.

These communications coordinate:

  • Synchronization requests
  • Health reporting
  • Status updates
  • Cluster membership
  • Node validation

Reliable communication depends on consistent timestamps between participating systems.

Why Accurate Time Matters

 

Timestamp Validation

Nearly every synchronization event uses timestamps to determine whether information is newer or older than existing data.

If one node’s clock is significantly ahead or behind another, valid updates may appear outdated and be ignored.

Synchronization Windows

Cluster synchronization occurs within expected timing windows.

When clocks drift outside acceptable tolerances, synchronization jobs may fail because events appear to occur outside their expected execution periods.

This often produces intermittent synchronization failures that become increasingly frequent over time.

Event Ordering

Security events are processed chronologically.

Accurate clocks allow Wazuh to correctly determine:

  • Event sequence
  • Alert timing
  • Correlation logic
  • Rule execution order

Time drift can reorder events, making investigations significantly more difficult.

Authentication and Certificate Validation

Certificates and authentication mechanisms depend heavily on correct system time.

Large clock differences may cause:

  • Certificate validation failures
  • Authentication errors
  • Secure connection failures
  • Cluster communication interruptions

Time synchronization is considered a fundamental security best practice by organizations such as NIST.

Scheduled Synchronization Tasks

Many internal maintenance operations execute according to scheduled intervals.

Examples include:

  • File synchronization
  • Integrity verification
  • Cluster health checks
  • Cleanup operations
  • Status reporting

When clocks differ between nodes, these scheduled operations may execute earlier or later than expected, eventually contributing to cluster desynchronization.


What Is Master-Worker Time Drift?

 

Definition of Time Drift

Time drift refers to the gradual divergence between the clocks of different systems.

In a Wazuh cluster, this means the master node and one or more worker nodes no longer report the same current time.

Although differences may begin as only a few milliseconds or seconds, they often increase over time if clocks are not regularly synchronized using a reliable time source.

Clock Skew Between Cluster Nodes

Clock skew is the measurable difference between the current time reported by two systems.

For example:

  • Master: 14:00:00
  • Worker: 13:59:45

A 15-second skew may already be enough to interfere with synchronization depending on the operation being performed.

Small differences often go unnoticed initially but can accumulate into larger synchronization problems.

Gradual Drift Versus Sudden Time Changes

Not all time drift occurs the same way.

Gradual drift usually results from normal hardware clock inaccuracies slowly accumulating over hours or days.

Sudden drift is typically caused by:

  • Manual time adjustments
  • Hypervisor synchronization
  • VM resume from snapshots
  • Improper NTP corrections
  • System restores

Sudden jumps are generally more disruptive because they immediately invalidate timestamp assumptions used throughout the cluster.


How Time Drift Causes Cluster Desync

 

Missed Synchronization Windows

Synchronization jobs expect participating nodes to execute within predictable timing intervals.

When clocks differ significantly, workers may attempt synchronization too early or too late, causing jobs to fail or be skipped altogether.

Conflicting Timestamps

Cluster components often compare timestamps to determine which copy of a file or configuration is most recent.

Incorrect clocks can cause:

  • Older files appearing newer
  • New configurations being ignored
  • Conflicting updates
  • Synchronization loops

Delayed File Replication

Workers may postpone accepting replicated files if timestamps suggest those files are outdated.

Over time this leads to increasing configuration differences between cluster members.

Failed Integrity Checks

Cluster integrity checks verify that synchronized files match expected values.

When timestamps no longer align properly, these validation routines may incorrectly identify legitimate files as inconsistent.

This often results in recurring cluster synchronization warnings.

Inconsistent Cluster State

Eventually, multiple synchronization failures create inconsistent cluster state.

Possible symptoms include:

  • Different rules on different workers
  • Missing agent information
  • Dashboard inconsistencies
  • Synchronization warnings
  • Cluster health alerts
  • Unexpected replication failures

Common Causes of Time Drift

 

NTP Not Installed

The most common cause of time drift is simply not running a Network Time Protocol (NTP) client.

Without periodic synchronization, system clocks naturally drift over time.

NTP Service Stopped

Installing NTP alone is not enough.

If the synchronization service crashes, becomes disabled, or fails during boot, clocks begin drifting almost immediately.

Administrators should routinely verify that the service remains active.

Firewall Blocking NTP

Many organizations restrict outbound traffic.

If UDP port 123 is blocked, systems cannot reach upstream NTP servers and eventually lose synchronization.

Virtual Machine Clock Issues

Virtual machines often experience clock drift more frequently than physical servers.

Heavy CPU contention, suspended VMs, live migration, and snapshot restores can all affect guest operating system time.

Hypervisor Time Synchronization Conflicts

Running both guest NTP synchronization and hypervisor-based time synchronization simultaneously can cause competing clock adjustments.

Major virtualization vendors recommend using a single authoritative time synchronization strategy to avoid oscillating clocks.

Incorrect Timezone Configuration

Timezone settings do not usually affect the underlying UTC clock used for synchronization, but inconsistent timezone configurations can complicate troubleshooting by making timestamps appear inconsistent across logs.

Administrators should standardize timezone settings wherever practical.

Manual Clock Changes

Manually changing the system clock on either the master or worker node can immediately introduce significant synchronization problems.

Production cluster nodes should obtain time exclusively from trusted NTP or sources rather than manual adjustments.

Hardware Clock Problems

Although less common, failing CMOS batteries, faulty real-time clocks (RTC), or hardware timing issues can prevent systems from maintaining accurate time between synchronizations.

These hardware problems are more frequently encountered on older physical servers than in modern virtualized environments.


Symptoms of Wazuh Cluster Desync

Cluster desynchronization does not always result in a complete cluster failure.

In many cases, the first signs are subtle inconsistencies that gradually become more severe as the master and worker nodes drift further apart.

Recognizing these symptoms early can help prevent configuration mismatches, missed security events, and prolonged troubleshooting.

Worker Nodes Show as Disconnected

One of the most obvious symptoms is that worker nodes appear disconnected or unavailable, even though the servers themselves are online and reachable.

You may observe:

  • Worker nodes disappearing from cluster status
  • Nodes repeatedly reconnecting
  • Intermittent communication failures
  • Cluster status alternating between healthy and degraded

If network connectivity has already been ruled out, inaccurate system time should be one of the first things you verify.

Cluster Health Reports Errors

The cluster health check may report synchronization failures or degraded cluster status.

Common messages include:

  • Cluster synchronization failures
  • Node integrity mismatches
  • Synchronization timeout errors
  • Failed cluster integrity checks

These warnings indicate that one or more workers are no longer maintaining a consistent state with the master.

Agent Data Appears Inconsistent

Another common symptom is inconsistent agent information across cluster nodes.

Examples include:

  • Different agent counts between nodes
  • Missing agents on one worker
  • Stale inventory information
  • Different vulnerability scan results
  • Delayed event visibility

This inconsistency occurs because agent metadata is no longer replicating correctly throughout the cluster.

Related Guide: Resolving Duplicate Name or IP Errors in Wazuh Agent Registration

Rules and Configuration Changes Do Not Propagate

Administrators may successfully update rules or configuration files on the master, only to discover that workers continue using outdated versions.

You might notice:

  • Custom rules only working on one node
  • Decoder updates not taking effect
  • Active Response configuration differences
  • Newly added configuration files missing from workers

This usually indicates that file synchronization has stopped functioning correctly.

Related Guide: How to Create Custom Detection Rules in Wazuh (With Examples)

API Responses Differ Between Nodes

If you query different cluster nodes directly through the Wazuh API, responses may not match.

Examples include:

  • Different cluster health information
  • Different agent lists
  • Missing configuration updates
  • Different node status information

Since every node should maintain the same cluster state, differing API responses are often a strong indicator of desynchronization.

Frequent Cluster Synchronization Warnings

The Wazuh logs may repeatedly report synchronization problems.

Common warning patterns include:

  • Synchronization retries
  • Integrity verification failures
  • Replication errors
  • Timeout messages
  • Cluster communication warnings

These warnings often appear long before a worker becomes completely disconnected, making them valuable early warning indicators.

Unexpected Authentication or Certificate Errors

Time drift can also affect secure communication between cluster nodes.

Symptoms may include:

  • TLS handshake failures
  • Certificate validation errors
  • Authentication failures
  • Secure API connection errors

Because certificates depend on accurate timestamps for their validity periods, even moderate clock skew can interrupt secure cluster communication.

Related Guide: How to Fix Wazuh Certificate Errors


How to Diagnose Cluster Desynchronization

Successfully resolving Wazuh Cluster Desync begins with confirming whether the issue is actually caused by time drift.

The following diagnostic steps help determine whether cluster synchronization is failing because of clock differences or another underlying problem.

Check Cluster Status

Start by verifying the overall health of the cluster.

Depending on your deployment, you can use the Wazuh cluster management commands to display:

  • Master node status
  • Connected worker nodes
  • Synchronization status
  • Node health
  • Cluster membership

Pay close attention to workers reported as disconnected, degraded, or repeatedly reconnecting.

Using Cluster Management Commands

Review the reported state of each cluster node and verify that:

  • Every expected worker appears in the cluster
  • Only one master node exists
  • No duplicate nodes are listed
  • Synchronization reports are healthy

Unexpected node states often indicate deeper synchronization issues.

Verifying Connected Nodes

Compare the cluster status with your actual infrastructure.

Ensure that:

  • Every running worker appears in the cluster
  • Offline nodes are expected
  • No workers repeatedly disappear and reappear

Intermittent connectivity combined with synchronization warnings frequently points toward time synchronization problems.

Verify System Time on Every Node

After confirming the cluster status, compare the clocks on every node.

Even small differences can eventually cause synchronization failures.

Linux date Command

Begin with the simplest check:

date

Run the command on the master and every worker.

The displayed time should be nearly identical.

timedatectl

For more detailed information:

timedatectl

This displays:

  • Current local time
  • UTC time
  • Timezone
  • NTP synchronization status
  • RTC configuration

It is often the quickest way to determine whether a server is synchronized.

Comparing UTC Timestamps

Always compare systems using UTC rather than local time.

UTC eliminates confusion caused by:

  • Timezone differences
  • Daylight Saving Time
  • Regional clock settings

Production clusters are generally easier to maintain when every node uses UTC.

Check NTP Synchronization Status

Next, verify that each server is actively synchronizing with a reliable time source.

chronyc tracking

If using Chrony:

chronyc tracking

Review values such as:

  • System time offset
  • Last synchronization
  • Estimated accuracy
  • Reference server

Large offsets suggest the node is drifting.

chronyc sources

To view active time sources:

chronyc sources

Verify that:

  • Servers are reachable
  • At least one source is selected
  • Synchronization is active

ntpq -p

For systems running the classic NTP daemon:

ntpq -p

Confirm that:

  • Upstream servers are reachable
  • One server is selected as the synchronization source
  • Delay and offset values remain reasonable

timedatectl status

On systems using systemd:

timedatectl status

Pay particular attention to whether NTP synchronization is reported as active.

Examine Wazuh Cluster Logs

The Wazuh logs often provide direct evidence of synchronization failures.

manager.log

Review the manager log for:

  • Communication failures
  • Authentication errors
  • Cluster warnings
  • Replication problems

These entries frequently reveal the first indication of cluster instability.

cluster.log

The dedicated cluster log is especially useful.

Look for entries involving:

  • Synchronization failures
  • Integrity check errors
  • Timeout messages
  • Worker disconnects
  • Retry attempts

Synchronization Error Messages

Repeated synchronization failures are more significant than isolated warnings.

If the same messages appear continuously, the underlying cause, such as clock drift, should be investigated immediately.

Related Guide: Troubleshooting Wazuh Manager Core Dumps (if log analysis reveals broader manager stability issues)

Compare File Timestamps

Cluster synchronization relies heavily on timestamps when determining whether files need to be replicated.

Compare modification times for critical files across the master and workers.

Shared Configuration Files

Ensure configuration files have matching modification times wherever appropriate.

Unexpected differences may indicate failed synchronization.

Rules

Compare custom rule files between nodes.

Differences suggest replication has stopped or is incomplete.

Related Guide: How to Test Wazuh Rules

Decoders

Custom decoders should also remain identical throughout the cluster.

If workers contain outdated decoder files, synchronization is likely failing.

Agent Information

Agent metadata should remain consistent across every node.

Missing or outdated records can indicate that replication is no longer functioning correctly.

Verify Timezone Configuration

Although Wazuh primarily relies on UTC timestamps internally, consistent timezone settings make troubleshooting much easier.

Verify that every node uses the expected timezone configuration.

Ensure Consistent Timezone Settings

Mixed timezone configurations can make log correlation unnecessarily difficult, especially when investigating synchronization events across multiple systems.

Prefer UTC in Production

Most enterprise deployments standardize on UTC because it:

  • Eliminates Daylight Saving Time issues
  • Simplifies log analysis
  • Makes timestamp comparisons straightforward
  • Reduces operational mistakes during troubleshooting

This recommendation aligns with operational guidance from organizations such as the Center for Internet Security, which advocates consistent time synchronization as part of secure system administration.


Fixing Wazuh Cluster Desync

Once you’ve confirmed that time drift is responsible for the cluster desynchronization, the solution is usually straightforward.

The goal is to ensure every node obtains accurate time from the same trusted source before allowing the cluster to resume normal synchronization.

Step 1: Identify the Node with Incorrect Time

Begin by comparing the clocks on every cluster node.

Determine:

  • Which node has drifted
  • The size of the clock difference
  • Whether multiple nodes are affected
  • Whether the master or worker is incorrect

Avoid changing multiple systems simultaneously until you’ve identified the source of the problem.

Step 2: Install an NTP Client

Every cluster node should run a reliable time synchronization service.

Popular options include:

Chrony

Chrony is the preferred choice for many modern Linux distributions because it:

  • Synchronizes quickly
  • Handles intermittent network connectivity well
  • Maintains excellent long-term accuracy
  • Performs well in virtual environments

systemd-timesyncd

Smaller deployments may use systemd-timesyncd, which provides lightweight NTP synchronization for systems using systemd.

Although simpler than Chrony, it is sufficient for many environments.

ntpd

The traditional NTP daemon remains widely supported and is still found in many existing enterprise deployments.

Regardless of which client you choose, every node should use the same synchronization strategy.

Step 3: Configure Reliable Time Servers

Choosing reliable upstream time sources is just as important as installing an NTP client.

Use:

  • Trusted public NTP pools
  • Internal enterprise time servers
  • GPS-backed organizational time servers
  • Authenticated enterprise time sources where available

Whenever possible, configure multiple servers to provide redundancy if one becomes unavailable.

Step 4: Force Time Synchronization

After configuring the time service, force an immediate synchronization so that clocks converge before restarting the cluster.

Depending on your chosen NTP implementation, this may involve restarting the synchronization service or using a command to perform an immediate clock correction.

Avoid manually adjusting the system clock unless absolutely necessary, as abrupt time changes can temporarily disrupt running services.

Step 5: Verify Clock Synchronization

Once synchronization completes, verify that every node now reports nearly identical UTC time.

Re-run:

  • date
  • timedatectl
  • chronyc tracking
  • chronyc sources
  • ntpq -p (if applicable)

Ideally, offsets should be measured in milliseconds rather than seconds.

Step 6: Restart Wazuh Manager Services

After the clocks are synchronized, restart the Wazuh manager services so cluster communications begin with a consistent time reference.

Once the services restart, monitor the logs for:

  • Successful worker connections
  • Synchronization completion
  • Integrity verification success
  • Cluster health improvements

Avoid restarting all nodes simultaneously in large production clusters unless your maintenance procedures specifically call for it.

Step 7: Confirm Cluster Synchronization Has Resumed

Finally, verify that normal cluster operation has resumed.

Check that:

  • All worker nodes appear connected
  • Cluster health reports no synchronization errors
  • Configuration changes replicate successfully
  • Agent information is consistent across nodes
  • Cluster logs no longer report synchronization warnings

Continue monitoring the cluster for several synchronization cycles to ensure the issue does not reappear.

If desynchronization returns after correcting the clocks, investigate other contributing factors such as unstable virtualization hosts, intermittent network connectivity, or storage-related replication issues.


Fixing Time Drift in Virtualized Environments

Virtualized environments introduce additional challenges for time synchronization.

Unlike physical servers, virtual machines rely on both their guest operating system and the underlying hypervisor to maintain accurate time.

If these mechanisms are not configured correctly, clock drift can occur even when NTP is installed and running.

The following recommendations help ensure stable time synchronization across common virtualization platforms.

VMware Time Synchronization

VMware virtual machines can synchronize time through both the guest operating system and VMware Tools.

While this provides flexibility, it can also introduce problems if both VMware Tools and an NTP client attempt to adjust the clock simultaneously.

For production Wazuh clusters:

  • Decide whether the hypervisor or the guest OS will be the authoritative time source.
  • If using Chrony or another NTP client inside the guest, consider disabling periodic VMware Tools time synchronization to avoid competing adjustments.
  • Verify that ESXi hosts themselves are synchronized with reliable NTP servers.

VMware recommends maintaining accurate host time before relying on guest synchronization.

Hyper-V Time Synchronization

Hyper-V provides an Integration Service that synchronizes guest clocks with the host.

This feature is useful for many workloads but can conflict with guest-based NTP synchronization.

For Wazuh deployments:

  • Ensure the Hyper-V host maintains accurate time.
  • Avoid frequent manual clock corrections on the host.
  • Determine whether Hyper-V synchronization or the guest NTP client should be responsible for maintaining time.

Using a single authoritative synchronization method minimizes unexpected clock adjustments.

KVM Virtual Machines

KVM guests generally rely on standard Linux time synchronization services such as Chrony or systemd-timesyncd.

Administrators should also verify:

  • The host clock is synchronized.
  • Guest virtual clocks are stable.
  • CPU scheduling delays are not causing excessive drift.
  • Virtual machines are not frequently suspended or restored from snapshots.

Monitoring time offsets after live migration or snapshot restoration is particularly important.

Cloud Instances (AWS, Azure, Google Cloud)

Cloud providers offer highly accurate time synchronization services that are optimized for their infrastructure.

When running Wazuh in cloud environments:

  • Use the provider’s recommended time synchronization service whenever possible.
  • Ensure security groups and firewalls allow NTP traffic if external servers are used.
  • Verify synchronization after instance restoration, scaling events, or image deployment.

Each cloud platform publishes best practices for maintaining accurate system time.

Avoid Conflicting Time Sources

One of the most common causes of recurring time drift is allowing multiple services to manage the system clock simultaneously.

Examples include:

  • Chrony and ntpd
  • Chrony and VMware Tools
  • Hypervisor synchronization plus guest synchronization
  • Multiple NTP daemons
  • Manual time adjustments alongside automated synchronization

Choose a single authoritative synchronization strategy for every node in the cluster.

After making configuration changes, verify that only one service is actively controlling the system clock.


Advanced Troubleshooting

If the clocks are synchronized but the cluster continues reporting desynchronization errors, another issue may be preventing successful communication or replication between nodes.

The following scenarios are among the most common.

Time Is Correct but Cluster Remains Desynchronized

If every node reports nearly identical UTC time, investigate other cluster components.

Possible causes include:

  • Network connectivity problems
  • Corrupted synchronized files
  • Cluster configuration mismatches
  • Authentication failures
  • Incomplete software upgrades
  • Storage issues

Verify that all nodes are running compatible Wazuh versions and identical cluster configurations.

Related Guide: How to Build a Wazuh Indexer Cluster

Firewall Blocking NTP Traffic

Even if the local clock currently appears correct, blocked NTP traffic prevents future synchronization.

Confirm that:

  • UDP port 123 is permitted.
  • Firewalls allow outbound NTP requests.
  • Internal NTP servers are reachable.
  • Security appliances are not filtering NTP traffic.

A temporary network interruption can eventually lead to noticeable clock drift.

DNS Problems Preventing NTP Resolution

Many NTP configurations reference hostnames instead of IP addresses.

If DNS resolution fails, time synchronization may silently stop.

Verify:

  • DNS resolution works correctly.
  • Configured NTP hostnames resolve successfully.
  • Resolver configuration is consistent across all nodes.

Testing hostname resolution is often overlooked during troubleshooting.

Chrony Reports Unsynchronized

Chrony may report that the system is not synchronized even when the service is running.

Common reasons include:

  • No reachable time servers
  • Incorrect firewall rules
  • Invalid server configuration
  • Excessive initial clock drift
  • Network connectivity problems

Use chronyc tracking and chronyc sources together to determine why synchronization has failed.

Multiple NTP Services Running Simultaneously

Running more than one synchronization service often creates unstable clocks.

For example:

  • Chrony and ntpd
  • Chrony and systemd-timesyncd
  • Multiple legacy synchronization services

Verify active services using your operating system’s service management tools and disable unnecessary synchronization daemons.

Large Time Offset After Long Downtime

Servers that have been powered off for weeks or months may accumulate significant clock drift.

Large offsets can cause:

  • Slow synchronization
  • Temporary authentication failures
  • Certificate validation problems
  • Cluster synchronization delays

Allow sufficient time for the clock to stabilize before evaluating cluster health.

Cluster Still Reports Integrity Errors

Even after correcting time synchronization, integrity warnings may continue until the next successful synchronization cycle.

Review the cluster logs to determine whether integrity checks eventually succeed.

If they continue failing:

  • Verify configuration consistency.
  • Compare synchronized files.
  • Confirm worker connectivity.
  • Check for interrupted synchronization jobs.

Persistent integrity failures often indicate an additional replication issue unrelated to system time.

Persistent File Synchronization Failures

If configuration files still fail to synchronize after correcting clock drift, compare the synchronized directories directly.

Look for:

  • Missing files
  • Permission differences
  • Corrupted configuration
  • Ownership mismatches
  • Storage or filesystem problems

Review cluster.log for repeated replication failures and investigate any associated filesystem errors.

Related Guide: How to Fix ossec.conf Syntax Errors in Wazuh Agents


Preventing Future Cluster Desynchronization

Preventing time drift is considerably easier than recovering from a desynchronized cluster.

Establishing consistent operational practices helps ensure that every node maintains accurate time and remains synchronized over the long term.

Standardize on UTC

Using UTC across every cluster node eliminates many common sources of confusion.

Benefits include:

  • Consistent log timestamps
  • Easier event correlation
  • Simpler troubleshooting
  • No Daylight Saving Time adjustments
  • Uniform timestamps across multiple geographic regions

Although local time can still be displayed in dashboards, maintaining servers in UTC simplifies administration.

Use Chrony Instead of Legacy NTP

For most modern Linux distributions, Chrony offers several advantages over traditional ntpd.

These include:

  • Faster synchronization
  • Better handling of intermittent connectivity
  • Improved accuracy in virtual environments
  • Lower resource usage
  • More responsive clock correction

Organizations deploying new Wazuh clusters should strongly consider Chrony unless another synchronization solution is already standardized.

Configure Multiple Reliable Time Sources

Avoid depending on a single NTP server.

Configure multiple trusted sources to improve resilience if one server becomes unavailable.

A typical configuration includes:

  • Primary enterprise time server
  • Secondary enterprise server
  • Additional trusted public servers
  • Geographic redundancy where appropriate

Redundant sources reduce the likelihood of synchronization interruptions.

Monitor Clock Drift Automatically

Rather than waiting for synchronization failures, proactively monitor system time.

Track:

  • Clock offset
  • Last synchronization time
  • NTP server availability
  • Synchronization status
  • Time source changes

Many infrastructure monitoring platforms can alert administrators before clock drift becomes severe enough to impact the cluster.

Alert on Unsynchronized Nodes

Automated alerts allow administrators to respond before synchronization failures affect production.

Useful alert conditions include:

  • NTP service stopped
  • Excessive clock offset
  • Unsynchronized Chrony status
  • Unreachable time servers
  • Repeated synchronization failures

Early notification significantly reduces the likelihood of prolonged cluster desynchronization.

Keep Wazuh Components Updated

Newer Wazuh releases frequently include improvements to clustering, synchronization, stability, and bug fixes.

Keep the following components updated together whenever possible:

  • Wazuh Manager
  • Wazuh Indexer
  • Wazuh Dashboard
  • Wazuh Agents

Avoid running mixed software versions for extended periods.

Related Guide: How to Upgrade a Wazuh Agent

Regularly Verify Cluster Health

Cluster health should be reviewed as part of routine operational maintenance.

Periodically verify:

  • Worker connectivity
  • Synchronization status
  • Cluster integrity
  • Replication success
  • Log health
  • Time synchronization status

Regular health checks help identify developing issues long before they affect production workloads.

Document Time Synchronization Settings

Finally, document the cluster’s time synchronization configuration so administrators can quickly verify or rebuild systems when necessary.

Your documentation should include:

  • Selected NTP client
  • Configured time servers
  • Firewall requirements
  • Timezone configuration
  • Hypervisor synchronization settings
  • Standard operating procedures for new nodes
  • Troubleshooting steps for synchronization failures

Well-maintained documentation reduces configuration drift over time and helps ensure that future cluster expansions follow the same synchronization standards.


Best Practices for Managing Wazuh Clusters

Maintaining a healthy Wazuh cluster requires more than simply fixing synchronization issues when they occur.

Following a consistent set of operational practices helps minimize the risk of cluster desynchronization, reduces downtime, and improves the overall reliability of your deployment.

Maintain Consistent Operating System Versions

Running different operating system versions across cluster nodes can introduce subtle compatibility differences in system libraries, time synchronization services, and networking behavior.

Whenever possible:

  • Use the same Linux distribution across all nodes.
  • Keep operating system versions aligned.
  • Apply security patches consistently.
  • Upgrade nodes using a documented maintenance process.

Standardized environments simplify troubleshooting and reduce unexpected behavior.

Synchronize Configuration Changes Carefully

Configuration changes should always originate from the master node and be allowed to propagate naturally through the cluster.

Avoid:

  • Editing configuration files directly on worker nodes.
  • Applying different rule sets to different servers.
  • Making simultaneous manual changes on multiple nodes.

After making changes, verify that synchronization completes successfully before introducing additional modifications.

Related Guides: How to Configure Wazuh Active Response

Perform Routine Cluster Health Checks

Regular health checks help detect problems before they affect production monitoring.

Include the following in your maintenance schedule:

  • Verify connected worker nodes.
  • Review cluster synchronization status.
  • Check for integrity warnings.
  • Confirm file replication.
  • Review recent cluster log entries.

Routine validation allows administrators to identify developing issues before they become service-impacting.

Validate Time After Reboots

Although most modern systems automatically synchronize their clocks during startup, this should never be assumed.

After rebooting any cluster node:

  • Confirm the system clock is synchronized.
  • Verify the NTP service is running.
  • Check that the node has rejoined the cluster successfully.
  • Review synchronization logs for errors.

This simple validation can prevent prolonged troubleshooting later.

Test Disaster Recovery Procedures

Disaster recovery plans should include cluster synchronization verification.

Periodically test scenarios such as:

  • Restoring virtual machine snapshots.
  • Recovering failed worker nodes.
  • Promoting replacement systems.
  • Restoring configuration backups.
  • Rejoining rebuilt nodes to the cluster.

Testing ensures recovery procedures do not accidentally introduce clock drift or configuration inconsistencies.

Monitor Cluster Logs Continuously

Cluster logs often provide early warning signs of synchronization issues.

Monitor for recurring:

  • Synchronization failures
  • Integrity check warnings
  • Worker disconnects
  • Authentication failures
  • Replication retries

Centralized log monitoring and alerting can help administrators detect these problems long before users notice missing data.

Related Guide: How to Configure Wazuh as a Centralized Syslog Server

Avoid Manual System Clock Changes

Production servers should never rely on manual clock adjustments during normal operation.

Instead:

  • Use a trusted NTP service.
  • Allow gradual synchronization where appropriate.
  • Investigate the underlying cause of clock drift instead of repeatedly correcting the clock manually.
  • Apply consistent time synchronization policies across every node.

Manual time changes can temporarily invalidate certificates, disrupt scheduled tasks, and interfere with cluster synchronization.


Frequently Asked Questions (FAQ)

Question: What causes Wazuh Cluster Desync?

The most common causes include:

  • System clock drift between nodes
  • NTP synchronization failures
  • Network connectivity problems
  • File replication issues
  • Cluster configuration mismatches
  • Authentication or certificate failures

Although time drift is a frequent cause, administrators should also verify networking, storage, and configuration consistency before concluding that clock synchronization is solely responsible.

Question: How much time drift can Wazuh tolerate?

There is no single published threshold that guarantees synchronization will continue operating normally.

Even relatively small clock differences can affect timestamp comparisons, scheduled synchronization tasks, certificate validation, and event ordering.

For best results, keep all cluster nodes synchronized within a few milliseconds by using a reliable NTP solution.

Question: Should every Wazuh node use the same NTP server?

Every node should synchronize with trusted and consistent time sources.

Many organizations configure:

  • Multiple internal enterprise NTP servers
  • Redundant public NTP servers
  • Provider-managed cloud time services

Using identical or closely synchronized time sources helps ensure consistent timestamps throughout the cluster.

Question: Can different timezones cause cluster problems?

Different timezones typically do not affect cluster synchronization because Wazuh relies on UTC timestamps internally.

However, inconsistent timezone settings can:

  • Complicate log analysis
  • Make troubleshooting more difficult
  • Create confusion when comparing timestamps across servers

For production deployments, configuring all servers to use UTC is considered a best practice.

Question: Does restarting the Wazuh Manager fix cluster desynchronization?

Restarting the Wazuh Manager may temporarily restore communication, but it does not resolve the underlying cause if the system clocks remain inaccurate.

Always correct the time synchronization issue first, then restart services if necessary to re-establish normal cluster communication.

Question: How do I verify that Chrony is synchronized?

Chrony provides several useful commands.

The most common are:

chronyc tracking

and

chronyc sources

These commands display:

  • Current synchronization status
  • Active time source
  • Estimated clock accuracy
  • System offset
  • Reachability of configured servers

A healthy Chrony configuration should report that the system is synchronized with a reliable source.

Question: Can virtualization cause Wazuh Cluster Desync?

Yes.

Virtual environments can introduce clock drift due to:

  • Host synchronization issues
  • Snapshot restoration
  • Live migration
  • CPU scheduling delays
  • Conflicting guest and hypervisor time synchronization

Following your virtualization platform’s recommended time synchronization practices greatly reduces these risks.

Question: How often should I monitor clock synchronization?

Critical production clusters should monitor synchronization continuously through automated health checks and alerting.

At a minimum, administrators should verify:

  • NTP synchronization status
  • Clock offset
  • Cluster health
  • Worker connectivity
  • Synchronization logs

as part of regular operational maintenance. Detecting clock drift early is far easier than recovering from a fully desynchronized cluster.


Conclusion

Accurate system time is one of the foundational requirements for a healthy Wazuh cluster.

Because master and worker nodes rely on timestamps to coordinate synchronization, replicate configuration changes, validate certificates, and maintain a consistent cluster state, even modest clock drift can lead to Wazuh Cluster Desync and a variety of operational problems.

Fortunately, diagnosing these issues follows a logical process.

Start by checking the cluster’s health, compare the system time across every node, verify that NTP synchronization is functioning correctly, and review the Wazuh logs for synchronization or integrity errors.

Once the clocks are synchronized, confirm that file replication resumes normally and that every worker returns to a healthy state.

Long-term stability depends on prevention rather than recovery.

Standardizing on UTC, using a modern synchronization service such as Chrony, configuring redundant time sources, monitoring clock drift automatically, and performing routine cluster health checks can dramatically reduce the likelihood of future desynchronization issues.

By combining reliable time synchronization with consistent operational practices, you can keep your Wazuh cluster synchronized, resilient, and ready to deliver accurate security monitoring across your entire infrastructure.

Recommended related guides:

Be First to Comment

    Leave a Reply

    Your email address will not be published. Required fields are marked *