How to Configure Wazuh Cross Cluster Search

As Wazuh deployments grow, organizations often expand beyond a single cluster. Large enterprises may deploy separate Wazuh Indexer clusters for different geographic regions, business units, cloud environments, or compliance boundaries.

While this architecture improves scalability and data isolation, it also introduces a new challenge: searching and analyzing security data across multiple clusters.

Without a centralized search capability, security analysts must switch between clusters to investigate incidents, correlate events, and hunt for threats.

This fragmented workflow slows investigations and increases the risk of overlooking important indicators of compromise.

Wazuh Cross Cluster Search (CCS) solves this problem by allowing a single Wazuh Indexer cluster to search indexes stored on one or more remote clusters.

Because Wazuh Indexer is built on OpenSearch, it inherits OpenSearch’s Cross-Cluster Search functionality, enabling organizations to query distributed security data without replicating indexes or merging environments.

In this guide, you’ll learn how Wazuh Cross Cluster Search works, its architecture, prerequisites, configuration steps, security considerations, performance optimization techniques, troubleshooting methods, and best practices for operating CCS in production environments.


What Is Wazuh Cross Cluster Search?

Wazuh Cross Cluster Search (CCS) enables one Wazuh Indexer cluster to search indexes located on other Wazuh Indexer clusters as though they were part of a single environment.

Instead of copying security data into one central cluster, CCS sends search requests from a local cluster to one or more remote clusters.

Each remote cluster executes the query against its local indexes and returns only the matching results.

The coordinating cluster combines these responses into a unified search result for the user.

Because CCS searches data where it already resides, organizations avoid the storage overhead and synchronization challenges associated with duplicating indexes across environments.

This capability is particularly valuable for organizations operating multiple Wazuh deployments across different data centers, cloud providers, or geographic regions.

Definition of Cross Cluster Search (CCS)

Cross Cluster Search is an OpenSearch feature that allows a cluster to execute search requests against indexes stored on remote OpenSearch clusters.

Rather than replicating data, CCS establishes trusted connections between clusters.

Queries are securely forwarded to remote clusters, executed locally, and the resulting documents are returned to the coordinating cluster.

This architecture provides:

  • Unified searches across multiple clusters
  • Minimal additional storage requirements
  • Near real-time access to remote data
  • Centralized threat hunting without data duplication

Since Wazuh Indexer uses OpenSearch as its search engine, administrators can leverage CCS using native OpenSearch capabilities.

How CCS Works in Wazuh Indexer (OpenSearch)

Wazuh Indexer acts as the search engine for all Wazuh alerts, archives, vulnerability data, inventory information, and other indexed security events.

When Cross Cluster Search is configured:

  1. The local Wazuh Indexer cluster registers one or more remote clusters.
  2. Each remote cluster is assigned a unique alias.
  3. A search request references the alias along with the target index.
  4. The local coordinating node forwards the query.
  5. Remote clusters execute the search locally.
  6. Matching documents are returned.
  7. The coordinating node merges and ranks all results before returning them to the client.

For example:

GET us-east:wazuh-alerts-*/_search

Or search multiple clusters simultaneously:

GET us-east:wazuh-alerts-*,eu-west:wazuh-alerts-*/_search

This approach allows analysts to investigate incidents across globally distributed Wazuh environments using a single query.

Difference Between Local and Remote Cluster Searches

A standard Wazuh search operates only on indexes stored within the local cluster.

Cross Cluster Search expands this capability by forwarding searches to remote clusters while presenting the results as one combined dataset.

Local SearchCross Cluster Search
Searches one clusterSearches multiple clusters
Lowest latencySlight network overhead
No remote connections requiredRequires trusted remote cluster configuration
Limited visibilityOrganization-wide visibility
Simpler architectureSupports distributed deployments

Local searches remain the fastest option for day-to-day operations, while CCS provides broader visibility for enterprise-wide investigations.

Common Use Cases for CCS

Cross Cluster Search is commonly deployed in enterprise Wazuh environments where security data is distributed across multiple infrastructures.

Typical use cases include:

  • Searching alerts across regional SOC deployments
  • Investigating attacks spanning multiple cloud providers
  • Monitoring subsidiaries from a centralized security team
  • Keeping regulated data within specific jurisdictions while enabling centralized searches
  • Maintaining separate production, staging, and development Wazuh clusters
  • Supporting mergers and acquisitions where separate Wazuh deployments must remain independent
  • Performing enterprise-wide threat hunting across multiple business units

Organizations following zero-trust or data sovereignty requirements often prefer CCS because security data remains inside each region while still being searchable centrally.


Benefits of Wazuh Cross Cluster Search

Cross Cluster Search provides organizations with a scalable way to investigate threats across multiple Wazuh deployments without consolidating all indexed data into a single cluster.

Instead of increasing storage costs through data replication, CCS enables federated searches while allowing each cluster to operate independently.

For distributed Security Operations Centers (SOCs), managed security providers, and multinational enterprises, this significantly improves visibility and investigation speed.

Search Security Events Across Multiple Clusters

One of the biggest advantages of CCS is the ability to search security events stored across several Wazuh Indexer clusters with a single query.

Security analysts no longer need to:

  • Log into multiple dashboards
  • Repeat identical searches
  • Export results from different clusters
  • Manually correlate findings

Instead, CCS retrieves matching events from every configured remote cluster and combines them into one unified response.

This dramatically accelerates threat hunting, incident response, and forensic investigations.

Centralize Threat Investigations

Many organizations operate independent Wazuh environments for operational reasons, but their SOC still needs centralized visibility.

Cross Cluster Search enables analysts to investigate:

  • Malware outbreaks
  • Insider threats
  • Lateral movement
  • Credential compromise
  • Distributed attacks

from a single search interface.

Rather than consolidating all security data into one massive cluster, organizations maintain decentralized storage while centralizing investigations.

This aligns with recommendations from the MITRE ATT&CK framework, which emphasizes correlating activity across multiple systems to identify adversary behavior.

Maintain Separate Clusters for Compliance or Regions

Many regulations require organizations to keep security data within specific geographic regions.

Examples include:

  • GDPR
  • HIPAA
  • Government security frameworks
  • Financial regulations

Cross Cluster Search allows organizations to:

  • Keep European logs in European data centers
  • Keep North American logs locally
  • Separate cloud and on-premises environments
  • Isolate regulated business units

while still enabling centralized searches during investigations.

This provides a balance between compliance requirements and operational efficiency.

Reduce Operational Complexity

Without CCS, organizations often replicate indexes into centralized clusters.

That introduces additional complexity such as:

  • Replication pipelines
  • Storage duplication
  • Synchronization failures
  • Additional maintenance
  • Larger storage requirements

CCS avoids these problems because data remains in its original location.

Searches occur only when needed, reducing administrative overhead while simplifying infrastructure.

Support Large-Scale Distributed Environments

Large enterprises frequently deploy multiple Wazuh clusters because a single cluster cannot efficiently support every location worldwide.

Cross Cluster Search scales naturally for environments containing:

  • Multiple regions
  • Multiple cloud providers
  • Independent business units
  • Disaster recovery clusters
  • Managed customer environments

Rather than building one extremely large cluster, organizations can divide workloads while maintaining enterprise-wide search capabilities.

For additional guidance on scaling Wazuh infrastructure, see Step-by-Step Wazuh Manager Scaling Guide and The Complete Wazuh Cluster Architecture Guide.


How Wazuh Cross Cluster Search Works

Cross Cluster Search uses OpenSearch’s distributed search architecture to execute queries across multiple Wazuh Indexer clusters while presenting results as though they originated from a single environment.

Although the process appears simple to users, several components work together behind the scenes to route requests, authenticate clusters, execute searches, and aggregate results.

Local Cluster vs. Remote Clusters

A CCS deployment consists of:

  • Local cluster — receives search requests and coordinates execution
  • Remote clusters — store indexes and execute searches locally

The local cluster does not permanently store remote indexes.

Instead, it acts as the coordinator that knows how to contact trusted remote clusters whenever a search requires their data.

For example:

Local Cluster
      │
      ├────────► Remote Cluster A
      │
      ├────────► Remote Cluster B
      │
      └────────► Remote Cluster C

Each remote cluster continues operating independently even when participating in CCS.

Coordinating Nodes

A coordinating node receives the user’s search request and manages the entire search lifecycle.

Its responsibilities include:

  • Parsing the query
  • Determining which clusters contain the requested indexes
  • Forwarding requests
  • Waiting for remote responses
  • Merging search results
  • Sorting documents
  • Returning the final response

The coordinating node does not perform the search itself unless it also stores local shards.

In large environments, organizations often dedicate coordinating nodes to improve search performance and reduce resource usage on data nodes.

Cluster Aliases

Every remote cluster is registered using a unique alias.

Example:

us-east
eu-west
ap-southeast

Queries reference these aliases instead of IP addresses.

Example:

GET eu-west:wazuh-alerts-*/_search

Searching multiple clusters simultaneously:

GET us-east:wazuh-alerts-*,eu-west:wazuh-alerts-*/_search

Aliases simplify administration because cluster names remain consistent even if infrastructure changes.

Query Execution Process

A Cross Cluster Search generally follows these steps:

  1. An analyst submits a search query.
  2. The coordinating node identifies the referenced cluster aliases.
  3. The query is securely forwarded to each remote cluster.
  4. Each remote cluster executes the search against its local shards.
  5. Matching documents are returned to the coordinating node.
  6. Results from every cluster are merged.
  7. The final ranked response is returned to the client.

Because searches execute directly where the data resides, organizations avoid transferring entire indexes across the network.

According to OpenSearch documentation, this distributed execution model minimizes unnecessary data movement and helps maintain scalability across multiple clusters.

Authentication and Secure Communication

Since CCS enables communication between independent clusters, secure authentication is essential.

Best practices include:

  • Enable TLS encryption between clusters.
  • Use mutual certificate authentication where supported.
  • Restrict remote cluster access to trusted hosts.
  • Apply least-privilege permissions for search operations.
  • Limit firewall rules to required transport ports.
  • Monitor remote cluster connections for unusual activity.

Proper certificate management and transport-layer encryption help protect search traffic from interception or unauthorized access.

If you’re configuring secure communication between Wazuh components, see How to Fix Wazuh Certificate Errors for additional guidance.


Prerequisites

Before configuring Wazuh Cross Cluster Search (CCS), verify that both the local and remote Wazuh Indexer clusters meet the necessary software, networking, and security requirements.

Skipping these prerequisites can lead to failed cluster registrations, authentication errors, or unreliable search results.

Supported Wazuh and Wazuh Indexer Versions

Cross Cluster Search depends on the underlying OpenSearch version used by Wazuh Indexer.

As a best practice:

  • Use the same Wazuh version across all clusters whenever possible.
  • Ensure all Wazuh Indexer nodes run compatible OpenSearch versions.
  • Avoid mixing major OpenSearch versions unless explicitly supported.
  • Upgrade clusters before enabling CCS if significant version differences exist.

Running mismatched versions can result in unsupported APIs, incompatible search features, or serialization errors during remote queries.

Always verify compatibility before connecting production clusters.

Healthy Wazuh Indexer Clusters

Each participating cluster should be healthy before enabling Cross Cluster Search.

Verify that:

  • Cluster health is green whenever possible.
  • All primary shards are assigned.
  • Replica shards are healthy.
  • No nodes are disconnected.
  • No indexes are in a read-only state.
  • Disk usage remains below flood-stage watermarks.

A degraded remote cluster can still affect CCS queries by increasing latency or returning incomplete search results.

If your cluster is unhealthy, consider resolving those issues first by following How to Fix a Yellow Cluster Status in Wazuh Indexer or How to Fix Wazuh All Shards Failed Error.

Network Connectivity Between Clusters

All participating clusters must communicate over the network.

Verify:

  • Bidirectional connectivity (if required by your deployment)
  • Low network latency
  • Stable routing
  • No packet filtering between clusters
  • Reliable WAN connectivity for geographically distributed environments

Before configuring CCS, test connectivity using utilities such as:

ping remote-cluster.example.com

or

nc -zv remote-cluster.example.com 9300

If network connectivity is unstable, Cross Cluster Search performance will suffer regardless of cluster health.

Required Firewall Ports

Firewalls must allow transport-layer communication between participating clusters.

Typical required ports include:

PortPurpose
9300OpenSearch transport protocol (cluster-to-cluster communication)
9200REST API (optional for administration)

Only expose required ports to trusted systems.

Avoid allowing unrestricted inbound access from the Internet.

Firewall policies should follow the principle of least privilege.

TLS Certificates Configured

Production deployments should always encrypt communication between clusters.

Recommended practices include:

  • Enable TLS on transport traffic.
  • Use certificates signed by a trusted Certificate Authority (CA).
  • Validate certificate chains.
  • Rotate certificates before expiration.
  • Protect private keys.

Incorrect certificates are among the most common causes of failed remote cluster registration.

If you’re experiencing certificate-related issues, see How to Fix Wazuh Certificate Errors.

Administrative Privileges

Configuring Cross Cluster Search requires administrative access to Wazuh Indexer.

You’ll typically need permissions to:

  • Register remote clusters
  • Modify persistent cluster settings
  • Configure security roles
  • Manage certificates
  • Verify cluster health
  • Access cluster APIs

Users with read-only privileges generally cannot configure CCS.

DNS or Hostname Resolution

Clusters should resolve each other’s hostnames consistently.

Verify:

  • Forward DNS resolution
  • Reverse DNS resolution (recommended)
  • Hostname consistency across all nodes
  • No conflicting host entries

Test resolution before configuring CCS:

nslookup remote-cluster.example.com

or

host remote-cluster.example.com

Reliable hostname resolution simplifies administration and avoids future connectivity problems if IP addresses change.


Wazuh Cross Cluster Search Architecture

Cross Cluster Search follows a distributed architecture in which one Wazuh Indexer cluster coordinates search requests while one or more remote clusters execute searches against their local indexes.

Unlike cross-cluster replication, CCS does not move or synchronize data.

Each cluster remains autonomous, storing and managing its own indexes while responding to remote search requests when needed.

Single Coordinating Cluster

The coordinating cluster serves as the entry point for all CCS queries.

Its responsibilities include:

  • Receiving user search requests
  • Identifying the appropriate remote clusters
  • Forwarding search requests
  • Receiving responses
  • Aggregating results
  • Returning a unified response

Importantly, the coordinating cluster does not need to store copies of remote indexes.

Instead, it maintains metadata describing trusted remote clusters and their aliases.

Multiple Remote Clusters

One coordinating cluster can communicate with multiple remote Wazuh Indexer clusters simultaneously.

Example:

  • North America cluster
  • Europe cluster
  • Asia-Pacific cluster
  • Disaster recovery cluster

Each remote cluster:

  • Stores its own indexes
  • Maintains independent cluster health
  • Executes searches locally
  • Returns only matching documents

Adding additional clusters generally requires registering a new remote cluster alias rather than redesigning the architecture.

Data Flow During Searches

A Cross Cluster Search request typically follows this workflow:

  1. A user submits a search query.
  2. The coordinating cluster parses the request.
  3. Remote cluster aliases are identified.
  4. Search requests are forwarded to each remote cluster.
  5. Remote clusters search their local shards.
  6. Matching results are returned.
  7. The coordinating node merges and sorts all responses.
  8. A single result set is presented to the user.

Because only search results traverse the network, not entire indexes, bandwidth consumption remains relatively low.

Example Deployment Diagram

A typical production architecture looks like this:

                    +------------------------+
                    |  Wazuh Dashboard       |
                    +-----------+------------+
                                |
                                |
                    +-----------v------------+
                    | Local Wazuh Indexer    |
                    | Coordinating Cluster   |
                    +-----------+------------+
                                |
          ------------------------------------------------
          |                      |                       |
+---------v---------+  +---------v---------+  +----------v---------+
| Remote Cluster A  |  | Remote Cluster B  |  | Remote Cluster C   |
| North America     |  | Europe            |  | Asia-Pacific       |
+-------------------+  +-------------------+  +--------------------+

Each remote cluster remains operational even if another remote cluster becomes unavailable.

Best Practices for Production Deployments

To maximize reliability and performance:

  • Keep clusters running compatible software versions.
  • Deploy dedicated coordinating nodes in large environments.
  • Use descriptive cluster aliases.
  • Enable TLS for all transport traffic.
  • Monitor network latency between regions.
  • Avoid searching unnecessary indexes.
  • Use wildcard patterns carefully.
  • Monitor JVM heap utilization.
  • Regularly review cluster health.
  • Test CCS after upgrades.

Organizations operating large distributed environments should also implement sound shard planning to minimize search overhead.

For additional guidance, see How to Design a Wazuh OpenSearch Shard Allocation Strategy and How to Manage Wazuh Storage.


How to Configure Wazuh Cross Cluster Search

Once the prerequisites are complete, you can configure Cross Cluster Search by registering remote clusters, validating connectivity, assigning security permissions, and testing distributed queries.

The following example uses the OpenSearch REST API supported by Wazuh Indexer.

Step 1: Verify Cluster Health

Before connecting clusters, ensure each Wazuh Indexer cluster is healthy.

Check Cluster Status

Run:

curl -k -u admin:password https://localhost:9200/_cluster/health?pretty

Example output:

{
  "cluster_name": "wazuh-cluster",
  "status": "green",
  "number_of_nodes": 3,
  "active_primary_shards": 62,
  "active_shards": 124
}

Ideally, every participating cluster should report a green status.

Verify Node Communication

Confirm all expected nodes are participating:

curl -k -u admin:password https://localhost:9200/_cat/nodes?v

Review:

  • Node names
  • IP addresses
  • Roles
  • Heap utilization
  • CPU usage
  • Node availability

Resolve missing or disconnected nodes before proceeding.

Step 2: Configure Remote Cluster Settings

The coordinating cluster must know how to reach each remote cluster.

Add Remote Cluster Aliases

Register a remote cluster using the Cluster Settings API:

curl -k -u admin:password \
-X PUT "https://localhost:9200/_cluster/settings" \
-H "Content-Type: application/json" \
-d '
{
  "persistent": {
    "cluster": {
      "remote": {
        "us-east": {
          "seeds": [
            "10.10.1.15:9300"
          ]
        }
      }
    }
  }
}'

In this example:

  • us-east is the cluster alias.
  • The seeds list contains one or more transport-layer endpoints that the coordinating cluster uses to discover the remote cluster.

Configure Seed Nodes

For production deployments:

  • Specify multiple seed nodes for redundancy.
  • Use dedicated transport addresses.
  • Ensure each seed node is reachable.
  • Avoid relying on a single node.

Example:

"seeds": [
  "10.10.1.15:9300",
  "10.10.1.16:9300",
  "10.10.1.17:9300"
]

Persistent vs. Transient Settings

OpenSearch supports two cluster configuration types:

Setting TypePurpose
PersistentSurvives cluster restarts and should be used for production CCS configurations.
TransientTemporary settings that are cleared after a restart and are best suited for testing or troubleshooting.

In most production environments, persistent settings are recommended.

Step 3: Verify Remote Cluster Connectivity

After registering the remote cluster, confirm that communication is working correctly.

Confirm Remote Cluster Registration

Retrieve the configured settings:

curl -k -u admin:password \
https://localhost:9200/_cluster/settings?pretty

Verify that the remote cluster alias appears under the cluster.remote configuration.

Validate Node Connections

Review remote cluster information:

curl -k -u admin:password \
https://localhost:9200/_remote/info?pretty

A healthy response should show:

  • Cluster alias
  • Connection status
  • Connected nodes
  • Seed nodes
  • Connection mode

If the cluster reports connected: false, investigate network connectivity, firewall rules, TLS configuration, or transport port accessibility.

Step 4: Configure Security Permissions

Cross Cluster Search should only be accessible to authorized users.

Required OpenSearch Security Roles

Administrators should create or assign roles that allow users to:

  • Search remote indexes
  • Read index metadata
  • Execute distributed search requests

Avoid granting unnecessary cluster administration privileges to analysts who only need search access.

User Permissions

Assign CCS privileges using role-based access control (RBAC).

Users should have access only to the indexes required for their responsibilities.

Restricting access minimizes the risk of exposing sensitive security data from unrelated business units or regions.

If you’re using custom RBAC configurations, see Troubleshooting Wazuh RBAC.

Cross-Cluster Authentication

Each participating cluster must trust the identity of the others.

Best practices include:

  • Use mutual TLS authentication.
  • Validate certificate chains.
  • Rotate certificates regularly.
  • Store private keys securely.
  • Audit authentication failures.

These measures help ensure that only trusted clusters can participate in Cross Cluster Search.

Step 5: Test Cross Cluster Search Queries

Once connectivity and security are configured, verify that searches return the expected results.

Query Syntax

CCS queries reference the remote cluster alias followed by the target index pattern:

cluster_alias:index_name/_search

Searching a Single Remote Cluster

Example:

GET us-east:wazuh-alerts-*/_search

This query searches only the wazuh-alerts-* indexes stored in the us-east remote cluster.

Searching Multiple Clusters Simultaneously

Search two remote clusters with a single request:

GET us-east:wazuh-alerts-*,eu-west:wazuh-alerts-*/_search

The coordinating cluster forwards the query to both remote clusters and merges the returned results.

Combining Local and Remote Indexes

You can also search local and remote indexes in the same request.

Example:

GET wazuh-alerts-*,us-east:wazuh-alerts-*/_search

This is particularly useful during enterprise-wide investigations where recent local events need to be correlated with historical or geographically distributed data.

Step 6: Validate Search Performance

After confirming functionality, evaluate how CCS performs under normal workloads.

Measure Query Latency

Compare:

  • Local search response times
  • Remote search response times
  • Multi-cluster search latency

Higher latency may indicate network congestion, overloaded remote clusters, or inefficient query patterns.

Confirm Expected Search Results

Verify that:

  • All expected remote indexes are searched.
  • Document counts match expectations.
  • Results from each cluster are included.
  • Filters and aggregations behave correctly.
  • No shard failures are reported.

Testing several query types helps ensure that CCS is functioning correctly across different investigation scenarios.

Review Cluster Logs

Finally, examine Wazuh Indexer and OpenSearch logs for warnings or errors related to Cross Cluster Search.

Look for messages involving:

  • Remote cluster connectivity
  • TLS handshake failures
  • Authentication or authorization issues
  • Shard failures
  • Query timeouts
  • Transport-layer communication

Routine log reviews can identify configuration issues early, helping maintain reliable and secure cross-cluster search operations.


Common Cross Cluster Search Commands

Once Cross Cluster Search (CCS) is configured, you’ll frequently use a handful of OpenSearch APIs to manage remote clusters, verify connectivity, and troubleshoot search issues.

These commands allow administrators to inspect the current configuration, add or remove remote clusters, and validate that searches are working correctly.

View Remote Cluster Configuration

To display all configured remote clusters:

curl -k -u admin:password \
https://localhost:9200/_remote/info?pretty

Example output:

{
  "us-east": {
    "connected": true,
    "mode": "sniff",
    "num_nodes_connected": 3,
    "max_connections_per_cluster": 3,
    "initial_connect_timeout": "30s"
  },
  "eu-west": {
    "connected": true,
    "mode": "sniff",
    "num_nodes_connected": 3
  }
}

Review the following values:

  • Cluster alias
  • Connection status
  • Number of connected nodes
  • Connection mode
  • Connection timeout

A value of "connected": true confirms that the coordinating cluster can communicate with the remote cluster.

Add a Remote Cluster

Register a new remote cluster using the Cluster Settings API.

curl -k -u admin:password \
-X PUT https://localhost:9200/_cluster/settings \
-H "Content-Type: application/json" \
-d '
{
  "persistent": {
    "cluster": {
      "remote": {
        "asia": {
          "seeds": [
            "10.30.1.15:9300",
            "10.30.1.16:9300"
          ]
        }
      }
    }
  }
}'

This configuration:

  • Creates the alias asia
  • Registers two seed nodes
  • Stores the configuration as a persistent cluster setting

Using multiple seed nodes improves resiliency if one node becomes unavailable.

Update Remote Cluster Settings

You can modify an existing remote cluster by updating its seed nodes.

Example:

curl -k -u admin:password \
-X PUT https://localhost:9200/_cluster/settings \
-H "Content-Type: application/json" \
-d '
{
  "persistent": {
    "cluster": {
      "remote": {
        "asia": {
          "seeds": [
            "10.30.1.20:9300",
            "10.30.1.21:9300",
            "10.30.1.22:9300"
          ]
        }
      }
    }
  }
}'

After updating the configuration, verify connectivity using the _remote/info API.

Remove a Remote Cluster

To remove a remote cluster:

curl -k -u admin:password \
-X PUT https://localhost:9200/_cluster/settings \
-H "Content-Type: application/json" \
-d '
{
  "persistent": {
    "cluster": {
      "remote": {
        "asia": null
      }
    }
  }
}'

Removing a remote cluster immediately prevents new Cross Cluster Search requests from targeting that cluster.

Check Cluster Health

Before troubleshooting CCS, confirm that the coordinating cluster is healthy.

curl -k -u admin:password \
https://localhost:9200/_cluster/health?pretty

A healthy cluster should ideally report:

"status": "green"

If the cluster reports a yellow or red status, investigate those issues before diagnosing CCS problems.

For additional guidance, see How to Fix a Yellow Cluster Status in Wazuh Indexer.

Test Search Queries

Verify that Cross Cluster Search is working correctly.

1. Search a single remote cluster:

GET us-east:wazuh-alerts-*/_search

2. Search multiple clusters:

GET us-east:wazuh-alerts-*,eu-west:wazuh-alerts-*/_search

3. Search local and remote indexes together:

GET wazuh-alerts-*,us-east:wazuh-alerts-*/_search

Successful queries confirm that remote cluster registration, networking, and security settings are functioning correctly.

Inspect Connected Nodes

Display all nodes in the local cluster:

curl -k -u admin:password \
https://localhost:9200/_cat/nodes?v

Review:

  • Node roles
  • Heap usage
  • CPU utilization
  • Node names
  • IP addresses

This command helps determine whether overloaded coordinating nodes could be affecting Cross Cluster Search performance.


Understanding Cross Cluster Search Query Syntax

Cross Cluster Search extends standard OpenSearch queries by introducing cluster aliases.

Instead of searching only local indexes, you specify which remote cluster contains the data you want to search.

Understanding the syntax allows you to build efficient searches that span one or many Wazuh Indexer clusters.

Cluster Alias Format

Every remote cluster is referenced using its configured alias.

General format:

cluster_alias:index_name

Example:

us-east:wazuh-alerts-*

Here:

  • us-east is the remote cluster alias.
  • wazuh-alerts-* is the target index pattern.

Searching One Index

To search a specific index on a remote cluster:

GET us-east:wazuh-alerts-4.x-2026.07.26/_search

This limits the search to a single index, making it more efficient than searching all indexes.

Searching Multiple Indexes

You can target several indexes on the same remote cluster.

Example:

GET us-east:wazuh-alerts-*,us-east:wazuh-archives-*/_search

This searches both alert and archive indexes stored on the us-east cluster.

Wildcard Searches

Wildcards simplify searches across many similarly named indexes.

Example:

GET us-east:wazuh-alerts-*/_search

Wildcards are especially useful for daily or monthly indexes.

However, broad wildcard searches may increase query latency because they touch more shards.

Multi-Cluster Searches

One of CCS’s greatest strengths is searching multiple remote clusters simultaneously.

Example:

GET us-east:wazuh-alerts-*,eu-west:wazuh-alerts-*,asia:wazuh-alerts-*/_search

The coordinating node sends the query to each remote cluster independently before combining the results.

This enables enterprise-wide threat hunting with a single request.

Local and Remote Searches in One Query

You can search local indexes alongside remote indexes.

Example:

GET wazuh-alerts-*,us-east:wazuh-alerts-*/_search

This is useful when:

  • Recent events are stored locally.
  • Historical data resides remotely.
  • Analysts need complete visibility without running multiple searches.

Filtering Results

Cross Cluster Search supports the same Query DSL used for local searches.

Example:

GET us-east:wazuh-alerts-*/_search
{
  "query": {
    "match": {
      "rule.level": 12
    }
  }
}

Additional filters may include:

  • Agent ID
  • Hostname
  • Event category
  • MITRE ATT&CK technique
  • Rule ID
  • Source IP
  • Username

Because filtering occurs on the remote cluster, only matching documents are returned, reducing network traffic.

Time-Based Searches

Security investigations frequently focus on specific time ranges.

Example:

GET us-east:wazuh-alerts-*/_search
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "now-24h",
        "lte": "now"
      }
    }
  }
}

Restricting searches to relevant time windows offers several benefits:

  • Faster response times
  • Reduced cluster workload
  • Lower network utilization
  • More focused investigation results

This aligns with OpenSearch best practices, which recommend limiting queries to only the data needed rather than scanning all historical indexes.


Performance Optimization Tips

Cross Cluster Search introduces additional network communication and distributed query processing.

While OpenSearch is designed to scale efficiently, following performance best practices helps maintain fast search response times as your Wazuh deployment grows.

Use Dedicated Coordinating Nodes

Large deployments benefit from dedicated coordinating nodes that handle search requests without storing data.

Advantages include:

  • Reduced CPU usage on data nodes
  • Better search throughput
  • Lower indexing contention
  • Improved query scalability

Separating coordination from data storage is especially beneficial in environments with frequent enterprise-wide searches.

Limit Wildcard Searches

Wildcard searches are convenient but can become expensive when they match hundreds or thousands of indexes.

Instead of:

wazuh-alerts-*

Prefer narrower patterns such as:

wazuh-alerts-2026.07*

or specific daily indexes whenever possible.

Reducing the number of queried indexes decreases shard activity and improves response times.

Optimize Shard Allocation

Search performance depends heavily on how indexes are distributed across the cluster.

Recommendations include:

  • Balance shards evenly across nodes.
  • Avoid oversized shards.
  • Avoid thousands of tiny shards.
  • Monitor shard allocation after node changes.
  • Regularly review disk utilization.

For more information, see How to Design a Wazuh OpenSearch Shard Allocation Strategy.

Reduce Network Latency

Cross Cluster Search performance depends on network quality.

To minimize latency:

  • Deploy clusters in regions with reliable connectivity.
  • Use high-bandwidth links between data centers.
  • Reduce unnecessary firewall inspection.
  • Monitor WAN performance.
  • Avoid routing CCS traffic through congested network paths.

Even small improvements in round-trip latency can noticeably reduce search response times across multiple remote clusters.

Search Only Required Indexes

Searching every available index increases:

  • CPU utilization
  • Memory usage
  • Disk activity
  • Network traffic
  • Query execution time

Whenever possible:

  • Specify index names directly.
  • Use date-based index patterns.
  • Restrict searches to relevant business units.
  • Limit time ranges.

Smaller search scopes generally produce faster results.

Tune JVM Heap Settings

Search-intensive workloads require adequate JVM memory.

Recommendations include:

  • Allocate approximately 50% of system RAM to the JVM (up to recommended limits).
  • Avoid excessive heap allocation that reduces filesystem cache.
  • Monitor garbage collection activity.
  • Watch for memory pressure during large distributed searches.

Improper heap sizing may increase garbage collection pauses and degrade search performance.

If you’re experiencing memory-related issues, see How to Tune OpenSearch Heap Size to Stop Wazuh High Memory Crashes.

Cache Frequent Queries

Repeated searches often benefit from OpenSearch’s query caching mechanisms.

Frequently executed dashboards, reports, and saved searches may return faster after caching.

To maximize cache effectiveness:

  • Use consistent query patterns.
  • Avoid unnecessary query variations.
  • Restrict searches to active indexes.
  • Monitor cache hit ratios.

Caching is particularly effective for dashboards that refresh at regular intervals with similar filters.

Monitor Search Performance

Performance optimization should be an ongoing process.

Regularly monitor:

  • Search latency
  • Query throughput
  • JVM heap usage
  • CPU utilization
  • Network latency
  • Shard allocation
  • Slow query logs
  • Search thread pools

OpenSearch’s Performance Analyzer and monitoring APIs can help identify bottlenecks before they impact users.

For broader guidance on scaling and optimizing Wazuh environments, see Step-by-Step Wazuh Manager Scaling Guide and How to Manage Wazuh Storage.


Security Best Practices

Because Cross Cluster Search allows one Wazuh Indexer cluster to access data stored on other clusters, it expands the trust boundary of your environment.

Proper security controls help prevent unauthorized access, protect sensitive security events, and maintain compliance with organizational and regulatory requirements.

Following these best practices will help ensure your Cross Cluster Search deployment remains secure without sacrificing operational efficiency.

Enable TLS Encryption

Always encrypt communication between coordinating and remote clusters using Transport Layer Security (TLS).

TLS provides:

  • Encryption of search traffic
  • Mutual authentication between clusters
  • Protection against man-in-the-middle attacks
  • Verification of trusted cluster identities

Use certificates issued by a trusted Certificate Authority (CA) and verify that every participating node trusts the same CA or an approved certificate chain.

Avoid disabling certificate verification in production environments, as doing so weakens the security of inter-cluster communication.

For more information on securing Wazuh communications, see How to Fix Wazuh Certificate Errors.

Restrict Remote Cluster Access

Only trusted Wazuh Indexer clusters should be permitted to establish Cross Cluster Search connections.

Consider the following recommendations:

  • Allow only approved cluster IP addresses.
  • Restrict transport-layer ports using firewall rules.
  • Block unnecessary inbound connections.
  • Separate management and production networks when possible.
  • Remove obsolete remote cluster configurations promptly.

Reducing the number of systems that can communicate with the coordinating cluster minimizes the attack surface.

Apply Least-Privilege Permissions

Not every administrator or analyst requires unrestricted access to every remote cluster.

Implement role-based access control (RBAC) so users receive only the permissions required to perform their responsibilities.

For example:

  • SOC analysts receive read-only search access.
  • Cluster administrators manage CCS configuration.
  • Security engineers maintain certificates and networking.
  • Service accounts receive only the permissions necessary for automation.

Restricting permissions helps limit the impact of compromised accounts and accidental configuration changes.

If you’re using custom security roles, see Troubleshooting Wazuh RBAC: Why Your Custom User Roles Aren’t Applying.

Rotate Certificates Regularly

Certificates should be rotated before expiration to reduce the risk of compromised credentials remaining valid.

Establish a certificate management process that includes:

  • Monitoring expiration dates
  • Renewing certificates before they expire
  • Revoking compromised certificates immediately
  • Protecting private keys
  • Validating certificate chains after renewal

Routine certificate rotation improves the long-term security of your Cross Cluster Search deployment.

Audit Cross-Cluster Queries

Search activity should be logged and reviewed regularly.

Audit logs can help identify:

  • Unauthorized searches
  • Excessive query activity
  • Access outside normal business hours
  • Failed authorization attempts
  • Suspicious investigation patterns

Regular auditing supports incident response, compliance reporting, and forensic investigations.

Limit Exposed Network Endpoints

Only the services required for Cross Cluster Search should be reachable between clusters.

Best practices include:

  • Expose only required transport ports.
  • Disable unused services.
  • Restrict management interfaces.
  • Use VPNs or private network links for inter-cluster communication.
  • Regularly review firewall rules.

Reducing exposed services decreases the likelihood of unauthorized access.

Monitor Authentication Failures

Authentication failures often provide early warning of configuration problems or attempted attacks.

Monitor logs for:

  • Invalid certificates
  • Failed TLS handshakes
  • Authentication failures
  • Unknown remote clusters
  • Repeated connection attempts
  • Expired credentials

Investigating these events promptly helps maintain a secure and reliable Cross Cluster Search environment.

The OpenSearch Security plugin documentation recommends enforcing strong authentication, encrypted transport communication, and role-based access controls for production deployments.


Common Cross Cluster Search Problems and Solutions

Even after a successful configuration, administrators may occasionally encounter connectivity, authentication, or performance issues with Cross Cluster Search.

Most problems stem from networking, security configuration, or cluster health rather than the search functionality itself.

The following sections cover the most common issues and how to resolve them.

Remote Cluster Not Connected

 

Causes

Common causes include:

  • Incorrect seed node configuration
  • Firewall blocking transport traffic
  • Remote cluster unavailable
  • DNS resolution failures
  • Transport port inaccessible
  • TLS handshake failures

Resolution

Verify that:

  • Seed node addresses are correct.
  • Transport port 9300 is reachable.
  • The remote cluster is healthy.
  • DNS resolution is functioning correctly.
  • TLS certificates are valid.
  • The _remote/info API reports "connected": true.

Search Requests Timeout

 

Causes

Timeouts commonly occur when:

  • Network latency is high.
  • Queries search too many indexes.
  • Remote clusters are overloaded.
  • Shards are heavily utilized.
  • Large wildcard searches are executed.

Resolution

Reduce query scope by:

  • Limiting time ranges.
  • Searching fewer indexes.
  • Optimizing shard allocation.
  • Improving network connectivity.
  • Increasing search timeout values when appropriate.
  • Monitoring coordinating node resource usage.

TLS Certificate Validation Errors

 

Causes

Certificate validation failures often result from:

  • Expired certificates
  • Incorrect certificate chains
  • Untrusted Certificate Authorities
  • Hostname mismatches
  • Missing intermediate certificates

Resolution

Verify that:

  • Certificates are valid.
  • Certificate chains are complete.
  • Hostnames match certificate Subject Alternative Names (SANs).
  • All nodes trust the issuing CA.
  • Certificates have not expired.

If certificate issues persist, review How to Fix Wazuh Certificate Errors.

Authentication Failed

 

Causes

Authentication failures may occur because of:

  • Incorrect security roles
  • Invalid user credentials
  • Misconfigured authentication backend
  • Missing permissions
  • Certificate trust failures

Resolution

Check:

  • Assigned OpenSearch roles.
  • User permissions.
  • Authentication backend configuration.
  • Security plugin logs.
  • TLS trust relationships.

Grant only the permissions required for Cross Cluster Search operations.

No Such Remote Cluster

 

Causes

This error usually indicates that:

  • The cluster alias is misspelled.
  • The remote cluster has not been registered.
  • The configuration was removed.
  • Persistent settings were not saved correctly.

Resolution

Verify the configured aliases:

curl -k -u admin:password \
https://localhost:9200/_remote/info?pretty

Confirm that:

  • The alias exists.
  • The spelling matches the query.
  • The cluster reports a connected status.

Partial Search Results

 

Causes

Partial results may occur when:

  • One remote cluster becomes unavailable.
  • Some shards fail.
  • Network interruptions occur.
  • Query execution exceeds timeout limits.

Resolution

Review:

  • Cluster health
  • Shard allocation
  • Search response warnings
  • Transport logs
  • Remote cluster availability

If shard-related errors appear, see How to Fix Wazuh All Shards Failed Error.

High Search Latency

 

Causes

High latency is often caused by:

  • Slow WAN connections
  • Excessive wildcard searches
  • Large shard counts
  • Overloaded coordinating nodes
  • High JVM memory pressure
  • Resource contention on remote clusters

Resolution

Improve performance by:

  • Using dedicated coordinating nodes.
  • Optimizing shard allocation.
  • Searching smaller index ranges.
  • Reducing wildcard usage.
  • Monitoring JVM heap utilization.
  • Improving network connectivity between regions.

For additional optimization guidance, see How to Design a Wazuh OpenSearch Shard Allocation Strategy and How to Tune OpenSearch Heap Size to Stop Wazuh High Memory Crashes.

Remote Index Not Found

 

Causes

This error commonly occurs when:

  • The index name is incorrect.
  • The index has been deleted.
  • Index State Management (ISM) has removed the index.
  • The wrong cluster alias is specified.

Resolution

Verify:

  • The remote cluster alias.
  • Index names.
  • Index retention policies.
  • Available indexes using the Cat Indices API.

If indexes are automatically managed, review How to Configure Wazuh Index State Management (ISM).


Real-World Example

 

Scenario

A multinational financial services organization operates three independent Wazuh environments across North America, Europe, and Asia-Pacific.

Each region maintains its own three-node Wazuh Indexer cluster to satisfy local data residency regulations, minimize search latency, and isolate operational workloads.

Although regional security teams manage their own infrastructure, the organization’s central Security Operations Center (SOC) is responsible for investigating enterprise-wide threats.

Analysts frequently need to determine whether suspicious activity observed in one region also appears in another.

Without Cross Cluster Search, investigators must log into each Wazuh Dashboard separately, execute identical searches, export results, and manually correlate findings, adding significant time to incident response.

To streamline investigations, the administrators deploy a dedicated coordinating Wazuh Indexer cluster and configure Cross Cluster Search.

They register each regional cluster using descriptive aliases such as na, eu, and apac, establish secure TLS-encrypted transport connections, verify mutual certificate trust, configure firewall rules to permit only required transport traffic, and assign least-privilege OpenSearch security roles for SOC analysts.

Once the configuration is complete, analysts can execute a single Cross Cluster Search query that simultaneously searches all three regions.

During a ransomware investigation, for example, the SOC searches for a malicious file hash and discovers related alerts originating from endpoints in Europe and Asia-Pacific within seconds.

The combined search results enable analysts to correlate indicators of compromise, identify lateral movement across regions, determine the scope of the incident, and generate centralized reports, all while security data remains stored within its respective geographic region to satisfy compliance requirements.

This architecture improves investigation speed, reduces operational overhead, preserves regional autonomy, and allows the organization to scale its Wazuh deployment globally without replicating large volumes of indexed security data.


Best Practices for Managing Cross Cluster Search

A properly configured Wazuh Cross Cluster Search environment requires ongoing maintenance to remain secure, reliable, and performant.

As organizations add more clusters, increase security event volume, or expand into additional regions, proactive management becomes essential.

The following best practices help maintain stable communication between Wazuh Indexer clusters and ensure analysts can perform reliable distributed threat investigations.

Keep All Clusters on Compatible Versions

Maintaining compatible Wazuh and Wazuh Indexer versions across clusters reduces the risk of unexpected CCS failures.

Recommended practices include:

  • Upgrade clusters using a controlled rollout process.
  • Review OpenSearch compatibility before upgrades.
  • Test Cross Cluster Search functionality after version changes.
  • Avoid connecting clusters running unsupported major versions.

Version mismatches may cause:

  • Query compatibility issues
  • Authentication failures
  • API differences
  • Unexpected search behavior

Before upgrading production clusters, validate CCS functionality in a test environment.

For upgrade guidance, see How to Upgrade a Wazuh Agent and ensure your overall Wazuh deployment follows a controlled upgrade strategy.

Use Descriptive Cluster Aliases

Cluster aliases make CCS queries easier to understand and maintain.

Instead of generic names:

cluster1
cluster2
cluster3

use descriptive aliases:

north-america
europe
asia-pacific

Benefits include:

  • Easier query creation
  • Faster incident investigations
  • Reduced administrative mistakes
  • Better documentation

Clear aliases are especially important when analysts search multiple clusters during security investigations.

Monitor Inter-Cluster Network Latency

Network performance directly affects Cross Cluster Search response times.

Monitor:

  • Round-trip latency
  • Packet loss
  • Bandwidth utilization
  • Transport connection stability
  • Regional network performance

High latency between clusters can result in:

  • Slow searches
  • Query timeouts
  • Delayed dashboards
  • Poor analyst experience

For geographically distributed deployments, consider placing coordinating nodes closer to analysts while maintaining secure connections to remote clusters.

Regularly Validate Remote Connections

Remote cluster connectivity should be checked continuously rather than only when problems occur.

Regular validation should include:

curl -k -u admin:password \
https://localhost:9200/_remote/info?pretty

Confirm:

  • Remote clusters remain connected.
  • Expected nodes are available.
  • Authentication succeeds.
  • No transport errors exist.

Automated health checks can alert administrators before CCS failures impact investigations.

Limit Searches to Necessary Indexes

Broad searches across every cluster and index increase resource consumption.

Avoid unnecessary queries such as:

*:*

Instead, use targeted searches:

wazuh-alerts-2026.07*

or:

europe:wazuh-alerts-*

Limiting searches improves:

  • Query performance
  • Network efficiency
  • Cluster stability
  • Analyst response times

Maintain Consistent Index Naming Conventions

Consistent index naming makes CCS queries easier to manage.

For example, all clusters should follow similar patterns:

wazuh-alerts-*
wazuh-archives-*
wazuh-monitoring-*

Avoid situations where one cluster uses:

security-alerts-*

while another uses:

wazuh-events-*

Inconsistent naming creates confusion and increases the complexity of search queries.

Automate Configuration Backups

Cross Cluster Search configuration should be included in regular backup procedures.

Back up:

  • Remote cluster settings
  • Security role configurations
  • TLS certificates
  • Cluster configuration files
  • Index management policies

Automated backups help recover quickly after:

  • Hardware failures
  • Configuration mistakes
  • Cluster rebuilds
  • Disaster recovery events

For broader backup strategies, see How to Backup Wazuh Manager Configuration.

Monitor Cluster Health Continuously

CCS reliability depends on the health of every participating cluster.

Monitor:

  • Cluster status
  • Node availability
  • Disk usage
  • JVM memory
  • Shard allocation
  • Search latency
  • Failed queries

Useful APIs include:

GET /_cluster/health

and:

GET /_cat/nodes?v

Continuous monitoring allows administrators to detect problems before they affect threat investigations.


Frequently Asked Questions (FAQ)

Question: What Is Wazuh Cross Cluster Search?

Wazuh Cross Cluster Search is a capability that allows one Wazuh Indexer cluster to search security data stored across multiple remote Wazuh Indexer clusters.

Instead of logging into separate environments, analysts can execute queries from one location and receive combined results from multiple clusters.

Question: Does Cross Cluster Search Copy Data Between Clusters?

No.

Cross Cluster Search does not replicate or copy indexes between clusters.

Instead:

  1. A query is sent from the coordinating cluster.
  2. Remote clusters search their own local indexes.
  3. Matching results are returned.
  4. The coordinating cluster combines the responses.

This reduces storage requirements and allows each cluster to maintain independent ownership of its data.

Question: Can I Search Multiple Wazuh Indexer Clusters at Once?

Yes.

CCS allows administrators and analysts to query multiple clusters in a single request.

Example:

GET north-america:wazuh-alerts-*,europe:wazuh-alerts-*/_search

The coordinating cluster distributes the query across each configured remote cluster and aggregates the results.

Question: What Ports Are Required for Cross Cluster Search?

The primary port required for Wazuh Indexer Cross Cluster Search is:

PortPurpose
9300OpenSearch transport communication

Port 9200 may be required for administrative API access but is not used for normal cluster transport communication.

Firewall rules should restrict access to trusted Wazuh Indexer nodes only.

Question: How Do I Secure Cross Cluster Search?

Secure CCS by implementing:

  • TLS encryption between clusters
  • Trusted certificates
  • Restricted firewall rules
  • Role-based access control
  • Least-privilege permissions
  • Authentication monitoring
  • Regular certificate rotation

Security should be configured before connecting production clusters.

Question: Can Cross Cluster Search Impact Performance?

Yes.

CCS performance depends on:

  • Network latency
  • Number of searched clusters
  • Number of queried indexes
  • Shard distribution
  • Remote cluster workload
  • Coordinating node resources

To improve performance:

  • Search only required indexes.
  • Limit wildcard usage.
  • Optimize shards.
  • Use dedicated coordinating nodes.
  • Monitor query latency.

Question: Does Cross Cluster Search Work Across Different Wazuh Versions?

Cross Cluster Search requires compatible Wazuh Indexer and OpenSearch versions.

While minor version differences may work in some situations, production environments should keep clusters aligned whenever possible.

Before upgrading, verify compatibility and test CCS functionality.

Question: How Do I Troubleshoot Remote Cluster Connection Issues?

Start by checking remote cluster status:

curl -k -u admin:password \
https://localhost:9200/_remote/info?pretty

Then verify:

  1. Network connectivity.
  2. Firewall rules.
  3. Transport port availability.
  4. TLS certificates.
  5. Authentication permissions.
  6. Remote cluster configuration.

Common problems include:

  • Invalid seed nodes
  • Certificate validation errors
  • DNS failures
  • Missing permissions
  • Unhealthy remote clusters

For related troubleshooting, see:


Conclusion

Wazuh Cross Cluster Search provides a scalable way to investigate security events across multiple Wazuh Indexer clusters without requiring organizations to centralize or duplicate their data.

By connecting remote clusters through OpenSearch’s distributed search architecture, security teams can maintain separate regional or compliance-focused deployments while still achieving centralized threat visibility.

Configuring CCS involves several important steps:

  1. Verify the health of all Wazuh Indexer clusters.
  2. Configure remote cluster aliases and seed nodes.
  3. Establish secure TLS communication.
  4. Configure appropriate security permissions.
  5. Validate remote connectivity.
  6. Test distributed search queries.
  7. Monitor performance and reliability.

Security should remain a priority throughout the deployment lifecycle.

Encrypt inter-cluster communication, restrict access, apply least-privilege permissions, and regularly audit Cross Cluster Search activity.

Performance optimization is equally important.

Limiting searches to required indexes, maintaining efficient shard allocation, monitoring network latency, and tuning cluster resources help ensure fast and reliable investigations.

As Wazuh environments continue to expand across regions, cloud platforms, and business units, Cross Cluster Search provides a practical approach for enterprise-wide threat hunting while preserving data sovereignty and operational independence.

Administrators should regularly validate remote cluster connectivity, review cluster health, and monitor search performance to maintain a reliable distributed security monitoring architecture.

With proper configuration and ongoing maintenance, Wazuh Cross Cluster Search enables security teams to investigate threats faster and gain a unified view of activity across their entire organization.

Be First to Comment

    Leave a Reply

    Your email address will not be published. Required fields are marked *