Bulk indexing fails with HTTP 403 after migrating from Elasticsearch to OpenSearch

Adit Yadav

Updated September 25, 2026 04:45

Applies to: Mattermost Server self-hosted in a multi-node or High Availability deployment with a dedicated job server, after switching the search backend from Elasticsearch to AWS OpenSearch. Any server version that supports the opensearch backend.

Symptoms: Bulk indexing runs slowly or makes no progress, the searchable document count on the new OpenSearch domain barely increases, and the indexer logs HTTP 403 responses on bulk flushes.

Confirmation status: Customer-confirmed

🛑 Problem

Search indexing jobs do not run on every node. They run on the node or nodes with the job server enabled (JobSettings.RunJobs). In a multi-node deployment, that is frequently not the node an administrator is looking at when they change the search configuration or read the logs.

If the job server has not been given the new backend configuration, or is running an older Mattermost version that predates support for it, it continues writing to the previous Elasticsearch endpoint. Those writes are rejected with HTTP 403, typically for an authorization or request-signing reason, since the old domain's access policy no longer matches. Meanwhile the other nodes connect to the new OpenSearch domain successfully for read traffic, which produces a misleading mix of 200 responses to the new endpoint and 403 responses to the old one in the same log file.

The result looks like a slow or broken index rather than a misconfigured node, because the job reports activity while almost nothing is being written.

Symptoms

error [2026-09-17 17:13:59.074 +09:00] Error from elasticsearch bulk indexer caller="elasticsearch/indexing_job.go:51" worker_name=EnterpriseElasticsearchIndexer error="flush: [403 Forbidden] {"message":"Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header. (Hashed with SHA-256 and encoded with Base64) Authorization=xxxxxxxxxxxxxx="}"

Bulk index POSTs go to the previous search domain hostname, not the new one.
Other requests in the same logs reach the new domain and return 200.
The searchable document count on the new domain grows very slowly or not at all.
The nodes differ in Mattermost version, or in their effective search configuration.

✅ Solution

Get the job server onto the same version and the same search configuration as the rest of the cluster, then purge and reindex in the correct order.

Step 1: Identify the node actually running the job

Find the node with JobSettings.RunJobs set to true. This is the node whose configuration and version matter for indexing. Read its logs, not the logs of a node that only serves requests.

Step 2: Stop the running index job

Stop the in-progress bulk index before changing anything, so that the job is not writing while the configuration changes underneath it.

Step 3: Align the job server's version and configuration

Bring the job server to the same Mattermost version as the rest of the cluster. Then confirm, on that node:

ElasticsearchSettings.Backend = opensearch
ElasticsearchSettings.ConnectionUrl = <new OpenSearch endpoint>

Restart the job server after these are correct.

⚠️ Important: Mattermost supports at most one minor version difference between nodes, and only during a rolling upgrade. Running different versions in a cluster outside an upgrade is not supported and produces exactly this class of intermittent, node-dependent failure.

Step 4: Purge, then index

In System Console > Environment > Elasticsearch, select Purge Indexes, then Index Now. Purging before bulk indexing is required every time.

⚠️ Important: Purge only after ConnectionUrl points at the new endpoint. Purging while the configuration still points at the old domain purges the wrong indexes.

Step 5: Avoid a search outage during the reindex

Set Enable Elasticsearch for search queries to false before bulk indexing so that database search continues to serve users, and let the bulk index finish completely before turning it back on. Enabling search queries against a partially built index returns incomplete results.