Elasticsearch/OpenSearch ClientKey Path Overwritten with Masked Value on Config Save (v10.11.15-v10.11.17)

Applies to: Mattermost Server v10.11.15, v10.11.16, and v10.11.17 with Elasticsearch/OpenSearch indexing enabled and a ClientKey file path configured

Symptoms: After saving Elasticsearch settings (via the System Console or mmctl config edit), the ClientKey field is rewritten to a literal ******************************** string. On the next restart, the affected node fails to load the TLS key, falls back to PostgreSQL for search, and post search returns intermittent or empty results.

🛑 Problem

In v10.11.15 the ClientKey field was incorrectly added to the System Console masking logic alongside actual secrets, even though it is only a file path. When an administrator saves Elasticsearch settings, the masked placeholder is written back as the literal value. On the next Mattermost restart the affected node logs Opensearch.createClient: ent.elasticsearch.create_client.client_key_missing, The specified key does not exist., OpenSearch becomes unreachable, and post searches silently fall back to the PostgreSQL search replica. Because PostgreSQL search can hit recovery conflicts on a streaming replica, the same search returns results some of the time and empty results other times, which appears to users as intermittent search behavior. The ClientCert field is unaffected because it was never in the masking list.

Symptoms

error [...] Opensearch.createClient: ent.elasticsearch.create_client.client_key_missing, The specified key does not exist. caller="platform/searchengine.go:15"
warn  [...] Encountered error indexing post caller="searchlayer/post_layer.go:31" search_engine=opensearch error="Opensearch.IndexPost: , "
warn  [...] Error flushing live indexing buffer caller="opensearch/bulk.go:137"

Additional symptoms:

• Searches for terms containing hyphens (for example, DIS-24787) return no results because the PostgreSQL fallback treats the hyphen as an exclusion operator
• The same search query returns inconsistent results across attempts (some hits, some empty)
• System Console > Environment > Elasticsearch > Test Connection returns Test Failure: Could not open the client key file for opensearch
• The persisted ClientKey value in the config is ******************************** instead of the original file path

✅ Solution

The fix is shipped in Mattermost v10.11.18 (released 2026-05-21). The changelog entry for that release records "Fixed a regression with saving various masked fields from the System Console." Upgrade to v10.11.18 or later, then restore the ClientKey path to its correct value.

Upgrade Mattermost to v10.11.18 or later

1. Upgrade the Mattermost binary on every node in the cluster to v10.11.18 or later.
2. Restart Mattermost on each node.
3. In System Console > Environment > Elasticsearch, set Client Certificate Key path back to the correct relative or absolute path (for example, ./elasticsearch/<host>.key).
4. Click Test Connection and confirm it returns success.
5. Save the configuration.

⚠️ Important: If you applied a workaround on an earlier version (for example, uploading the key file with the literal masked filename ******************************** to the configured file store path), remove that workaround after upgrading and confirming the correct key path is saved. The release also includes security fixes, so plan the upgrade accordingly.

Verify post search has recovered

1. Run a post search for a known term that previously failed (for example, a Jira-style ticket key containing a hyphen) and confirm results are returned.
2. Tail the Mattermost logs for one app node and confirm the client_key_missing error and Encountered error indexing post warnings are no longer present.
3. If you want to recover posts that were created on the affected node during the outage window, run a bulk reindex via System Console > Environment > Elasticsearch > Index Now. Do not purge the existing index before reindexing; a reindex without purge overwrites in place and keeps search functional during the run.

If you must change config before upgrading

When using mmctl to change Elasticsearch settings on an affected version, avoid mmctl config edit for these fields because the editor displays masked placeholders that get saved back as literals. Use a patch file that explicitly includes the correct ClientKey value alongside any other change, for example:

{
  "ElasticsearchSettings": {
    "Trace": "",
    "ClientKey": "./elasticsearch/<host>.key"
  }
}

Then apply it with mmctl config patch patch.json.

Additional Resources

For more information, see:

Mattermost v10.11.18 release notes

Fix PR for the masked field regression