Elasticsearch Startup Failure Causes Memory Leak and Repeated OOM Kills (v11.7+)

Marco Kundt

Updated August 06, 2026 16:35

Applies to: Mattermost Server v11.7.0 and later, with Elasticsearch indexing enabled (ElasticsearchSettings.EnableIndexing: true). OpenSearch deployments are not affected.

Status: Open, no fix available (as of 2026-08-06). The memory leak in the retry loop itself is an unpatched Mattermost defect; the steps below remove the trigger and stop the leak without waiting for an upstream fix.

Symptoms: Mattermost Server memory climbs continuously until the operating system kills the process; the server restarts and the cycle repeats.

🛑 Problem

On Elasticsearch deployments running Mattermost v11.7 or later, the server retries starting the search engine every 5 minutes when it fails to come up, and each failed retry leaks memory that is never released, regardless of why the retry failed. Memory grows continuously until the operating system kills the process, which restarts and repeats the cycle. The leak is in Mattermost's retry loop itself, so any condition that makes Elasticsearch reject the server's startup request, such as a missing required plugin, an account without index-template permission, or a disk-watermark block, triggers it.

Symptoms

Memory on affected nodes climbs continuously, independent of user or message load. CPU stays low.
Nodes are killed by the operating system and restart automatically. Memory resets on restart, then climbs again at the same rate.
Search results are stale or missing. With Database Search enabled (the default), search falls back to the database silently, so Elasticsearch can have been broken well before the upgrade that exposed it.
The support packet reports elastic.status: OK and Test Connection succeeds. Neither check covers the step that is failing, so neither rules this out.
Server logs show nothing about the search engine, even at DEBUG. Step 3 covers checking Elasticsearch directly instead of relying on Mattermost's logs.
OpenSearch deployments retry startup failures the same way but do not leak memory doing it; only Elasticsearch is affected.

✅ Solution

Fix the Elasticsearch startup failure. Memory only accumulates while the retry loop is running, so ending that loop stops the leak.

Step 1: Stop the leak now

System Console > Environment > Elasticsearch > Enable Elasticsearch Indexing, set to false.

"ElasticsearchSettings": { "EnableIndexing": false }

⚠️ Important: This takes effect immediately; no restart is required to stop new leaks. Memory already leaked is not released until affected nodes are restarted. Perform a rolling restart of all nodes afterward to reclaim it.

Step 2: Check for the `analysis-icu` plugin on every Elasticsearch node

Run both commands from a Mattermost node, against the endpoint configured in ElasticsearchSettings.ConnectionURL, so they use the same DNS, network path, and credentials as Mattermost. Omitting the password after --user makes curl prompt for it instead of leaving it in shell history.

curl --silent --show-error --fail-with-body --user elastic \
  '<connection-url>/_cat/nodes?v&h=name,ip,roles,version&s=name'

curl --silent --show-error --fail-with-body --user elastic \
  '<connection-url>/_cat/plugins?v&h=name,component,version&s=name,component'

The plugin listing covers the whole cluster, so there must be one analysis-icu row for every node returned by the node listing. A missing row, or an empty plugin response, means the plugin is absent from at least one node. Install it there and restart that node:

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-icu

Reference: Set up Elasticsearch

Step 3: If the plugin is already on every node, check Elasticsearch directly

Check Elasticsearch itself, without re-enabling indexing:

curl --silent --show-error --user elastic '<connection-url>/_cluster/health?pretty'

curl --silent --show-error --user elastic \
  '<connection-url>/_cluster/settings?pretty&include_defaults=true&filter_path=**.read_only*'

curl --silent --show-error --user elastic '<connection-url>/_index_template/<index-prefix>posts?pretty'

A _cluster/health status other than green, or true under any read_only/read_only_allow_delete key in _cluster/settings, means the cluster is blocking writes after crossing a disk watermark.
An existing _index_template/<prefix>posts response with mappings that differ from what Mattermost expects indicates a conflicting template from a prior configuration.
Confirm the account in ElasticsearchSettings.Username (or API key) has the manage_index_templates cluster privilege; without it, every index-template call is rejected.

Fix whichever of these Elasticsearch reports.

Step 4: Re-enable indexing and reindex

System Console > Environment > Elasticsearch > Enable Elasticsearch Indexing, set to true, then Index Now.

A full reindex is required because nothing was indexed while the engine was down. Confirm memory stays flat over the following 24 to 48 hours; if it climbs again, the engine is still failing to start and Step 3 needs to be repeated.

⚠️ Important: The leak itself, not just this trigger, is an open Mattermost defect: any future cause of a failed Elasticsearch startup reproduces it. If memory climbs again after a clean reindex, or you cannot identify the rejection cause in Step 3, open a support case referencing this behavior so it can be tracked against the fix.

If you cannot fix the startup failure right away

Leave indexing disabled and pick one of the following.

Option 1: Disable Database Search as well (recommended for high post volumes). Prevents search from falling back to the database, which degrades noticeably at high post volumes.

System Console > Environment > Database > Disable Database Search, set to true.

"SqlSettings": { "DisableDatabaseSearch": true }

⚠️ Important: For high post volume deployments, we also strongly recommend disabling Database Search once Elasticsearch or AWS OpenSearch is fully configured and running. The Mattermost Server will fall back on Database search if Elasticsearch or OpenSearch are unavailable, which can lead to performance degradation on high post volume deployments.

Reference: Disable database search

Option 2: Keep Database Search running on dedicated read/search replicas. If search must keep working, leave Database Search enabled and keep its load off the primary database.

System Console > Environment > Database > Disable Database Search, keep set to false (default).
Configure SqlSettings.DataSourceSearchReplicas with one or more dedicated read replica connection strings so search queries do not compete with primary application traffic.

"SqlSettings": { "DataSourceSearchReplicas": ["<search-replica-dsn>"] }

⚠️ Important: Mattermost database search starts to show performance degradation at around 2 million posts, on a server with 32 GB RAM and 4 CPUs. If you anticipate your Mattermost server reaching more than 2.5 million posts, we recommend enabling Elasticsearch or AWS OpenSearch Service for optimum search performance before reaching 3 million posts.

Reference: Mattermost Enterprise Search