AWS OpenSearch bulk index only indexes a single day

Adit Yadav

Updated November 14, 2025 10:09

Problem

When running a bulk index job against AWS OpenSearch, only a single day of data is indexed even though the job reports success. Historical data remains missing from search results. Running Bulk Index does not backfill historical content.

AWS OpenSearch blocks wildcard destructive operations that the Mattermost purge process relies on. Attempts to manually purge or delete indices do not resolve the issue.

Symptoms

Users experiencing this issue will see:

flush: 404 page not found

Additional symptoms include:

Successful bulk index job messages but only one day of data indexed
Purge Indexes command failing silently or partially completing
AWS OpenSearch restrictions preventing action.destructive_requires_name=false from being set

Solution

Follow these steps to resolve the indexing issue and backfill historical data:

Add Ignored Purge Indexes

Navigate to System Console → Environment → Elasticsearch → Indexes to skip while purging and add the following configuration:

.plugins-*,.kibana*,.opendistro*,.opensearch*,.ql-*,.tasks*

This ensures Mattermost skips protected AWS system indices during purge operations.

Purge and Reindex

Perform the following actions in order:

Disable Elasticsearch indexing
Re-enable Elasticsearch indexing
Run Purge Indexes
Run Index Now

Remove Old Indexing Jobs

Old indexing jobs should be deleted from the database. First, verify existing jobs:

SELECT Id, Type, Data, CreateAt, Status
FROM Jobs
WHERE Type = 'elasticsearch_post_indexing'
ORDER BY CreateAt DESC;

Delete the old indexing jobs:

DELETE FROM Jobs
WHERE Type = 'elasticsearch_post_indexing';

Verify the deletion:

SELECT COUNT(*)
FROM Jobs
WHERE Type = 'elasticsearch_post_indexing';

After deleting the old jobs, trigger Bulk Indexing again, and historical data should begin to populate.

Check OpenSearch Templates (Optional)

If needed, verify index templates using the following command:

curl -X GET <OPENSEARCH_URL>/_index_template -u USER:PASS

Important: Ensure you have a database backup before deleting jobs from the database. The purge and reindex process may take significant time depending on the volume of historical data.