Server Fails to Start After Upgrade Due to Read Replica Lag During App Migrations

Applies to: Mattermost Server v11.7.0 and later, self-hosted, licensed deployments with SqlSettings.DataSourceReplicas configured (e.g. AWS Aurora with a reader endpoint)

Symptoms: The server fatally exits during an app migration on the first boot after an upgrade.

🛑 Problem

Some app databse migrations write a row on the writer and immediately read it back to continue startup. That read-back is not pinned to the writer; on a licensed deployment with SqlSettings.DataSourceReplicas configured, it is routed to a read replica like any other read. If the replica has not yet applied the just-committed write, the lookup finds nothing and the server treats this as fatal, aborting startup. In HA, every node hits this independently on its own first boot, widening exposure under load.

Symptoms

Users or administrators experiencing this issue will see errors like:

fatal [...] Failed to run app migration caller="app/migrations.go:<line>" migration="<migration name>" error="<...>: sql: no rows in result set"

Additional symptoms:

• First boot only, on the version introducing the affected migration.
• Process exits immediately; no further log lines after the fatal line.
• A restart of the affected node typically resolves it, with no migration log lines on the successful boot.

✅ Solution

This is a read-after-write timing race against a read replica, not data loss or corruption. A restart reliably self-heals it because the migration's completion state is read from the writer, and by the time of the restart the previously written data has replicated. To avoid hitting this race during a production upgrade, force migration reads to the writer while the upgrading node completes its migrations.

Restart the affected node

Restart the Mattermost server process on the node that failed. On restart, the migration re-reads its previously written data; since that data has by then replicated to the read replica, the migration completes and startup proceeds normally.

systemctl restart mattermost

⚠️ Important: A successful app migration produces no log output; doAppMigrations only logs on failure (mlog.Fatal), never on success. The absence of migration log lines on a successful restart is expected and does not indicate a migration was skipped or failed silently.

Override DataSourceReplicas during the upgrade

Before starting the node that will run the new version's migrations, temporarily clear the read-replica configuration so all reads (including any migration re-reads) go to the writer. Restore the original value once that node has completed startup.

Where to change it: config.json, key SqlSettings.DataSourceReplicas (or environment variable MM_SQLSETTINGS_DATASOURCEREPLICAS). Set it to an empty list for the upgrading node's boot, then restore the original value(s) and restart again once startup succeeds.

MM_SQLSETTINGS_DATASOURCEREPLICAS=

⚠️ Important: Changes to SqlSettings.DataSourceReplicas require a server restart to take effect. In an HA cluster, upgrade and boot nodes one at a time using this override on the first node to run the new version, then roll the remaining nodes normally once migrations have completed and replicated.