Resolving Missing Job Execution Server in High Availability Screen

Adit Yadav

Updated March 31, 2025 09:59

Description

In some cases, after migrating from MySQL to PostgreSQL, the job execution server may disappear from the High Availability (HA) settings screen. This issue can occur due to outdated environment variables referencing the old database settings, causing the server to fail in joining the cluster. This guide provides steps to diagnose and resolve the issue.

Symptoms

The job execution server is missing from the High Availability (HA) settings screen.
Cluster logs show errors indicating the node could not join the cluster.

Environment

Mattermost High Availability deployment
Migrated from MySQL to PostgreSQL

Root Cause

Environment variables in the job execution server were still referencing the old MySQL database settings, preventing the server from joining the cluster.

Solution

Check Cluster Logs:

Review the logs for errors indicating cluster connection issues.

Sample Error:

{"timestamp": "2025-02-21 08:51:17.603 +09:00",
 "level": "info",
 "msg": "This node attempted to join the cluster, but could not find any nodes.",
 "caller": "cluster/cluster.go:403",
 "discovery_id": "cdw1toue9pdi8dikjyz4r6uzay"}

Confirm Configuration Settings:

Verify the HA cluster settings in your Mattermost configuration file. Example configuration:

"ClusterSettings": {
    "Enable": true,
    "ClusterName": "Staging",
    "OverrideHostname": "",
    "NetworkInterface": "",
    "BindAddress": "",
    "AdvertiseAddress": "",
    "UseIPAddress": true,
    "EnableGossipCompression": true,
    "EnableExperimentalGossipEncryption": false,
    "ReadOnlyConfig": false,
    "GossipPort": 8074,
    "StreamingPort": 8075,
    "MaxIdleConns": 100,
    "MaxIdleConnsPerHost": 128,
    "IdleConnTimeoutMilliseconds": 90000
}

Review Environment Variables:
- Check the following environment variables for outdated values:
  - MM_SQLSETTINGS_DATASOURCE
  - MM_SQLSETTINGS_DATASOURCEREPLICAS
  - MM_CONFIG
Update Environment Variables:
- Correct these variables to point to the new PostgreSQL database.
Restart the Server:
- After updating the environment variables, restart the Mattermost job execution server.
Verification:
- Confirm the job execution server appears in the HA settings screen.

Conclusion

Correcting outdated environment variables successfully restored the job execution server's visibility in the HA settings screen.