Troubleshooting Mattermost Kubernetes Deployment Errors in Air-Gapped Environments

Summary
This article helps diagnose and resolve common configuration issues when deploying Mattermost in Kubernetes using the Mattermost Helm chart, especially in air-gapped environments with external PostgreSQL.

Issue

During deployment, Mattermost pods enter a crash loop and job server pods fail to initialize. Logs show an error like:

Failed to load configuration: failed to create store: unable to load on store creation: parsing error at line 1, character 2: invalid character 'e' looking for beginning of value

Root Cause

The error typically indicates an improperly formatted or misreferenced secret value in your Helm configuration, particularly related to:

• global.features.database.existingDatabaseSecret

• global.features.database.external.dataSource

In some cases, the issue can stem from incorrect key references for database and license secrets.

Resolution Steps

1. Verify Secret Names and Keys
   Use the default secret names and keys unless absolutely necessary:

   • Database:

     • Secret name: mattermost-dbsecret

     • Key: mattermost.dbsecret

   • License:

     • Secret name: mattermost-license

     • Key: mattermost.mattermost-license

2. Avoid Overriding Without Matching Template Logic
   If you override the default secret names or keys in values.yaml, ensure they match the Helm chart’s template logic. Misalignments may not surface clearly in logs but can prevent the config from loading.

3. License Handling Tip
   If Helm-based injection of the license fails, you may manually upload the license via the System Console under System Console > Site Configuration > License.

Best Practices

• Always validate values.yaml against the latest Mattermost Helm chart (avoid using outdated or external forks).

• For air-gapped deployments, ensure all required image dependencies and configs are mirrored and accessible internally.