S3 Storage Connection Fails in Kubernetes Deployment

Applies to:
Mattermost Self-Hosted v10.10 and later on Kubernetes with custom CA or on-prem S3-compatible storage.

Summary

When deploying Mattermost on Kubernetes using S3 or S3-compatible storage, file uploads may fail, and the application logs show messages such as:

Error decoding the config
Unable to connect to S3. Verify your Amazon S3 connection authorization parameters and authentication settings.

Problem

Customers using on-prem or self-hosted S3-compatible storage (for example, MinIO, Ceph, or OpenShift-based storage) report that Mattermost cannot connect to S3 despite valid credentials.

Typical signs include:

• “Error decoding the config” in server logs
• “Unable to connect to S3” errors during startup or file upload
• Verified S3 credentials that work outside the Kubernetes cluster
• Environments running custom certificate authorities (CAs)

The root cause is usually that the Mattermost application container lacks access to the custom CA certificate required to establish a trusted connection to the internal S3 endpoint.

Solution

To resolve this issue, mount your organization’s custom CA certificate into the Mattermost pod and update the container’s trust store.

Below is a working configuration example for adding a trusted CA using init containers and ConfigMaps.

1. Update your mattermost-install.yaml

apiVersion: installation.mattermost.com/v1beta1
kind: Mattermost
metadata:
  name: mattermost
  namespace: mattermost
spec:
  podExtensions:
    initContainers:
      - name: init-ca-cert
        image: <UBI_or_RHEL_based_container>
        command: ["sh"]
        args: ["-c", "$(SETUP_SCRIPT)"]
        env:
        - name: TRUSTED_CERT
          valueFrom:
            configMapKeyRef:
              name: trusted-ca
              key: ca.crt
        - name: SETUP_SCRIPT
          valueFrom:
            configMapKeyRef:
              name: setup-script
              key: setup.sh
        securityContext:
          privileged: true
        volumeMounts:
          - name: cert-bundles-pem-vol
            mountPath: /certs/pem
          - name: cert-bundles-openssl-vol
            mountPath: /certs/openssl

  volumes:
    - name: trusted-ca-vol
      configMap:
        name: trusted-ca
        items:
          - key: ca.crt
            path: ca.crt
    - name: cert-bundles-pem-vol
      emptyDir: {}
    - name: cert-bundles-openssl-vol
      emptyDir: {}
  volumeMounts:
    - name: trusted-ca-vol
      mountPath: /etc/pki/ca-trust/source/anchors/ca.crt
      subPath: ca.crt

2. Add the setup script ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: setup-script
  namespace: mattermost
data:
  setup.sh: |
    echo "$TRUSTED_CERT" > /etc/pki/ca-trust/source/anchors/trusted_ca.crt
    update-ca-trust extract
    cp -r /etc/pki/ca-trust/extracted/pem /certs/
    cp -r /etc/pki/ca-trust/extracted/openssl /certs/

3. Add your trusted CA ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: trusted-ca
  namespace: mattermost
data:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

After redeploying Mattermost with these changes, the container will trust your internal CA and successfully connect to the S3 endpoint.

Next Steps

• Verify the deployment using kubectl logs to ensure S3 initialization succeeds.
• Confirm that file uploads and attachments work as expected.
• For production environments, keep custom CAs managed in a secure, versioned ConfigMap.