Investigating a Suspected Plugin Memory Leak and Capping Server Memory Growth

Marco Kundt

Updated September 02, 2026 17:04

Applies to: Self-hosted Mattermost Server (all supported versions) running server-side plugins, on systemd or Docker.

Symptoms: The mattermost process grows in resident memory over time, and a plugin is suspected as the cause.

🛑 Problem

Server-side plugins run as separate child processes of the main mattermost process, so a plugin that retains memory shows growth in its own process rather than in the server's. When memory growth appears in the main process instead, the plugin is not necessarily the source, and log lines attributed to a plugin do not establish where the memory is held. Without a heap profile the growth cannot be assigned to any component, and if the host exhausts available RAM and swap, the kernel's out-of-memory killer can select a process anywhere on the system, which can take down unrelated services alongside Mattermost.

Symptoms

Resident memory of the mattermost process climbs steadily and does not return to baseline after periods of low activity.
Growth well beyond the node's expected working set, relative to the published reference architecture for its user count, indicates abnormal retention rather than normal usage.

🔍 Diagnostic Steps

These steps attribute the growth to a specific component and cap the damage in the meantime. They do not stop a leak; a confirmed leak needs a fix in the component that owns it.

Capture a Support Packet While Memory Is Still Growing

Generate the packet before the process is restarted or killed, while memory is still elevated. The packet includes <node-id>/heap.prof, <node-id>/cpu.prof, and goroutine dumps per node, which is the only evidence that attributes memory to a component.

mmctl system supportpacket -o /var/tmp/mm_support_packet_growth.zip

Read the profile with pprof, sorted by retained bytes:

go tool pprof -top -sample_index=inuse_space heap.prof
go tool pprof -web -sample_index=inuse_space heap.prof

⚠️ Important: A packet taken after a restart is of no diagnostic use for this purpose. The heap profile reflects the state at capture time, so the capture has to happen during the growth.

Sample the Plugin Subprocesses Over Time with childmem

childmem samples the child processes of a named parent process on a timer and writes the readings to CSV, which makes growth in a specific plugin visible across a run.

go build -o childmem .
./childmem -pname mattermost -includeParent -output /var/tmp/child_mem.csv

Leave it running for the duration of the suspected leak, then compare each plugin's memory column over time in the CSV. Growth confined to one plugin's row points at that plugin; growth across the parent row with flat plugin rows points elsewhere, and the heap profile is then the only way forward.

Cap Memory for the systemd Unit

Setting a limit on the unit converts a host-wide out-of-memory event into one confined to Mattermost, so other services on the node survive. Add MemoryMax to the [Service] section of /etc/systemd/system/mattermost.service, sized to leave headroom for the OS and any co-located services:

[Unit]
Description=Mattermost
After=network.target

[Service]
Type=notify
ExecStart=/opt/mattermost/bin/mattermost
TimeoutStartSec=3600
KillMode=mixed
Restart=always
RestartSec=10
WorkingDirectory=/opt/mattermost
User=mattermost
Group=mattermost
LimitNOFILE=49152
MemoryMax=<value, e.g. 6G>

[Install]
WantedBy=multi-user.target

Reload the unit definition and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart mattermost

Restart=always with RestartSec=10 brings the server back automatically when the limit is reached. Confirm the limit is active:

systemctl show mattermost -p MemoryMax -p MemoryCurrent

⚠️ Important: Plugin subprocesses share the server's cgroup, so the limit applies to the unit as a whole and cannot isolate a single plugin. Reaching it terminates the whole Mattermost service, including plugins.

Cap Memory for a Docker Deployment

The official Compose file already sets a mem_limit on the mattermost service. Adjust it to match the host, keeping it below total host RAM:

  mattermost:
    image: mattermost/${MATTERMOST_IMAGE}:${MATTERMOST_IMAGE_TAG}
    restart: ${RESTART_POLICY}
    mem_limit: <value, e.g. 6G>

Apply it by recreating the container:

docker compose up -d mattermost

Verify the limit and watch usage:

docker inspect -f '{{.HostConfig.Memory}}' mattermost
docker stats mattermost

⚠️ Important: docker stats reports the container total, not per-process figures. To see individual plugin processes, run childmem or the equivalent sampling inside the container with docker exec.

Contact Mattermost Support

If the heap profile does not explain the growth, or it points at a plugin, open a support case and attach the packet captured during growth along with the memory and hook-rate graphs for the window preceding it.

Additional Resources

Related to

OOM
memory