Problem
After events such as upgrading Mattermost, the server may fail to start or enter a repeated crash loop a few minutes after launch. This behavior is commonly caused by the PostPersistentNotifications background job reprocessing older posts, especially those containing mentions generated by bots or automated systems.
In affected versions, older post metadata may not be parsed correctly. When the persistent notifications worker encounters a nil pointer condition while reading this metadata, it triggers a panic that brings down the server. Because the worker retries every 4–5 minutes, this results in repeated crashes and the appearance that the upgrade itself is failing.
When this issue occurs, the logs typically show repeated panics from the persistent notifications worker. These entries confirm that the crash loop is caused by this background job reprocessing older posts.
jobs.go:233","worker_name":"PostPersistentNotifications","job_type":"post_persistent_notifications","panic":"runtime error: invalid memory address or nil pointer dereference"
Symptoms
- Mattermost starts and then crashes after 1–4 minutes
- Server repeatedly restarts in a crash loop
- Upgrade appears stuck or slow to progress
- Temporary unavailability during server startup
Solution
The recommended workaround is to temporarily disable persistent notifications during the upgrade. This prevents the background worker from running and stops the nil pointer crash loop.
You can apply this by setting the environment variable:
Step 1: Add the variable to your Mattermost service file
Step 2: Restart Mattermost
Once this variable is set, Mattermost will start normally without entering a crash loop. Persistent notifications can be re-enabled after upgrading, or you may upgrade directly to v11.3+ for the permanent fix.
Comments
Article is closed for comments.