Calls Recording Fails With "failed to initialize job service" After Reboot

Applies to: Mattermost Server v9.x and later with the Calls plugin and self-hosted calls-offloader (Linux, particularly RHEL/CentOS/Fedora)

Symptoms: Calls recording stops working after a host reboot or weekly patching cycle, and calls-offloader panics during the authentication step of /jobs/init.

🛑 Problem

The calls-offloader service keeps its authentication data in an on-disk bitcask database. By default, this database lives at /tmp/calls-offloader-db. On RHEL-family distributions, /tmp is periodically cleared, which corrupts the bitcask database. On the next /jobs/init call from the Calls plugin, the offloader panics while looking up auth data instead of returning cleanly.

Symptoms

The Mattermost server log shows:

failed to initialize job service

with the connection closed by the offloader (FIN/EOF). The offloader's standard log shows the call as a success, so the panic is only visible in journalctl:

journalctl -u calls-offloader

Additional symptoms:

• Recording fails to start when a call is recorded.
• Issue appears immediately after a reboot, package update, or weekend patching cycle.
• The offloader's own log file looks clean because the panic only lands in journald.

✅ Solution

Recreate the bitcask database and relocate it off /tmp so it survives future cleanup.

Recover the Offloader

1. Confirm the current database location. Look for the offloader's startup line in journald:

journalctl -u calls-offloader | grep "initiated data store"

If STORE_DATASOURCE is not set in the service environment, the path defaults to /tmp/calls-offloader-db.

2. Stop the offloader service:

sudo systemctl stop calls-offloader

3. Remove the corrupted database:

sudo rm -rf /tmp/calls-offloader-db

4. Restart the service:

sudo systemctl start calls-offloader

5. In the System Console, disable and re-enable the Calls plugin to trigger a fresh /jobs/init. Because self-registration is enabled, the offloader recreates the database and the plugin re-registers automatically.

Relocate the Database Off /tmp

1. Pick a persistent path next to the binary, for example /usr/local/var/calls-offloader/db, and set it in the service environment:

STORE_DATASOURCE=/usr/local/var/calls-offloader/db

2. Restart the offloader service.
3. Restart the Calls plugin.

⚠️ Important: Also point the offloader log file off /tmp for the same reason. Anything under /tmp on RHEL-family hosts is subject to periodic cleanup and will recur after the next reboot.

Additional Resources

For more information, see:

Mattermost Calls self-hosted job service