Problem
In some self-hosted Mattermost deployments, administrators may observe that all users are suddenly logged out and shown the message “Your session has expired.” After this occurs, users may be unable to log back in. Login attempts can return “Incorrect password”, even when valid credentials are used.
Password reset attempts may also fail. In some cases, password reset emails are not delivered. In other cases, users receive the email but the reset link is already expired upon arrival. This results in a complete authentication outage.
This behavior can possibly occur when the server or VM hosting Mattermost reaches 100% root disk utilization. When disk space is exhausted, the system can no longer write logs or temporary files. The database may stop accepting connections, preventing Mattermost from validating sessions or generating password reset tokens.
This can be an infrastructure-related issue and is not caused by a defect in the Mattermost application itself.
Original exception was:
OSError: [Errno 28] No space left on device
Symptoms
- All users logged out simultaneously
- Login attempts fail with “Incorrect password”
- Password reset emails not delivered or links already expired
- Mattermost authentication unavailable
- Database stops accepting connections
Solution
To resolve this issue,in case you are seeing space related errors, ensure that sufficient disk space is available on the server or VM hosting Mattermost. Mattermost and its database require free disk space to write logs, temporary files, and authentication data.
If the root or boot disk has reached 100% utilization, restore disk capacity by increasing the disk size, expanding the filesystem, or removing unnecessary data, depending on your infrastructure setup. Its suggested to deploy some monitoring mechanism to your infrastructure to track such alerts efficiently.
After disk space has been restored, restart the following services to allow Mattermost to resume normal operation:
Mattermost service
Once sufficient disk space is available and services are running normally, users should be able to log in successfully and use password reset functionality without further issues.
Comments
Article is closed for comments.