Grafana Dashboards are Missing Performance Data

Problem

After setting up Prometheus and Grafana data based on E20 (Enterprise) Performance Monitoring, most of the published Mattermost dashboards are missing data (see example below):

Solution

A common cause of this issue is the interval Prometheus is configured to use to scrape the Mattermost cluster for performance metrics data.

The Mattermost dashboards are designed such that many of the measures are looking at the "rate of change" of a given metric. This rate of change is typically set to calculate over a 1 minute window of data. If you have Prometheus set to scrape on a 1 minute interval, the Grafana dashboards will not have enough data to compute the "rate of change" and will not display anything.

To address this, ensure the sampling interval you have configured in Prometheus is set to 30 seconds or less.