![]() For the purpose of this blog entry, I am going to import a dashboard on this link Its time to import a grafana dashboard for Kafka lag monitor. So our Prometheus server is now able to scrape Kafka lag monitor for metrics. Type in kafka_consumergroup_group_lag metric. The metrics from Kafka lag exporter can be queried like any other metrics. kafka-lag-exporter -Dconfig.file=/home/ec2-user/kafka-lag-exporter-0.6.2/bin/nf Query Kafka lag metrics using PromQL Kafka Lag Monitor can be found on GitHub on this link. ![]() Kafka Lag exporter is used to monitor this metric and use it as a health indicator of how quickly/slowly data in Kafka topic is being consumed. Lag may go up or down for various reasons. In Kafka, every consumer group ingests data at a certain number of messages/second. Monitoring Kafka Lags is equally important to measure the performance of Kafka consumer applications and your data pipeline. Now let’s turn our attention to monitoring Kafka topic lags. You can explore various individual metrics and come up with something new! Download Kafka Lag exporter You can now add/change/remove charts to suit your requirements. Step 3 -Select the data source and folder name. Step 2 – You can import by typing the id assigned by grafana website to the dashboard or directly paste the JSON. Step 1 – Press the button as shown below To import a grafana dashboard follow these steps Its time to import a grafana dashboard for Kafka brokers. So our Prometheus server is now able to scrape Kafka broker metrics. This metric is available to a jmx_exporter by default Grafana Dashboard for Kafka Brokers The metrics from JMX exporter can be queried like any other metrics. This would bring up something similar as below Query Prometheus using PromQL You can check if Prometheus server is able to scrape the metrics is by navigating to Prometheus UI on Prometheus can also show you if it is scrapping metrics of Kafka Brokers prometheus -config.file= "prometheus.yml" =400d = "data/" In my case it is $KAFKA_HOME/conf Step – 2 – Edit kafka-server-start.sh Step – 1 – Goto the Kafka configuration directory ![]() Integrating Kafka and JMX exporter is easy and it requires only one line added! ? Here goes I downloaded and have stored my configuration for JMX exporter in the $KAFKA_HOME/config/jmx_exporter.yml Configure Kafka broker There is already a sample configuration file to get us started and it is available on this link. Naming and filtering of the metrics can be done via regex expressions as a configuration in a YAML file.
0 Comments
Leave a Reply. |