#2624 Update: Telegraf Metrics for InfluxDB/Grafana (#4764)

* #2624 added config dialogs * Refactor Telegraf installation script to check if the service is already installed
2025-04-13 14:19:20 +02:00 · 2024-10-04 23:11:36 +02:00 · 2024-10-04 23:11:36 +02:00 · d61d1b90fb
commit d61d1b90fb
parent 18fc1e3866
10 changed files with 4328 additions and 3640 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -6,6 +6,7 @@
 - Update: LNDK 0.2.0 (Pay BOLT12 offers with LND) [details](https://github.com/lndk-org/lndk/releases/tag/v0.2.0)
 - Update: Helipad (Podcasting 2.0 Boostagram reader) v0.2.0 [details](https://github.com/Podcastindex-org/helipad/releases/tag/v0.2.0)
 - Update: Mempool 3.0.0 [details](https://github.com/mempool/mempool/releases/tag/v3.0.0)
+- Update: Telegraf Metrics for InfluxDB/Grafana [details](https://github.com/raspiblitz/raspiblitz/tree/dev/home.admin/assets/telegraf/README.md)
 - Update: RTL v0.15.2 [details](https://github.com/Ride-The-Lightning/RTL/releases/tag/v0.15.2)
 - Update: Jam (JoinMarket Web UI) v0.3.0 [details](https://github.com/joinmarket-webui/jam/releases/tag/v0.3.0)
 - Update: CLBOSS 0.13.3 [details](https://github.com/ZmnSCPxj/clboss/releases/tag/v0.13.3)
--- a/home.admin/_provision_.sh
+++ b/home.admin/_provision_.sh
@ -734,6 +734,15 @@ else
  echo "Provisioning Tailscale - keep default" >> ${logFile}
 fi

+# Telegraf
+if [ "${telegrafMonitoring}" = "on" ]; then
+  echo "Provisioning Telegraf - run config script" >> ${logFile}
+  /home/admin/_cache.sh set message "Setup Telegraf"
+  sudo -u admin /home/admin/config.scripts/bonus.telegraf.sh on >> ${logFile} 2>&1
+else
+  echo "Provisioning Telegraf - keep default" >> ${logFile}
+fi
+
 # custom install script from user
 customInstallAvailable=$(ls /mnt/hdd/app-data/custom-installs.sh 2>/dev/null | grep -c "custom-installs.sh")
 if [ ${customInstallAvailable} -gt 0 ]; then
--- a/home.admin/assets/telegraf/README.md
+++ b/home.admin/assets/telegraf/README.md
@ -1,34 +1,150 @@
-[telegraf](https://www.influxdata.com/time-series-platform/telegraf/) is a metric collection tool by influxData.
-It is opensource and works fine with [influxDB](https://www.influxdata.com/products/influxdb-overview/) as the timeseries database and [Grafana](https://grafana.com/grafana/) as the graphics front end.
+# Monitoring RaspiBlitz with Telegraf
+    
+[telegraf](https://www.influxdata.com/time-series-platform/telegraf/) is a metric collection tool by influxData. It is opensource and can push performance data of RaspiBlitz to an external [influxDB](https://www.influxdata.com/products/influxdb-overview/) server which then can be used as a datasource for the [Grafana](https://grafana.com/grafana/)WebUI to monitor RaspiBlitz in a Dashboard like this:

-You may take a look [here](https://github.com/gcgarner/IOTstack) for a nice dockerized installation of influxDB and Grafana on a raspberry pi.
+![Grafana Dashboard Example](grafana_dashboard.png)

-Make sure to have a telegraf section in your `/mnt/hdd/raspiblitz.conf`
-you have to manually edit them into `/mnt/hdd/raspiblitz.conf` before calling
+InfluxDB & Grafana should run on separate hardware - you might use an old RaspberryPi4 for that and take a look [here](https://github.com/gcgarner/IOTstack) for a nice dockerized installation of influxDB and Grafana.

+### Prepare your InfluxDB for RaspiBlitz
+
+Before you can start Teelegraf on RaspiBlitz you need to prepare a InfluxDB database for RaspiBlitz and create a user with password. So on your InfluxDB/Grafana machine follow the steps:
+
+Start the InfluxDB shell:
+```
+influx
+```
+
+If you havent yet create an `admin` user:
+```
+CREATE USER "admin" WITH PASSWORD '[ADMIN-PASSWORD]' WITH ALL PRIVILEGES;
+```
+
+Create database & user:
+```
+CREATE DATABASE raspiblitz;
+SHOW DATABASES;
+CREATE USER "raspiblitz" WITH PASSWORD '[RASPIBLITZ-PASSWORD]';
+GRANT ALL ON raspiblitz TO raspiblitz;
+```
+
+If you havent yet create a read-only user for Grafana:
+```
+CREATE USER "grafana" WITH PASSWORD '[GRAFANA-PASSWORD]';
+GRANT READ ON raspiblitz TO grafana;
+```
+
+To check if all is ready:
+```
+SHOW DATABASES;
+SHOW USERS;
+```
+
+You should see a database called `raspiblitz` and at least three users `admin, raspiblitz & grafana`.
+
+You can now exit the InfluxDB with:
+```
+exit
+```
+
+Make yure that the InfluxDB configuration file contains the the `[http]` section:
+```
+auth-enabled = true
+```
+
+On our Grafana/InfluxDB machine the config could be edited with:
+```
+sudo nano /etc/influxdb/influxdb.conf
+````
+Save the file (Ctrl+O, Enter) and exit (Ctrl+X).
+
+You my need to restart the InfluxDB server:
+```
+sudo systemctl restart influxdb
+```
+
+### Activate Telegraf on RaspiBlitz
+
+Back on your RaspiBlitz you can switch Telegraf service on with the following install script:
 ```
 /home/admin/config.scripts/bonus.telegraf.sh on
 ```

-You have to provide a running influxDB / Grafana infrastructure elsewhere (reachable for the RaspiBlitz)
+Enter the connection details the script asks you for.

-# telegraf section for raspiblitz.conf
-All telegraf switches and configuration variables. You may copy & paste them into your RaspiBlitz configuration at `/mnt/hdd/raspiblitz.conf`, after editing and providing the proper values which match your environment
+### Check Incoming Data on Grafana/InfluxDB server
+
+If install on the RaspiBlitz went thru OK you can go back to your Grafana/InfluxDB server and see if it received some basic CPU data:
+```
+influx -username admin -password '[ADMIN-PASSWORD]'
+USE raspiblitz;
+SHOW MEASUREMENTS;
+```
+
+If data is being written successfully, you should see measurements like cpu, mem, disk, etc.

 ```
-# all telegraf switches and configuration variables
-#
-# switch telegraf service and metrics capturing on/off
-telegrafMonitoring=on
-#
-# the full url to your influxDB data ingestion point, with port
-telegrafInfluxUrl='http://192.168.2.46:8086'
-#
-# the name of your influxDB database
-telegrafInfluxDatabase='raspiblitz'
-#
-# credentials for this database
-telegrafInfluxUsername='telegraf'
-telegrafInfluxPassword='metricsmetricsmetricsmetrics'
-
+SELECT * FROM cpu LIMIT 5;
 ```
+
+This should display the first five entries from the cpu measurement. If you see that data you can close the influxDB shell with:
+```
+exit
+```
+
+### Add Grafana Datasource
+
+Now configure Grafana to use the InfluxDB Data Source.
+
+Log In to Grafana. Use your Grafana credentials.
+
+- Click on the Configuration (gear icon) in the left sidebar.
+- Click on Data Sources.
+- Click Add data source and select InfluxDB.
+
+Configure the InfluxDB Data Source:
+```
+HTTP: 
+  URL: http://localhost:8086
+Auth: 
+  Leave default unless specific settings are needed.
+InfluxDB Details:
+  Database: raspiblitz
+  User: grafana
+  Password: [GRAFANA-PASSWORD]
+HTTP Method: POST (recommended)
+```
+
+Click the Save & Test button to verify the connection.Now 
+
+We need the datasource UID for later - to get it:
+
+- Click on Data Sources.
+- Click on the new created InfluxDB data source.
+
+Find the UID:
+
+In the URL of your browser, you should see something like this:
+```
+http://<your_vm_ip>:3000/datasources/edit/<data_source_uid>
+```
+
+The part after `/edit/` is the UID for your data source. It will be a unique identifier like `cOZgpReMk`
+
+We will need your personal datasource UID in the next step.
+
+### Create Grafana Dashboard
+
+Open the json data for the Dashboard in a new browser tab - following this link: https://raw.githubusercontent.com/raspiblitz/raspiblitz/refs/heads/dev/home.admin/assets/telegraf/raspiblitz-grafana-dashboard.json 
+
+Copy all JSON data (to paste it later).
+
+In your Grafana WebUI: Click on “+” → “Import”.
+
+Now paste the JSON data from above into the field 
+
+_BUT BEFORE YOU SAVE_ 
+
+replace ALL InfluxDB UIDs in that json with your personal one.
+
+
--- a/home.admin/assets/telegraf/grafana_dashboard.png
+++ b/home.admin/assets/telegraf/grafana_dashboard.png
--- a/home.admin/assets/telegraf/raspiblitz-grafana-dashboard.json
+++ b/home.admin/assets/telegraf/raspiblitz-grafana-dashboard.json
--- a/home.admin/config.scripts/bonus.labelbase.sh
+++ b/home.admin/config.scripts/bonus.labelbase.sh
@ -148,11 +148,8 @@ if [ "$1" = "1" ] || [ "$1" = "on" ]; then
  echo "# Installing ${APPID} ..."


-
-
-
  # create a dedicated user for the app
-  # BACKGROUND is here to seperate running apps by unix users
+  # BACKGROUND is here to separate running apps by unix users
  # and only give file write access to the rest of the system where needed.
  echo "# create user"
  # If the user is intended to be loeed in to add '--shell /bin/bash'
--- a/home.admin/config.scripts/bonus.telegraf.sh
+++ b/home.admin/config.scripts/bonus.telegraf.sh
@ -3,14 +3,16 @@

 ###############################################################################
 #   File:   bonus.telegraf.sh
-#   Date:   2020-10-03
 ###############################################################################

 # command info
 if [ $# -eq 0 ] || [ "$1" = "-h" ] || [ "$1" = "-help" ]; then
- echo "config script to switch the telegraf metric collection on or off"
- echo "bonus.telegraf.sh [on|off|status]"
- exit 1
+  echo "# config script to switch the telegraf metric collection"
+  echo "# detailed setup info: github.com/raspiblitz/raspiblitz/tree/dev/home.admin/assets/telegraf"
+  echo "# bonus.telegraf.sh status ---> get status of telegraf service"
+  echo "# bonus.telegraf.sh on     ---> install & config"
+  echo "# bonus.telegraf.sh off    ---> uninstall & reset config"
+  exit 1
 fi

 # at this point the config file exists and can be sourced
@ -50,14 +52,151 @@ if [ "$1" = "status" ]; then
 fi


+# Function to check if input is empty
+function check_empty() {
+    if [ -z "$1" ]; then
+        return 1
+    else
+        return 0
+    fi
+}
+
+# Function to validate IP address or domain with optional http(s) and port
+function validate_url() {
+    if [[ "$1" =~ ^(https?://)?([a-zA-Z0-9.-]+|\b\d{1,3}(\.\d{1,3}){3}\b):[0-9]+$ ]]; then
+        return 0
+    else
+        return 1
+    fi
+}
+
 ###############################
 # switch on
 if [ "$1" = "1" ] || [ "$1" = "on" ]; then
+
  echo "*** INSTALL TELEGRAF ***"
+
+  # check installed by looking for service
+  serviceInstalled=$(sudo systemctl status telegraf --no-page 2>/dev/null | grep -c "telegraf.service - The plugin-driven")
+  if [ ${serviceInstalled} -eq 1 ]; then
+    echo "# Telegraf service is installed."
+    echo "# If you want to reset config and reinstall, please switch off first."
+    exit 0
+  fi
+
+  # check if config data in raspiblitz.conf is available
+  configMissing=0
+  if [ ${#telegrafInfluxUrl} -eq 0 ]; then
+    echo "# Missing telegrafInfluxUrl in raspiblitz.conf"
+    configMissing=1
+  fi
+  if [ ${#telegrafInfluxDatabase} -eq 0 ]; then
+    echo "# Missing telegrafInfluxDatabase in raspiblitz.conf"
+    configMissing=1
+  fi
+  if [ ${#telegrafInfluxUsername} -eq 0 ]; then
+    echo "# Missing telegrafInfluxUsername in raspiblitz.conf"
+    configMissing=1
+  fi
+  if [ ${#telegrafInfluxPassword} -eq 0 ]; then
+    echo "# Missing telegrafInfluxPassword in raspiblitz.conf"
+    configMissing=1
+  fi
+
+  # enter config if missing (first init)
+  if [ ${configMissing} -eq 1 ]; then
+  
+    # Display the info box with whiptail
+    whiptail --title "Metrics Setup Information" --yesno "To run the Telegraf metrics service you need an external monitoring server running Grafana & InfluxDB. Please prepare InfluxDB database & user as described in github.com/raspiblitz/raspiblitz/tree/dev/home.admin/assets/telegraf Choose YES if all is ready to config RaspiBlitz Telegraf service." 11 75;
+    if [ $? -eq 1 ]; then
+      echo "# user cancel"
+      exit 0
+    fi
+
+    # Collect telegrafInfluxUrl
+    telegrafInfluxUrl=""
+    while true; do
+      telegrafInfluxUrl=$(whiptail --inputbox "Enter the IP address or domain followed by port of your metrics InfluxDB (e.g., http://192.168.1.1:8086):" 8 78 "${telegrafInfluxUrl}" --title "InfluxDB Connection" 3>&1 1>&2 2>&3)
+      exitstatus=$?
+      if [ $exitstatus -ne 0 ]; then
+        echo "Operation canceled by user."
+        exit 1
+      fi
+      if ! check_empty "$telegrafInfluxUrl"; then
+        whiptail --msgbox "Input cannot be empty. Please enter a valid URL." 8 78
+        continue
+      fi
+      if ! validate_url "$telegrafInfluxUrl"; then
+        whiptail --msgbox "Invalid format. Please enter a valid IP address or domain followed by a port, with optional http(s) prefix." 8 78
+        continue
+      fi
+      # Perform a test using curl to check if the service is running
+      [[ $telegrafInfluxUrl =~ ^(http|https):// ]] || telegrafInfluxUrl="http://$telegrafInfluxUrl"
+      if curl --output /dev/null --silent --head --fail "${telegrafInfluxUrl}/ping"; then
+        echo "OK Service is running at $telegrafInfluxUrl."
+        break
+      else
+        whiptail --msgbox "Was not able to connect to ${telegrafInfluxUrl} - please make sure InfluxDB is running and reachable for RaspiBlitz." 8 78
+        continue
+      fi
+    done
+
+    # Collect telegrafInfluxDatabase
+    while true; do
+      telegrafInfluxDatabase=$(whiptail --inputbox "Enter the name of the database where to store the metrics:" 8 78 "raspiblitz" --title "InfluxDB Database" 3>&1 1>&2 2>&3)
+      exitstatus=$?
+      if [ $exitstatus -ne 0 ]; then
+        echo "Operation canceled by user."
+        exit 1
+      fi
+      if ! check_empty "$telegrafInfluxDatabase"; then
+        whiptail --msgbox "Input cannot be empty. Please enter a valid database name." 8 78
+        continue
+      fi
+      break
+    done
+
+    # Collect telegrafInfluxUsername
+    while true; do
+      telegrafInfluxUsername=$(whiptail --inputbox "Enter the username that is allowed to write on that database:" 8 78 "raspiblitz" --title "InfluxDB Username" 3>&1 1>&2 2>&3)
+      exitstatus=$?
+      if [ $exitstatus -ne 0 ]; then
+        echo "Operation canceled by user."
+        exit 1
+      fi
+      if ! check_empty "$telegrafInfluxUsername"; then
+        whiptail --msgbox "Input cannot be empty. Please enter a valid username." 8 78
+        continue
+      fi
+      break
+    done
+
+    # Collect telegrafInfluxPassword
+    while true; do
+      telegrafInfluxPassword=$(whiptail --passwordbox "Enter the password for that username:" 8 78 --title "InfluxDB Password" 3>&1 1>&2 2>&3)
+      exitstatus=$?
+      if [ $exitstatus -ne 0 ]; then
+        echo "Operation canceled by user."
+        exit 1
+      fi
+      if ! check_empty "$telegrafInfluxPassword"; then
+        whiptail --msgbox "Input cannot be empty. Please enter a valid password." 8 78
+        continue
+      fi
+      break
+    done
+
+    # save the config data to raspiblitz.conf
+    /home/admin/config.scripts/blitz.conf.sh set telegrafInfluxUrl "${telegrafInfluxUrl}"
+    /home/admin/config.scripts/blitz.conf.sh set telegrafInfluxDatabase "${telegrafInfluxDatabase}"
+    /home/admin/config.scripts/blitz.conf.sh set telegrafInfluxUsername "${telegrafInfluxUsername}"
+    /home/admin/config.scripts/blitz.conf.sh set telegrafInfluxPassword "${telegrafInfluxPassword}"
+  fi
+
  # source and target dir for copy operation
  telegraf_source_dir=${resources_dir}
  telegraf_target_dir=/etc/telegraf
-  #
+  
  # full path to telegraf config file for sed-replace operation
  telegraf_conf_file=${telegraf_target_dir}/telegraf.conf

@ -160,8 +299,12 @@ if [ "$1" = "0" ] || [ "$1" = "off" ]; then
  # let apt-get remove the package
  sudo apt-get remove -y telegraf

-  echo "*** telegraf remove: set 'telegrafMonitoring=off' in config file 'raspiblitz.conf'"
+  echo "*** telegraf switch off and remove config ***"
  /home/admin/config.scripts/blitz.conf.sh set telegrafMonitoring "off"
+  /home/admin/config.scripts/blitz.conf.sh delete telegrafInfluxUrl
+  /home/admin/config.scripts/blitz.conf.sh delete telegrafInfluxDatabase
+  /home/admin/config.scripts/blitz.conf.sh delete telegrafInfluxUsername
+  /home/admin/config.scripts/blitz.conf.sh delete telegrafInfluxPassword

  echo "*** remove telegraf done ***"

--- a/home.admin/config.scripts/bonus.template.sh
+++ b/home.admin/config.scripts/bonus.template.sh
@ -11,7 +11,7 @@
 APPID="template" # one-word lower-case no-specials

 # clean human readable version - will be displayed in UI
-# just numbers only seperated by dots (2 or 0.1 or 1.3.4 or 3.4.5.2)
+# just numbers only separated by dots (2 or 0.1 or 1.3.4 or 3.4.5.2)
 VERSION="0.1"

 # the git repo to get the source code from for install
@ -158,7 +158,7 @@ if [ "$1" = "1" ] || [ "$1" = "on" ]; then
  /home/admin/config.scripts/bonus.nodejs.sh on

  # create a dedicated user for the app
-  # BACKGROUND is here to seperate running apps by unix users
+  # BACKGROUND is here to separate running apps by unix users
  # and only give file write access to the rest of the system where needed.
  echo "# create user"
  # If the user is intended to be loeed in to add '--shell /bin/bash'
--- a/home.admin/config.scripts/cl.backup.sh
+++ b/home.admin/config.scripts/cl.backup.sh
@ -466,7 +466,7 @@ if [ ${mode} = "seed-import-gui" ]; then
      sudo chown admin:admin /var/cache/raspiblitz/.seed.tmp

      # dialog to enter
-      dialog --backtitle "RaspiBlitz - Recover from Core Lightning seed" --inputbox "Please enter/paste the SEED WORD LIST:\n(just the words, seperated by spaces, in correct order as numbered)" 9 78 2>/var/cache/raspiblitz/.seed.tmp
+      dialog --backtitle "RaspiBlitz - Recover from Core Lightning seed" --inputbox "Please enter/paste the SEED WORD LIST:\n(just the words, separated by spaces, in correct order as numbered)" 9 78 2>/var/cache/raspiblitz/.seed.tmp
      wordstring=$(cat /var/cache/raspiblitz/.seed.tmp | sed 's/[^a-zA-Z0-9 ]//g')
      sudo shred -u /var/cache/raspiblitz/.seed.tmp 2>/dev/null
      echo "processing ..."
--- a/home.admin/config.scripts/lnd.backup.sh
+++ b/home.admin/config.scripts/lnd.backup.sh
@ -617,7 +617,7 @@ if [ ${mode} = "seed-import-gui" ]; then
      sudo chown admin:admin /var/cache/raspiblitz/.seed.tmp

      # dialog to enter
-      dialog --backtitle "RaspiBlitz - Recover from LND seed" --inputbox "Please enter/paste the SEED WORD LIST:\n(just the words, seperated by spaces, in correct order as numbered)" 9 78 2>/var/cache/raspiblitz/.seed.tmp
+      dialog --backtitle "RaspiBlitz - Recover from LND seed" --inputbox "Please enter/paste the SEED WORD LIST:\n(just the words, separated by spaces, in correct order as numbered)" 9 78 2>/var/cache/raspiblitz/.seed.tmp
      wordstring=$(cat /var/cache/raspiblitz/.seed.tmp | sed 's/[^a-zA-Z0-9 ]//g')
      sudo shred -u /var/cache/raspiblitz/.seed.tmp 2>/dev/null
      echo "processing ..."