Zabbix Notifications

Zabbix

It all started when my Plex Media Server/Playground box (a little NUC8i3BEK) started having some not so good fan noises emanating from it. I ordered a new fan on Amazon to replace it and go fucking figure the fan noises stop when the new replacement fan arrived. So I shelved the new part, I still have it because I am sure I will need it eventually but for now the original is working fine but I don’t trust it and this leads me finally to my point. I needed a way to monitor my CPU temperatures AND send me a notification. There are a ton of ways to see your CPU temperature but not a whole lot of ways to be notified when they hit certain thresholds. Hello Zabbix!

This took me my entire night to get done, so it shouldn’t for you. You’re welcome.

After Googling for a bit I came across an AskUbuntu Question that sent me in the direction of Zabbix. I’ll drop the link to the question so you can see the other options available similar to Zabbix. I went through the list and settled on Zabbix it seemed to fit the bill for what I wanted.

If you go to their downloads page there is a handy chart with step-by-step directions on how to install Zabbix on your system. I went with v6.4 – Ubuntu – 22.04 – Server, frontend, Agent – MySQL – Nginx. I will list those steps here for Zabbix on Ubuntu with Nginx, if you have a different system OS or Apache you should check out the download page for install directions.

  1. Requirements
  2. lm-sensors
  3. Setup Zabbix repo
  4. Install Zabbix
  5. Create the database
  6. Configure Nginx
  7. Zabbix Agents
  8. Start it up!
  9. Gmail media type notification
  10. Create a custom item
  11. Create a custom Trigger
  12. Create a custom Action

 

Requirements

I am assuming you have the following installed or at least something similar, and configured; Ubuntu (22.04), PHP (8.2), Nginx, MySQL, a working sendmail system or a Gmail account with an application password for SMTP – or another service that Zabbix supports, like Discord. But I am only covering Gmail here.

As far as the Gmail setup goes that needs attention after Zabbix is installed also. Alerts > Media Types > Gmail (or another service).

 

lm-sensors

I have seen some templates and other configuration files out there for supposed “drop-in” CPU temperature monitoring but they are all pretty dated and I did not trust them. So I forged my own path. Of the options I saw for gathering the temperatures there was lm-sensors and SNMP. I went with lm-sensors as I have played with SNMP in the past and I am pretty sure lm-sensors is the easy fast way to go. We’ll install lm-sensors first then Zabbix.

 

Install lm-sensors, this is how we get the temperatures of our devices (hopefully they are compatible).

That’s it, quick and easy. Time to run it, just type sensors  in a terminal.

 

It will spit out a bunch of info hopefully, the more devices and drives you have the more temperatures you should see. This is what I have on my little sever box.

Now sensors will spit out a bunch of info but we only want the temperature, just the numerical data no text. To get that we can use grep. Fore example this is what I used to strip out the text.

This leaves me with just “46.2” for example.

 

Setup repository for Zabbix

 

Install Zabbix

There are Docker and Docker-Compose versions of Zabbix and normally that is the route I would go but after looking at the massive config.yaml file for getting it up on Docker I said hell nah way too complicated! But it does exist and it is an option.

 

Create a MySQL database for Zabbix

Login to MySQL

Then at the mysql>  prompt enter the following commands one line at a time to create a database user.

Import the Zabbix data – this can take a few minutes depending on your server speed.

Disable log_bin_trust_function_creators option after importing the data.

Edit the file and update the password and other fields sudo nano /etc/zabbix/zabbix_server.conf

 

Configure Nginx for Zabbix

Edit the Nginx file  /etc/zabbix/nginx.conf  and uncomment these lines (server_name can be a domain or IP). Change the port if you need to.

Restart and enable Zabbix to run

 

Zabbix Agent

Edit the configuration file for agents.

Edit the Hostname if you wish.

Find and uncomment UnsafeUserParameters=1 .

Save, exit.

 

Restart it all and lets test!

 

Zabbix Interface

Open your browser and navigate to the Zabbix landing page!

Default login info

There should be a default host “Zabbix server” already configured for you. Click on it and you will see a bunch of options.

 

Gmail Service for Notifications

You need to setup your notification server. I am using Gmail but there are other options for you to use.

Alerts > Media Types > Gmail (or another service). Configure and enable this service. You will probably need to use Gmail application passwords to get around the 2FA.

 

Create a Custom Item

Now we need to add our own custom item to monitor CPU temperatures and alert us when it hits a limit.

Click on Zabbix server and then click on Items.

 

The important part is the Key , that’s our sensor(s) we are going to monitor. For example, to set it to monitor my CPU core temperatures I need to use sensor[coretemp-isa-0000,temp1] & sensor[coretemp-isa-0000,temp2]. That specifies the CPU and which core temperature. I have two cores so I will be making two Items one for each core.

Then before you leave that screen clone the Item for however many cores or devices you want to monitor so you don’t have to enter all the information again, you just have to change the Key .

Now that you have your Items set we need to create the Trigger for our alerts.

 

Create a new Trigger

Now need to create a new trigger for that item we just made.

For example you could set your triggering expression as this: last(/server/temp)>75

And then you could set your recovery expression as: last(/server/temp)<=70

That would activate when the “server” hits 75C and it would recover when it it hits 70C. If you leave recovery as “Expression” as soon as it dips below the initial activation temperature it will recover.

 

New Trigger Action

Next is to crate a trigger action for the trigger for the item. Alerts > Actions > Trigger Actions .

  • Create a new action. Give it a name “Send Notification Email”, enable the action, add some conditions to monitor, and under Operations is where you setup the notification rules.

On the Action tab fill in the form:

  • Under Operations > “Send message to user: via all media”, and select yourself as the user. Start in: immediately.
  • Recovery operations & Update operations: notify all involved. Check all three boxes.

Save & Exit.

 

Congratulations! Now you have your item, action and trigger setup. You should now start to receive email alerts whenever your trigger hits its limit!

 

 

 

 

Some helpful links:

CPU temperature monitoring with Zabbix

https://www.zabbix.com/documentation/current/en/manual/appendix/items/sensor

https://www.zabbix.com/documentation/6.2/en/manual/quickstart/item

https://github.com/B1T0/zabbix-basic-cpu-temperature

https://www.alibabacloud.com/blog/how-to-install-zabbix-monitoring-server-on-ubuntu-20-04_597802

https://www.cyberciti.biz/faq/install-sensors-lm-sensors-on-ubuntu-debian-linux/

Zabbix Alerts: Setup Zabbix Email Notifications & Escalations

Zabbix Alerts: Setup Zabbix Email Notifications & Escalations

Zabbix Monitoring Temperatures Via SNMP

 

 


Discover more from Its_All.Lost

Subscribe to get the latest posts sent to your email.

Leave a Reply