[arm-allstar] Fwd: prometheus/grafana measurements

[John Hogenmiller]

(I sent this earlier, but did not see it show up in the archive.. I had a
screenshot attached which might have been too large.  Resending without the
screenshot, apologies if this is a duplicate).


Happy New Years Eve Eve.

If none of these tools (Grafana, Prometheus, etc) are familiar to you, this
probably won't be that interesting. If you have familiarity with these
tools or want to learn, I think this is an achievable addition.

I have had some extra m monitoring setup on my repeaters using
telegraf/influxdb/grafana, and I've been meaning to share it out. However,
it was a bit complex.  For instance, I had to compile the telegraf client
for Arch/Arm, and you need a separate PC to run your own stack.  This week
I redid all my local work to use Grafana Cloud and Grafana Agent, which has
an armv7 binary already compiled. Instead of influxdb, I switched over to
generating prometheus metrics, which can be published to a Prometheus
instance, such as the free one from Grafana cloud.

My notes and example files are here:
https://gist.github.com/ytjohn/40ecfffb349a4c19bf8ee99b89210ecc

Here is a screenshot of my setup showing the last 3 hours of activity. The
Grafana free plan lets you keep 14 days of activity.  If you do self host
grafana, you can store years worth of data, and you can also make these
dashboards public. But the free cloud account is a good way to get started.

I have a few more things I'd like to do. I am converting from a JSON/influx
data to prometheus. If an asterisk image is unreachable, I should change
"up" to 0 instead of the current php "die" statements. Grafana could then
alert on unreachable nodes. I'd also like it to not need the node
parameter, but rather figure it out from reading allmon.ini.  I also want
to read the astdb to get the node description/frequency/pl information.

Let me know if this is of interest, I can provide updates as I improve the
script, dashboards, etc. I'll also roll back to a self-hosted instance at
some point, which I do using docker/docker-compose (but not on the hamvoip
nodes).  Is there any other stats people would like to see?  For instance,
I'm also looking to add the lag/jitter from "iax2 show channels".

73s,
John/KB3DFZ