[arm-allstar] Fwd: prometheus/grafana measurements

[Stu2]

Thanks for the notes, John.

I'm doing something similar. 
https://www.stu2.net/wiki/index.php/Instrumenting_the_Repeater This work 
was done before prometheus was available. I'm trying to find time to 
rethink this architecture and make it more direct like yours.  (i.e. 
fewer steps.) The actual architecture for our site is more complicated. 
In addition to the published architecture, we ship data via mqtt to a 
cloud instance, too. That way, accessing the grafana graphs go faster. 
The repeater has a DSL line and it's very slow.

I like the idea of counting keyups and transmit time. That would be 
interesting to see. Our power meter (ESP32 based) readings are graphed 
over time, so you get a sense of how much the repeaters are used. Also 
thanks for access to the python script for generating prometheus metrics.

I'm also running the allstar node for our repeater. Your work may come 
in handy to add to our grafana dashboards.

73

Stu2, W7IY

On 12/30/22 16:07, John Hogenmiller via ARM-allstar wrote:
> (I sent this earlier, but did not see it show up in the archive.. I had a
> screenshot attached which might have been too large.  Resending without the
> screenshot, apologies if this is a duplicate).
>
>
> Happy New Years Eve Eve.
>
> If none of these tools (Grafana, Prometheus, etc) are familiar to you, this
> probably won't be that interesting. If you have familiarity with these
> tools or want to learn, I think this is an achievable addition.
>
> I have had some extra m monitoring setup on my repeaters using
> telegraf/influxdb/grafana, and I've been meaning to share it out. However,
> it was a bit complex.  For instance, I had to compile the telegraf client
> for Arch/Arm, and you need a separate PC to run your own stack.  This week
> I redid all my local work to use Grafana Cloud and Grafana Agent, which has
> an armv7 binary already compiled. Instead of influxdb, I switched over to
> generating prometheus metrics, which can be published to a Prometheus
> instance, such as the free one from Grafana cloud.
>
> My notes and example files are here:
> https://gist.github.com/ytjohn/40ecfffb349a4c19bf8ee99b89210ecc
>
> Here is a screenshot of my setup showing the last 3 hours of activity. The
> Grafana free plan lets you keep 14 days of activity.  If you do self host
> grafana, you can store years worth of data, and you can also make these
> dashboards public. But the free cloud account is a good way to get started.
>
> I have a few more things I'd like to do. I am converting from a JSON/influx
> data to prometheus. If an asterisk image is unreachable, I should change
> "up" to 0 instead of the current php "die" statements. Grafana could then
> alert on unreachable nodes. I'd also like it to not need the node
> parameter, but rather figure it out from reading allmon.ini.  I also want
> to read the astdb to get the node description/frequency/pl information.
>
> Let me know if this is of interest, I can provide updates as I improve the
> script, dashboards, etc. I'll also roll back to a self-hosted instance at
> some point, which I do using docker/docker-compose (but not on the hamvoip
> nodes).  Is there any other stats people would like to see?  For instance,
> I'm also looking to add the lag/jitter from "iax2 show channels".
>
> 73s,
> John/KB3DFZ
> _______________________________________________
>
> ARM-allstar mailing list
> ARM-allstar at hamvoip.org
> http://lists.hamvoip.org/cgi-bin/mailman/listinfo/arm-allstar
>
> Visit the BBB and RPi2/3/4 web page - http://hamvoip.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: RepeaterArchitecture2022.pdf
Type: application/pdf
Size: 38155 bytes
Desc: not available
URL: <http://lists.hamvoip.org/pipermail/arm-allstar/attachments/20230104/9e642d68/attachment.pdf>