Snap collector

[kindermoumoute]

This is a follow-up of the conversation from Slack with @daniellee @jtlisi @thomastaylor312 about creating a Snap daemon collector. Here is a list of ideas:

• Integrate the collector into Snap.
• Collect tasks hit / miss / fail / state per task.
• Count task metrics (see PoC).
• Collect metrics from Go runtime (e.g. we could use the go-metrics lib)

[daniellee]

While waiting for something better, we are using this collector that I threw together for monitoring snap tasks (the one linked above as PoC):

https://github.com/raintank/snap-plugin-collector-snapstats

It's quite basic and only collects totals. We use the total count for failed tasks for alerting.

Here is what it collects:

• Total number of disabled Snap tasks
• Total number of Running Snap tasks
• Total number of Stopped Snap tasks
• Total sum of the all the snap tasks' hit counts
• Total sum of the all the snap tasks' failed counts

Here is an example of an alert in Grafana:

So that is the use case I am interested in -> monitoring Snap tasks in a production environment.