Prometheus exporter

[alanhamlett]

Now that #108 provides queue metrics, next we should build a Prometheus exporter to aggregate and monitor these metrics.

I guess that would be a daemon that could be installed on a single machine to export Task.task_count_from_queue and make them available for Prometheus.

It's common for the Prometheus exporter to be tightly integrated into a product, for example:
https://www.cockroachlabs.com/docs/stable/monitor-cockroachdb-with-prometheus.html

[thomasst]

@jkemp101 is the perfect person to talk about this.

[jkemp101]

I think there are three different areas for tasktiger metrics. The first bullet is pretty easy. I've only been working slowly on the other two.

• General queue metrics: This is pretty easy to do already using the output from get_queue_stats(). We could include a daemon script to setup the prometheus side but not sure it is worth adding prometheus as a project dependency just for that. Maybe include something in an examples or contrib directory?
• Internal task metrics: Getting metrics from the forked child process for each task is a little tricky since they are short lived. I have been looking at having the child process flush stats to disk before exiting so that a separate process can grab them and export via prometheus. The worker process could also grab these metrics if we decide to include prometheus support into the main codebase.
• Individual worker stats: It could be nice for each worker to export some metrics like the number of tasks executed and number of failed tasks. I'm not sure how important this really is if you have logging and other instrumentation.

[alanhamlett]

Maybe include something in an examples or contrib directory?

A contrib folder for a prometheus exporter daemon example sgtm. For task and worker metrics, would an example task decorator be sufficient? That's how I currently export task timing and success/failure metrics from all tasks.

[jkemp101]

I think an example using a decorator would be good first step to show how to export those metrics. Are you just starting the prometheus http server in a separate thread from the main tasktiger process to export the metrics?

[superbogy]

Is it in progress?

[superbogy]

I think you can refer to the practice of the lib of php prometheus client. php-fpm worker is a short lived process too. I think it is a good way to flush stats to redis in stead of disk. @jkemp101