Post Docker containers statistics to Slack

In this post we are going to see how to monitor Docker containers resource usage statistics and send alarm notifications to a Slack channel.
The Docker Engine allows you to see these statistics running the docker stats command. It returns a live data stream for running containers.

Here you can find the official Docker documentation of the command: Docker stats
If you are wondering what Slack is, let me just say that it is an instant messaging and collaboration system based on channels.
You can read more here: Slack

We are going to monitor the container’s resources using a Python script. There are a lot of container management systems but I found some of them too complicated or not very useful. If you want something light, easy and open-source I suggest you Portainer.io (I am going to write a post about it).

We are going to use the docker-py Python client library to connect to the Docker Remote API.
Here you can find the library Github repository: docker-py.
If you do not want to use Python, here you can find a list of client libraries for other programming languages: Docker Remote API client libraries

Install the library with pip:

Connect to the Docker Deamon and to the Docker Remote API (specify the Docker server address):

For each running containers we can now stream the resource statistics.
To list all the running containers use the .containers.list() method.

To stream the statistics for a given container use the client.stats method. It takes the container name as argument and returns a generator (Wiki Python Generator).

Here you can find the official documentation of Low-level API: docker-py low-level API.
The stats method returns a JSON with the following format (note the CPU and memory usage information):

We are going now to analyze the resource usage statics and eventually notify an alarm message to Slack (if the usage of some resources exceed our thresholds).
I am not going to post here how to extract the resource usage from the JSON object, but you can find the full code in this Github repository: mz1991/docker-stats-slack.

To send the notification to Slack we will use the Slack Webhook integration. Webhooks are a simple way to post messages from external sources into Slack. They make use of normal HTTP requests with a JSON payload that includes the message text and some options.
You can read more here: Slack Incoming Webhooks.
I assume you configured the Slack Incoming Webhook integration for your Slack team and you have the Webhook URL.
To configure the Incoming Webhook integration for your Slack team, you can use the following URL: https://[your_slack_team].slack.com/apps/A0F7XDUAZ-incoming-webhooks

With the Webhook integration we do not need any Slack library, to post a message to the Slack channel we just need to post (HTTP post) a message to the Webhook URL endpoint.

The format of the JSON we are going to post is the following:

You need to specify the channel where you want to post the message, the display name of the user with an emoji and the text of the message.

You can find the Webhook documentation here: https://api.slack.com/incoming-webhooks
and the list of available emoji here: Slack Emoji Cheat Sheet

To post the message we are going to use the Request class (built-in in the Python urllib3 module).

The posted message will look like this:

message_to_slack
We saw how to stream the containers statistics and how to post an alarm message to a Slack channel.
I build a Python script that uses a set of environment variables for the Slack channel configuration and the resource usage thresholds.

These are the environment variables needed:

  • SLACK_WEBHOOK_URL: the webhook url for your Slack team
  • SLACK_CHANNEL: the channel id (where the message will be posted)
  • SLACK_USERNAME: the username for the incoming messages
  • SLACK_EMOJI: the emoji for the incoming messages
  • MEMORY_PERCENTAGE: maximum percentage of RAM memory used for each container. When the percentage of used memory will exceed this threshold, an alarm will be posted to the Slack channel
  • CPU_PERCENTAGE: maximum percentage CPU usage for each container. When the CPU percentage usage will exceed this threshold, an alarm will be posted to the Slack channel
  • SLEEP_TIME (seconds) : interval between each message posted to Slack. Number of seconds between messages, unique for container.

To run the script use the following commands:

All the code from this post can be found in this Github repository: mz1991/docker-stats-slack.
Feel free to fork it and to add further features.