RRDtool – InfluxDB – Grafana / Read – Write – Monitor

During my first few weeks of working at a TELCO company, I was tasked with handling RRD files. RRD’s (round-robin database) are data storage files containing time-series data with ranging resolutions. Because of the way data is stored in RRD files, the system storage footprint is always constant. I was responsible for reading the data stored in these RRD files, writing them into our company’s InfluxDB and monitoring the data values through Grafana.

Due to the nature of the formatting, directly opening an RRD file will not yield any legible results, as it contains binarized and encrypted text. Utilizing RRDtool, we’re able to read RRD files in a more understandable format. We install this tool via:

sudo apt install rrdtool

After installation, we’re able to get the dump of the RRD as an XML and write it as an output:

rrdtool dump [FILENAME].rrd > test.xml

In my case, I needed to extract the last hour’s values of the AVERAGE cf in the 300-second step. In order to parse the RRD file to retrieve these values, I came up with two methods.

My first method was to parse the XML output itself, and the second was to use the Python binding, python-rrdtool. After experimenting and testing with both, using python-rrdtool ended up being much faster. As it yields far better results, I recommend that anyone wishing to parse RRD files use python-rrdtool.

The other two tools I utilized were InfluxDB and Grafana. 

I was able to set up both of these open-source tools with relative ease thanks to Docker. To obtain their images, simply use the following lines:

docker pull grafana/grafana
docker pull influxdb

After obtaining the images, we run the containers as follows (default options shown below):

docker run -d --name=grafana -p 3000:3000 grafana/grafana
docker run -d --name=influxdb -p 8086:8086 influxdb

Now that both of the containers are up and ready. It’s time to create a new database to insert metrics in. To achieve that, please follow the codes below.

docker exec -it influxdb bash
-->
influx
->
CREATE DATABASE pings;
USE pings;
CREATE USER data WITH PASSWORD 'data';
GRANT ALL ON pings TO data;

Initialization is completed. I have a table named as pings. I can reach it with the credentials that username:dataand password:data. Now, I may use the Python script that reads RRD file and insert the extracted values into InfluxDB.

I implemented other components to this stack, but due to confidentiality reasons, I’m unable to share the code itself as of now. But I am able to offer some tips that will definitely make your life easier when faced with these problems that I ran into.

1. The RRD file doesn’t provide an explicit timestamp when parsing via Python. Therefore, you need to extract the starting or ending timestamp from the XML data. From there, you’re able to create a range by incrementing the extracted timestamp by the time step. (i.e. if your step is 300, then add/subtract 300 seconds from your timestamp incrementally)
2. Convert timestamps to UTC to produce a better visualization and support various data source locations. After inserting into InfluxDB, you’re able to set timezones on Grafana to prevent any detrimental changes to your data.
3. Write your insertion code as a function in order to use it frequently as needed.

After inserting successfully, let’s visualize our data points from InfluxDB on a Grafana dashboard. 

We’re almost done with our RRDtool – InfluxDB – Grafana stack. Lastly, create a dashboard by going to Dashboards > Manage from the left sidebar and add a panel. 

Select your query parameters (don’t forget to indicate your timezone)

As you can see, the data comes from RRD files processed by a Python script and is then inserted to InfluxDB. From there, the InfluxDB data is shown on our Grafana dashboard we just made. If you want streamed data, you’ll need to set a cronjob to periodically run your Python script or daemonize your script via some time-related rules and Supervisord.

There is definitely room for improvement, but fundamentally this is how I approached my monitoring stack. 

Additionally, I incorporated Tableau into this stack to generate reports and further visualization. I plan to write more about this in my next blog post, which may include more details from this post. 

UPDATE: After sharing this post on Twitter, it was recognized by the official InfluxData account and that made me so proud.

I’m glad to mention that this integration was