Today, monitoring and measuring all applications and hosts is crucial for distributed systems. If your organization manages to choose a reliable, efficient, and fast monitoring system with cutting-edge technologies, you will have a battle. The features like quick alerts and integrated visualization tools help fix issues faster. If your organization manages to find one such tool with all these features, you will be able to discover and fix system failures before they lead to actual problems.

There are many cases where bugs and performance issues still make their way out into the production environment. Thus, to prevent your distributed systems from major mishaps, it is best to invest in an open-source tool – Riemann that can be easily configured as per your requirements. Written in Clojure language, it is an event-based monitoring tool that collects events and feeds them into a powerful Stream Processing Language for filtration, processing, and getting summarized.

The Riemann monitoring tool also uses API and supports various plugins. It is fast, highly configurable, and integrates with many popular tools like Grafana, Nagios, Puppet, Chef, Cassandra, Graphite, etc.

Here we will discuss Riemann in detail, covering all its features and benefits. Further, we will discuss the use cases of Riemann as a monitoring tool. Additionally, we will highlight how it can be integrated with other monitoring tools and frameworks.

What is Riemann?

Riemann is a monitoring tool that helps companies manage their servers and applications. It enables employees to keep track of the events that happen on their systems and input them into a stream processing language for alteration and summarization.

The main purpose of Riemann is to automate monitoring and measuring collected events from hosts and applications. It is an open-source monitoring tool written in Clojure, ideal for a highly distributed environment.

Using TCP, Riemann receives events, i.e., structured log entries or ad-hoc events created by other active monitoring systems in the form of protocol buffers. After collecting the events, these are passed into a stream processing language for filtering and aggregating. Later, admins are reported about every exception in your app via email.

Using the Riemann monitoring tool, the organization can track latency distribution and user activity of each web app.

It is a piece of open-source software designed to process events, but before using it, make sure you are familiar with events, streams, and the index.

• Events The events are the fundamental building blocks of Riemann. Riemann receives these events, also referred to as structured log entries, from hosts and applications, which are passed on for processing, alteration, and exporting to other systems. These structured log entries are treated as immutable maps containing multiple “custom” fields with string values.
• Streams The second layer is known as streams. Every time an event is collected, it is appended to one of several streams that aggregate, modify, and escalate events based on different parameters. In simple terms, streams are like pipelines through which all events flow. If the events match the predicate, they are passed to child streams or sub-streams, and if they don’t match, events are no longer filtered or partitioned.
• The Index The index is a table that manages all the information about the service monitored by the tool. The minute you instruct Riemann about the events you want to index, it generates a service for each indexed event. Once a new service is generated, the index will keep an account of the most recent event linked to that service. In simple terms, the index is a table that updates you about the current status of the service. You can also run queries against it via streams or third-party services.

Apart from monitoring and measuring events, you can also integrate the monitoring tool with Librato, Graphite, and Grafana. It is not limited to just those integrations, Riemann has a wide range of integrations that allow the users to monitor their applications or servers.

Before learning about the tools that work great with the Riemann monitoring tool, let us have a look at some of its features and benefits.

Features of Riemann

Riemann is a monitoring tool that provides an overview of the entire system. It is used to monitor application performance and identify any anomalies in the system. The Riemann monitoring tool has a lot of features that make it stand out from other monitoring tools. 

Here are some of the following features that make Riemann one of the best open-source monitoring tools in the market. 

1. Anomalies Detection Riemann is a monitoring tool that helps to detect anomalies in the system and is used for real-time event processing. It provides a single interface for collecting, processing, aggregating, and analyzing metrics from many different sources.
2. Easy to Monitor Systems Another feature of the Riemann monitoring tool is it has a simple interface. As a result, it can be easily used by DevOps engineers who are looking for an easy way to monitor their systems or by operations personnel who want to be alerted when something goes wrong. It is also ideally suited for monitoring a wide range of systems, including healthcare IT, media broadcasting, and industrial control systems.
3. Alert Options The monitoring tool provides quick alerts on discovering anything suspicious or uncertain. The alert feature also includes an actionable step that can be taken to remedy the issue. In fact, the Alert tool can monitor nearly any endpoint, including servers, routers, applications, and more.
4. Distributed Architecture Today, monitoring infrastructure in real-time and getting accurate metrics is quite challenging, but Riemann’s low latency makes it easy for DevOps engineers to get accurate metrics from distributed systems in real-time. Riemann has the ability to monitor large-scale distributed systems in real time.
5. API Support for Data Collection API is a core component of Riemann that plays a key role in accessing data that further aids the organization fuel project development. Riemann uses API to collect data from external sources, applications, and servers.

Advantages of Riemann

• Riemann is quick, powerful, and graphs everything.
• Riemann is a distributed open-source tool written in Clojure, and its syntax is concise and extendable.
• Riemann is flexible and gives you the ability to adapt to complex situations.
• It makes it for engineers to quickly discover the source of errors and performance bottlenecks in production.
• Riemann offers quick alerts and notifies users via phone, SMS, and email.
• Supports integration with PagerDuty for SMS or phone alerts.
• Makes outages visible to engineers within milliseconds.
• It has the capability of handling millions of events per second.
  
• Allows users to identify hotspots and unbalanced loads instantly.
• Allows forwarding event stream to Graphite, Libarto, and other tools.
• Helps monitor the latency distribution for each web app.
• Allows users to throttle or roll up multiple events in a single message.
• Supports low latency and is ideal for highly distributed, scalable systems.
• Supports multiple client languages, including Clojure, Go, Java, Python, Graphite, etc.
• Highly Configurable, Reliable, and Scalable monitoring tool.

Tools that work with Riemann

Riemann can be integrated with various databases and monitoring tools. Check out some of the tool that work great with Riemann:

1. Cassandra An open-source distributed database designed to store and manage large data sets across multiple data centers. Cassandra is a highly scalable and fault-tolerant database system that helps organizations maintain a quick response time for every system. A java tool was written by Blue Mountain Capital that helps developers keep track of Cassandra by emitting metrics as Riemann events.
2. collectd A Unix daemon used by organizations for collecting stats, analyzing, and discovering system and application performance. It is a free-to-use open-source computer software written in C language. Compared to other popular software, it is quite easy to understand and port. It also supports various plugins, including the Write Riemann plugin that stores a value in Riemann and uses it for querying, manipulating, and visualizing.
3. Graphite A monitoring tool that helps collect and visualize time-series data on a massive scale. It can be easily configured and used for monitoring the behavior and patterns of each system and application. Most companies use Graphite for monitoring data over time. It also supports integration with Riemann. Riemann stores and queries all events transferred via carbon’s plaintext protocol.
4. Ganglia A distributed monitoring system that uses XML for data representation, RRD for data storage and visualization, and XDR for data transport. It is an open-source project used by DevOps engineers for high-performance computing systems and gaining high concurrency. Using the powerful event stream processor (Riemann), DevOps engineers can also get information related to threshold and alerts on Ganglia metric data at scale.
5. JMX Organizations that use Java Management Extensions technology (JMX) are able to better manage their JVM resources and gain deeper insight into their application. JMX monitoring solutions also provide real-time visibility into custom metrics. The Clojure fork of riemann-jmx helps monitor several JVMs and send Hbase metrics to Riemann.
6. HBase A distributed, open-sourced, non-relational database that also supports integration with Riemann. HBase is mostly run by organizations when they want read/write access to big data in real time. It keeps you updated about the average load, the total number of region servers available, requests, etc. With Riemann-hbase, users can transfer most of the JMX metrics to Riemann from an HBase server.
7. Logstash Αn open server-side data processing pipeline that makes use of Elastic Stack monitoring features to get a deeper insight into the current status and health of running Logstash instances. It also supports various plugins, such as the Riemann output plugin that aids in transferring log events to Riemann.
8. MySQL Α relational database management system designed to access and process stored data. It is a lightweight, reliable database released under the GNU GPL (General Public License) license. One of the popular databases used by many high-profile brands and organizations, including Facebook, GitHub, NASA, etc. The tool also supports integration with Riemann to keep track of the MySQL server’s replication status.
9. Nagios Α continuous monitoring tool used by organizations to monitor systems, networks, and IT infrastructure. It is free software that keeps track of all hosts and services and updates instantly when an event of failure occurs. It is an enterprise-grade monitoring tool that uses Riemann-ocsp (gateway) to send events to Riemann.
10. Puppet Α software configuration management tool used for pulling the strings attached to multiple application servers at once. The tool is highly compatible with Linux and Windows, but can also be used on Cisco switches, and macOS servers. Using the ICMP ping and Puppet-riemann, users can track hosts from puppet and send reports to Riemann

Use Cases

If your organization is planning to set up a distributed environment that monitors complex data in real time, it is best to invest in Riemann. 

Riemann for DevOps Engineers 

Using its event-based polling and stream processing technology, it enables organizations to collect data from hosts and applications quite conveniently. Also, the low latency characteristics supported by the tool help DevOps engineers to process real-time data and get warnings in case of a system failure. DevOps engineers can also use the open-source tool for manipulating its usage as per the infrastructure design.

How to Install and Configure Riemann on Ubuntu

In this section, we will show you how to install Riemann and stream events on the Riemann dashboard.

Install Java and Other Dependencies

Riemann is based on Java. So you will need to install Java to your system. You can install Java 8 by running the following command:

apt-get install openjdk-8-jdk

Once the Java is installed, you can verify the Java version using the following command:

java -version

You should see the following output:

openjdk version "1.8.0_342"
OpenJDK Runtime Environment (build 1.8.0_342-8u342-b07-0ubuntu1~20.04-b07)
OpenJDK 64-Bit Server VM (build 25.342-b07, mixed mode)

Next, you will need to install some required dependencies to your server.

apt install git curl autoconf bison build-essential libssl-dev libyaml-dev libreadline6-dev zlib1g-dev libncurses5-dev libffi-dev libgdbm6 libgdbm-dev libdb-dev -y

Once all the dependencies are installed, you can proceed to the next step.

Install Riemann Server

By default, the Riemann package is not included in the Ubuntu default repository, so you will need to download it from the internet. You can download it with the following command:

wget http://aphyr.com/riemann/riemann_0.2.10_all.deb

Next, install the downloaded package using the following command:

dpkg -i riemann_0.2.10_all.deb

Once the Riemann is installed, edit the Riemann configuration file:

nano /etc/riemann/riemann.config

Find the following line:

(let [host "127.0.0.1"]

And, replaced it with the following line:

(let [host "your-server-ip"]

Save and close the file then start the Riemann service with the following command:

service start riemann

Next, check the status of the Riemann with the following command:

service riemann status

You will get the following output:

● riemann.service - LSB: Riemann server
     Loaded: loaded (/etc/init.d/riemann; generated)
     Active: active (running) since Thu 2022-09-08 09:50:29 UTC; 36s ago
       Docs: man:systemd-sysv-generator(8)
    Process: 16595 ExecStart=/etc/init.d/riemann start (code=exited, status=0/SUCCESS)
      Tasks: 22 (limit: 2347)
     Memory: 139.5M
     CGroup: /system.slice/riemann.service
             └─16617 java -cp /usr/share/riemann/riemann.jar: riemann.bin start /etc/riemann/riemann.config
Sep 08 09:50:29 ubuntu systemd[1]: Starting LSB: Riemann server...
Sep 08 09:50:29 ubuntu systemd[1]: Started LSB: Riemann server.

Install Riemann Dashboard and Client

Next, you will need to install the Riemann dashboard and client package to access the Riemann over the web.
First, install the Ruby package with the following command:

apt install ruby-full

Next, install Riemann dashboard and client package with the following command:

gem install riemann-tools riemann-dash riemann-client -y

Once all the packages are installed, you can proceed to the next step.

Create a Systemd Service File for Riemann Dash and Riemann Health

First, create a systemd service file for Riemann health with the following command:

nano /etc/systemd/system/riemann-health.service

Add the following lines:

[Unit]
Description=Riemann Health
After=network.target

[Service]
ExecStart=/usr/local/bin/riemann-health
PidFile=/var/run/riemann-health.pid
Restart=on-failure

[Install]
Alias=riemann-health.service

Save and close the file, then create a systemd service file for Riemann dash with the following command:

nano /etc/systemd/system/riemann-dash.service

Add the following lines:

[Unit]
Description=Riemann Dashboard
After=network.target
[Service]
ExecStart=/usr/local/bin/riemann-dash
PidFile=/var/run/riemann-dash.pid
Restart=on-failure
[Install]
Alias=riemann-dash.service

Save and close the file, then reload the systemd daemon to apply the changes:

systemctl daemon-reload

Next, start the Riemann dash and health service with the following command:

systemctl start riemann-dash
systemctl start riemann-health

You can now check the status of both services with the following command:

systemctl status riemann-dash riemann-health

You will get the following output:

● riemann-dash.service - Riemann Dashboard
Loaded: loaded (/etc/systemd/system/riemann-dash.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2022-09-08 09:53:41 UTC; 27s ago
Main PID: 16692 (riemann-dash)
     Tasks: 1 (limit: 2347)
     Memory: 29.2M
     CGroup: /system.slice/riemann-dash.service
             └─16692 /usr/bin/ruby2.7 /usr/local/bin/riemann-dash
Sep 08 09:53:41 ubuntu systemd[1]: Started Riemann Dashboard.
Sep 08 09:53:42 ubuntu riemann-dash[16692]: [2022-09-08 09:53:42] INFO  WEBrick 1.3.1
Sep 08 09:53:42 ubuntu riemann-dash[16692]: [2022-09-08 09:53:42] INFO  ruby 2.7.0 (2019-12-25) [x86_64-linux-gnu]
Sep 08 09:53:42 ubuntu riemann-dash[16692]: == Sinatra (v1.4.8) has taken the stage on 4567 for development with backup from WEBrick
Sep 08 09:53:42 ubuntu riemann-dash[16692]: [2022-09-08 09:53:42] INFO  WEBrick::HTTPServer#start: pid=16692 port=4567

● riemann-health.service - Riemann Health
Loaded: loaded (/etc/systemd/system/riemann-health.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2022-09-08 09:53:56 UTC; 25s ago
Main PID: 16711 (riemann-health)
     Tasks: 1 (limit: 2347)
     Memory: 21.7M
     CGroup: /system.slice/riemann-health.service
             └─16711 /usr/local/bin/riemann-health
Sep 08 09:53:56 ubuntu systemd[1]: Started Riemann Health.

By default, Riemann service listens on localhost on port 4567. You can check it with the following command:

ss -antpl | grep riemann

You will get the following output:

LISTEN   0        4096                 127.0.0.1:4567            0.0.0.0:*       users:(("riemann-dash",pid=16692,fd=6))                                        
LISTEN   0        4096                     [::1]:4567               [::]:*       users:(("riemann-dash",pid=16692,fd=5))   

Configure Nginx as a Reverse Proxy for Riemann

Riemann listens on localhost. So you can not access it from the remote machine. In this case, you will need to configure the Nginx as a reverse proxy to access the Riemann.

First, install the Nginx package using the following command:

apt install nginx -y

Once the Nginx is installed, edit the Nginx default virtual host file:

nano /etc/nginx/sites-available/default

Remove all lines and add the following configurations:

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    location / {
    proxy_pass http://localhost:4567;
    proxy_pass_request_headers      on;
    }

}

Save and close the file, then verify the Nginx for any syntax error:

nginx -t

If everything is fine, you should get the following output:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Next, restart the Nginx service to apply the changes:

systemctl restart nginx

You can now verify the Nginx status using the following command:

systemctl status nginx

You should see the following output:

● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-09-08 09:56:35 UTC; 17s ago
       Docs: man:nginx(8)
    Process: 17265 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 17281 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 17282 (nginx)
   Tasks: 2 (limit: 2347)
   Memory: 2.6M
   CGroup: /system.slice/nginx.service
             ├─17282 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             └─17283 nginx: worker process

Sep 08 09:56:35 ubuntu systemd[1]: Starting A high performance web server and a reverse proxy server...
Sep 08 09:56:35 ubuntu systemd[1]: Started A high performance web server and a reverse proxy server.

Access Riemann Dashboard

Now, open your web browser and access the Riemann dashboard using the URL http://your-server-ip:4567. You should see the Riemann dashboard on the following screen:

Now, press CTRL and click on the top area of the dashboard, then press the “e” on the keyboard. You should see the new popup screen:

Select Grid from the drop-down, then type the “true” in the Query box and click on the Apply button. You should see all streams flowing into Riemann dashboard as shown below:

Conclusion

Compared to other popular tools available in the market, Riemann is an easy-to-understand and open-source monitoring tool compatible with businesses of all sizes. Also, it is flexible and scalable and aids organizations in monitoring and measuring collected events from several hosts and applications. Another feature that makes it one of the best tools is it provides real-time failure alerts that help organizations maintain the system’s health and prevent it from system-down situations.

It supports API, and plugins, and integrates with various other tools like Graphite, Ganglia, HBase, Cassandra, Puppet, etc. Another best part about Riemann is it has the capability of handling millions of events per second and displays outages to DevOps engineers within milliseconds. No doubt, Riemann is one of the best open-source monitoring tools used for real-time event processing and detecting anomalies.

Check out its above-listed features, benefits, and more information to learn how it can be advantageous for your organization.