Monitoring HAProxy logs and metrics with Sumo Logic

HAProxy is one of the world’s most innovative and highest-performing load balancing solutions. The load balancer is critical for enabling high availability and supporting the dynamic scaling of infrastructure within modern applications. Because of its importance, engineers need tools that can quickly and effectively diagnose any problems with the load balancer if they arise.

HAProxy produces highly detailed logs in real-time that contain data about incoming requests, their assigned routing, and the status of downstream requests. This article will explain how to use Sumo Logic’s platform to import and analyze HAProxy logs. The Sumo Logic platform provides an easy-to-understand dashboard as well as advanced analysis and alerting tools that facilitate effective troubleshooting and a thorough understanding of the HAProxy load balancer’s performance.

An Architectural Overview

Before we start, let’s take a look at what we’re going to be setting up and how it all connects. If you’re following along and would like to try these steps out, you’ll need an HAProxy installation and a Sumo Logic account. You can sign up for a free trial of Sumo Logic here. You’ll just need to verify your email address, and then you’ll be ready to go.

Sumo Logic supports the collection of metrics and logs from HAProxy. It uses the HAProxy Telegraf plugin running within the HAProxy environment to collect metrics. The Sumo Logic Collector, which is also installed within the HAProxy environment, collects logs.

In this article, we will explore the process of collecting logs in a non-Kubernetes environment. If you would like to enable log collection in a Kubernetes environment, you can follow the instructions here. If you want to collect metrics as well, you can find detailed instructions at the links below:

• Metrics Collection within a Kubernetes Environment

• Metrics Collection within a Non-Kubernetes Environment

Configure HAProxy Logging

For this example, we’ve deployed an EC2 instance with HAProxy into an Amazon Web Services (AWS) account. We’ve configured round-robin load balancing between two additional EC2 instances, each running a simple web server. The frontend and backend configurations that we’re using in haproxy.conf are shown below for informational purposes.

The first step is to log in to the HAProxy server and edit the haproxy.cfg file. The default location is /etc/haproxy/haproxy.cfg. We will add a directive under the global section to direct logs to the syslog server listening on port 514.

Next, we’ll edit the configuration to direct rsyslog to listen on port 514. We added the configuration to a new file in /etc/rsyslog.d/haproxy.conf. This configuration opens up UDP port 514 to receive the log files from HAProxy and creates two output files based on the severity of the logs received.

Finally, we restart haproxy and rsyslog to enact the changes.

At this point, you can make a few HTTP requests to the HAProxy server, and you should see new logs being added to the haproxy-traffic.log. Now we’re ready to install the Collector, which will transmit logs to your Sumo Logic account.

Installing the Collector

We’re going to install an Installed Collector through the command line. (There are other installation options available if you want to do it a different way.) First, we’ll download the Collector while we’re still logged in to the HAProxy instance, and we’ll update its permissions with the command shown below:

While that’s downloading, you should log in to your Sumo Logic account and create a token. From the left-hand navigation panel, select Administration and then Security. Next, click on the Add Token button and enter a token name. When you click Save, your new token should appear. You can click on the token to view it. We’ll be using the Token String to install the Collector.

Next, let’s install the Collector. You’ll want to replace YOUR_TOKEN_STRING_HERE with your token string, and then update the name of the log file based on which logs you want to send to your Sumo Logic account. To do this, enter the following command:

This command installs the Sumo Logic Collector. Once the Collector is installed, we need to set up the source in our Sumo Logic account. There are two different sources that we can use: we can either pull the logs from the local files or use the Linux syslog as the source. For this example, we’re going to use the haproxy_traffic.log file as the source. If you’d like to try the syslog approach, you can find comprehensive instructions in the Sumo Logic Documentation.

Configuring the Source From Sumo Logic

Now, go back to your Sumo Logic account, and from the left-hand navigation panel, select Manage Data, and then Collection. Next, click on the Collection tab at the top of the main panel. Because we used an access token, the Collector is already associated with our account and ready to be added as a source. Your collection panel should look similar to the one shown below:

Click on the Add link for the Collector that you just installed and choose the Add Source option. The application will present you with various sources that you can add. In this case, we’ll select the Local File option under the File Sources section.

We’ll enter the name of our new source (we used HAProxy) and the name of the log file from which we’ll be retrieving data (which is /var/log/haproxy_traffic.log in this case). You can also retroactively collect logs from past events.

Enter the Source Host (localhost) and Source Category. We used HAProxy/Logs as the source category for our logs.

Next, we need to add four fields to the source. Sumo Logic will add these fields to incoming events to make it easier to filter the logs. You add them by clicking the +Add Field link.

Last but not least, check the following items under Advanced Options for Logs:

• Check Enable Timestamp Parsing.

• For Time Zone, select ignore time zone from log file and instead use, then choose the time zone for your cluster.

• For Timestamp Format, select automatically detect the format.

• The default value of UTF-8 works well for Encoding.

• Enable Multiline Processing and select the Infer Boundaries option.

Finally, click Save to complete the configuration for your HAProxy source.

Once saved successfully, you’ll see the configured source listed under the HAProxy Collector on the Collection tab.

Exploring the Sumo Logic Log Interface

If you return to the Collection page, you’ll be able to see the number of messages that the Collector has received from the load balancer as well as a helpful graphic showing the traffic for the past hour.

The easiest way to access the logs for this Collector is to hover over the name of the source that we just created and click on the Open in Log Search icon.

The log search page displays a visual representation of all received events. Starting at the top right corner of the page, you can view different time frames. On the left-hand side of the page, you can add and remove fields from the display.

The most powerful feature of this page is at the top: there is a text field that allows you to build custom queries to filter the log results that are displayed. This field uses the Sumo Logic Search Query language, which is explained very well in the Sumo Logic Documentation. This language goes beyond filtering, as it includes options for aggregating and performing complex analyses of event logs.

Two other valuable features of this page are the Live Tail and Add Monitor options which appear below the search box. Live tail allows you to view log events as the system receives them, and it’s handy if you want to view the current logs without logging in to the source server itself. You can also add a monitor that will automatically create an alert or event if a user-defined condition occurs within the log data. Finally, you have options to Save and Share queries that you’ve constructed.

Learning More

In this article, we’ve shown you how to connect the logs from HAProxy to your Sumo Logic account, as well as how to view, filter, and begin analyzing them. This setup is useful for small prototypes like the example we used above, and it becomes increasingly helpful and essential as your environment grows larger and more complex.

Still, we’ve hardly scratched the surface of what Sumo Logic can help you accomplish with your log data. If you learn best when you can experiment and explore a new system, the free trial offered by Sumo Logic is the perfect way to experience the power of their platform. As we mentioned above, the Sumo Logic Documentation is well-organized and provides detailed instructions that will help you get the most out of your trial. You can also reach out through the embedded chat feature on the Sumo Logic website or contact their support staff directly for ideas and tips on how you can leverage their platform for your specific use cases.