All you need to know about HAProxy log format

HAProxy is one of the fastest and most widely-used load balancing solutions available today. If you’re already using HAProxy, or if you’re considering using HAProxy in your environment, then this is a great place to start. On this page, we discuss HAProxy logging and why logging is such a vital component of the load balancer implementation. We then take a deep dive into the logging offered by HAProxy. Finally, you’ll read about working with the HAProxy logging format and how you can configure the logging to suit your needs better.

Why Load Balancer Logging is Critical

When connecting your application to the outside world, the load balancer is an essential component. In its simplest form, the load balancer is responsible for analyzing all inbound requests, determining their validity, and then passing them to the appropriate destination. The load balancer does more than just route traffic; it ensures that requests are dispersed evenly between all healthy instances of your application, preventing requests from reaching unhealthy instances and ensuring high availability.

As the critical component that handles all requests, the load balancer often becomes the focal point of investigations relative to connectivity issues. Timely and accurate logs are an essential part of any investigation. HAProxy logs can provide unique insights into all aspects of failed and successful requests, including performance and routing information.

HAProxy Logging 101

HAProxy provides extremely detailed logging with millisecond precision to provide actionable data about your infrastructure at a macro and a micro-level. Let’s look at some of the information you can glean from an HAProxy log and how each data point helps provide comprehensive observability for your application.

Traffic Metrics

Understanding the volume and size of traffic moving through your application can provide essential guidance about required resources downstream, and critical insights about how, why, and when failures might occur.

Request and Response Details

If you’re investigating error conditions that affect targeted users or regions, inspecting the headers, payloads, and status codes of requests and responses can help you isolate the exact cause of a problem and provide detailed information about the behaviors that clients may experience.

Routing Decisions

HAProxy logs also include information about each request’s decisions, such as the identity of which downstream target to which the request was routed, and filtering applied to requests and information about persistence between requests from the same client.

Error Tracing

Finally, the logs contain information that engineers can use to identify where request life cycle failures occur. The logs can also include information relating to active sessions and their termination status.

Default and Pre-Configured Logging Formats

HAProxy derives the logging format from settings within the HAProxy configuration file. You can use the default logging format by excluding the option directive from the configuration, or setting one of two pre-configured formats.

• For TCP or Layer 4 operation mode, the directive to include is option tcplog.

• For HTTP or Layer 7 operation mode, the configuration directive is option httplog.

We’ll break down an example of each log format below.

These example logs are from an instance of HAProxy version 2.4.2, installed on an AWS instance running AWS Linux 4.14 and configured to manage traffic between two HTTP servers within the same subnet. The logs were captured using RSyslog and stored locally on the host server.

Default Log Format (No option configured)

TCP / Layer 4 Log Format (option tcplog)

HTTP / Layer 7 (option httplog)

Customizing the HAProxy Log Format

While the default and pre-configured log formats available in HAProxy can provide substantial insights, you might have noticed an absence of metrics and log elements related to activities supporting SSL encryption, headers, and the payload itself. While these elements can significantly increase the amount of data in the log, they can also help resolve problems.

If you want to use a customized log format, you can replace the option direction with a log-format directive, followed by a string indicating the contents and format of the desired log messages. If we were to replicate the HTTP log format described above, the directive would be as follows:

A complete reference of available options and their correct usage is listed in the HAProxy Documentation. Some of the available options that you may want to include to support advanced troubleshooting include:

• Connection handshake time (%Th)

• SSL ciphers and versions (%sslc / %sslv)

• Request headers (%hr or %hrl for CLF formatting)

• Response headers (%hs or %hsl for CLF formatting)

• The full HTTP request (%r)

Managing Log Volume at Scale

HAProxy is one of the most performant load-balancing solutions currently available, and as such, can reasonably handle thousands of requests each second. While HAProxy can easily create a log entry for each request, you might find yourself in a position where you need to explore creative options to support the storage of these logs. Identifying alternate options is critical if your team prototyped an HAProxy installation that directed the logs to disk storage.

In addition to the volume of log data, you also need to search the logs for anomalies, or even better, automate anomaly detection to alert your team when problems occur. Partnering with a log management solution partner such as Sumo Logic will help resolve both of these problems and provide your teams with fast and reliable tools to support monitoring efforts and analysis when issues arise. You can learn more about integrating HAProxy with Sumo Logic here and sign up for a free trial to try it out for yourself.