SaltStack, also known simply as Salt, is a handy configuration management platform. Written in Python, it’s open source and allows ITOps teams to define “Infrastructure as Code” in order to provision and orchestrate servers.
But SaltStack’s usefulness is not limited to configuration management. The platform also generates logs, and like all logs, that data can be a useful source of insight in all manner of ways.
This article provides an overview of SaltStack logging, as well as a primer on how to analyze SaltStack logs with Sumo Logic.
Where does SaltStack store logs?
The first thing to understand is where SaltStack logs live. The answer to that question depends on where you choose to place them.
You can set the log location by editing your SaltStack configuration file on the salt-master. By default, this file should be located at /etc/salt/master on most Unix-like systems.
The variable you’ll want to edit is log_file. If you want to store logs locally on the salt-master, you can simply set this to any location on the local file system, such as /var/log/salt/salt_master.
Storing Salt logs with rsyslogd
If you want to centralize logging across a cluster, however, you will benefit by using rsyslogd, a system logging tool for Unix-like systems. With rsyslogd, you can configure SaltStack to store logs either remotely or on the local file system.
For remote logging, set the log_file parameter in the salt-master configuration file according to the format:
<file|udp|tcp>://<host|socketpath>:/.
For example, to connect to a server named mylogserver (whose name should be resolveable on your local network DNS, of course) via UDP on port 2099, you’d use a line like this one:
log_file: udp://mylogserver:2099
Colorizing and bracketing your Salt logs
Another useful configuration option that SaltStack supports is custom colorization of console logs. This can make it easier to read the logs by separating high-priority events from less important ones.
To set colorization, you change the log_fmt_console parameter in the Salt configuration file. The colorization options available are:
'%(colorlevel)s' # log level name colorized by level
'%(colorname)s' # colorized module name
'%(colorprocess)s' # colorized process number
'%(colormsg)s' # log message colorized by level
Log files can’t be colorized. That would not be as useful, since the program you use to read the log file may not support color output, but they can be padded and bracketed to distinguish different event levels. The parameter you’ll set here is log_fmt_logfile and the options supported include:
'%(bracketlevel)s' # equivalent to [%(levelname)-8s]
'%(bracketname)s' # equivalent to [%(name)-17s]
'%(bracketprocess)s' # equivalent to [%(process)5s]
How to Analyze SaltStack logs with Sumo Logic
So far, we’ve covered some handy things to know about configuring SaltStack logs. You’re likely also interested in how you can analyze the data in those logs. Here, Sumo Logic, which offers easy integration with SaltStack, is an excellent solution.
Sumo Logic has an official SaltStack formula, which is available from GitHub. To install it, you can use GitFS to make the formula available to your system, but the simpler approach (for my money, at least) is simply to clone the formula repository in order to save it locally. That way, changes to the formula won’t break your configuration. (The downside, of course, is that you also won’t automatically get updates to the formula, but you can always update your local clone of the repository if you want them.)
To set up the Sumo Logic formula, run these commands:
mkdir -p /srv/formulas # or wherever you want to save the formula
cd /srv/formulas
git clone https://github.com/saltstack-formulas/sumo-logic-formula.git
Then simply edit your configuration by adding the new directory to the file_roots parameter, like so:
file_roots:
base:
- /srv/salt
- /srv/formulas/sumo-logic-formula
Restart your salt-master and you’re all set. You’ll now be able to analyze your SaltStack logs from Sumo Logic, along with any other logs you work with through the platform.
Getting the Most Out of SaltStack Logs is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.
About the Author
Chris Tozzi has worked as a journalist and Linux systems administrator. He has particular interests in open source, agile infrastructure and networking. He is Senior Editor of content and a DevOps Analyst at Fixate IO.
