Troubleshooting Apps and Infrastructure Using Puppet Logs

If you’re working with Puppet, there’s a good chance you’ve run into problems when configuring a diverse infrastructure. It could be a problem with authorization and authentication, or perhaps with MCollective, Orchestration or Live Management. Puppet logs can provide a great deal of insight into the status of your apps and infrastructure across the data center.

Knowing where to look is just the first step. Knowing what to look for is another matter. Here’s a cheat sheet that will help you identify the logs that are most useful, and show you what to look for. I’ll also explain how to connect Puppet to a log aggregation and analytics tool like Sumo Logic.

Where are Puppet Logs Stored?

The Puppet Enterprise platform produces a variety of log files across its software architecture. This Puppet documentation describes the file path of the following types of log files:

• Master Logs: Master application logs containing information such as fatal errors and reports, warnings, and compilation errors.
• Agent Logs: Information on client configuration retrieved from the Master.
• ActiveMQ Logs: Information on the ActiveMQ actions on specific nodes.
• MCollective service logs: Information on the MCollective actions on specific nodes.
• Console logs: Information around console errors, fatal errors and crash reports.
• Installer logs: Contains information around Puppet installations, such as errors occurred installation, last installation run and other relevant information.
• Database logs: Information around database modifications, errors, etc.
• Orchestration logs: Information around orchestration changes.

The root of Puppet log storage is different depending on whether Puppet is running in a Unix-like system or in a Windows environment.

For *nix-based installs, the root folder for Puppet is /etc/puppetlabs/puppet.

For Windows-based installs the root folder for Puppet is: C:\ProgramData\PuppetLabs\puppet\etc for all versions of Windows server from 2008 onwards.

Modifying Puppet Log Configuration

The main setting that needs to be configured correctly to get the best from Puppet logs is the log_level attribute within the main Puppet configuration file. The log_level parameter can have the following values, with “notice” being the default value:

• debug
• info
• notice
• warning
• err
• alert
• emerg
• crit

The Puppet Server can also be configured to process logs. This is done using the Java Virtual Machine Logback library. An .xml file is created, usually named logback.xml, which can be piped into the Puppet Server at run time. If a different filename is used, it will need to be specified in the global.conf file.

The .xml file allows you to override the default root logging level of ‘info’. Possible levels are trace, debug, info and debug. For example, if you wanted to produce full debug data for Puppet, you would add the following parameter to the .xml file.

<root level=”debug”>

The Most Useful Puppet Logs

Puppet produces a number of very useful log files, from basic platform logs to full application orchestration reports. The most commonly used Puppet logs include:

Platform Master Logs

These give generalized feedback on issues such as compilation errors, depreciation warnings, and crash/fatal termination. They can be found at the following locations:

• /var/log/puppetlabs/puppetserver/puppetserver.log
• /var/log/puppetlabs/puppetserver/puppetserver-daemon.log

Application Orchestration Logs

Application orchestration is probably the single most attractive aspect of the Puppet platform. It enables the complete end-to-end integration of the DevOps cycle into a production software application. As a result, these logs are likely to be the most critical logs of all. They include:

• /var/log/pe-mcollective/mcollective.log – This log file contains all of the log entries that affect the actual MCollective platform. This is a good first place to check if something has gone wrong with application orchestration.
• /var/lib/peadmin/.mcollective.d/client.log – a log file to be found on the client connecting to the MCollective server, the twin to the log file above, and the second place to begin troubleshooting.
• /var/log/pe-activemq/activemq.log – a log file that contains entries for ActiveMQ.
• /var/log/pe-mcollective/mcollective-audit.log – a top-level view of all MCollective requests. This could be a good place to look if you are unsure of exactly where the problem occurred so that you can highlight the specific audit event that triggered the problem.

Puppet Console Logs

Also valuable are Puppet logs, which include the following:

• /var/log/pe-console-services/console-services.log – the main console log that contains entries for top-level events and requests from all services that access the console.
• /var/log/pe-console-services/pe-console-services-daemon.log – low-level console event logging that occurs before the standard logback system is loaded. This is a useful log to check if the problem involves the higher level logback system itself.
• /var/log/pe-httpd/puppet-dashboard/access.log – a log of all HTTPS access requests made to the Puppet console.

Advanced Logging Using Further Tools

The inbuilt logging functions of Puppet mostly revolve around solving issues with the Puppet platform itself. However, Puppet offers some additional technologies to help visualize status date.

One of these is the Puppet Services Status Check. This is both a top-level dashboard and a queryable API that provides top-level real-time status information on the entire Puppet platform.

Puppet can also be configured to support Graphite. Once this has been done, a mass of useful metrics can be analyzed using either the demo Graphite dashboard provided, or using a custom dashboard. The ready-made Grafana dashboard makes a good starting point for measuring application performance that will affect end users, as it includes the following metrics by default:

• Active requests – a graphical measure of the current application load.
• Request durations – a graph of average latency/response times for application requests.
• Function calls – graphical representation of different functions called from the application catalogue. Potentially very useful for tweaking application performance.
• Function execution time – graphical data showing how fast specific application processes are executed.

Using Puppet with Sumo Logic

To get the very most out of Puppet log data, you can analyze the data using an external log aggregation and analytics platform, such as Sumo Logic. To work with Puppet logs on Sumo Logic, you simply use the Puppet module for installing the Sumo Logic collector. You’ll then be able to visualize and monitor all of your Puppet log data from the Sumo Logic interface, alongside logs from any other applications that you connect to Sumo Logic. You can find open source collectors for Docker, Chef, Jenkins, FluentD and many other servers at Sumo Logic Developers on Github.

About the Author

Ali Raza is a DevOps consultant who analyzes IT solutions, practices, trends and challenges for large enterprises and promising new startup firms.

Troubleshooting Apps and Infrastructure Using Puppet Logs is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out Sumo Logic Developers for free tools and code that will enable you to monitor and troubleshoot applications from code to production.