In my last post I wrote about how DevOps’ emphasis on frequent release cycles leads to the need for more troubleshooting in production, and that developers are being frequently being drawn into that process. Troubleshooting applications in production isn’t always easy: For developers, the first course of action is to drop down to terminal, ssh into the environment (assuming you have access) and begin tailing log files to determine the current state. When the problem isn’t immediately obvious, they might tail -f the logs to a file, then grep for specific patterns. But there’s no easy way to search log tails in real time. Until now.
Now developers and team members have a new tool, called Sumo Logic Live Tail, that lets you tail log files into a window, filter for specific conditions and utilize other cool features to troubleshoot in real time. Specifically, Live Tail lets you:
- Pause the log stream, scroll up to previous messages, then jump to the latest log line and resume the stream.
- Create keywords that will then be used to highlight occurrences within the log stream.
- Filter log files on-the-fly in real time
- Tail multiple log files simultaneously by multi-tailing
- Launch Sumo Logic Search in context of Sumo Logic Live Tail (and vice versa)
Live Tail is immediately available from with the Sumo Logic environment, and coming soon is a command line interface (CLI) that will allow developers to launch live tail directly from the command line.
What Can I Do With Live Tail?
Troubleshoot Production Logs in Real Time
You can now troubleshoot without having to log into business critical applications. Users also can harness the power of Sumo Logic by being able to launch Search in the context of Live Tail and vice versa. There is simply no need to go between different tools to get the data you need.
Save Time Requesting and Exporting Log Files
As I mentioned, troubleshooting applications in production with tail-f isn’t always easy. First, you need to gain access to production log files. For someone managing sensitive data, admins may be reluctant to grant that access. Live Tail allows you to view your most recent logs in real time, analyze them in context, copy and share every time via secure email when there’s an outage, and set up searches based on live tail results using Sumo Logic.
Consolidate Tools to Reduce Costs
In the past, you may have toggled between two tools: one for tailing your logs and another for advanced analytics for pattern recognition to help with troubleshooting, proactive problem identification and user analysis. With Sumo Logic Live Tail, you can now troubleshoot from the Sumo Logic browser interface or from a Sumo Logic Command Line Interface without investing in a separate solution for live tail, thereby reducing the cost of owning licenses for multiple tools.
Getting Started
There are a couple of ways to initiate a Live Tail session. From the Sumo Logic web app:
- Go directly to Live Tail by hovering over the Search menu and clicking on the Live Tail menu item; or
- From an existing search, click the Live Tail link (just below the search interface).
In both instances, you’ll need to enter the name of the _sourceCategory, _sourceHost, _sourceName, _source, or _collector of the log you want to tail, along with any filters. Click Run to initiate the search query. That will bring up a session similar to Figure 1.
Figure 1. A Live Tail session.
To find specific information, such as errors and exceptions you can filter by keyword. Just add your keywords to the Live Tail query and click Run or press Enter. The search will be rerun with the new filter and those keywords will be highlighted on incoming messages, making easy to spot conditions. The screen clears, and new results automatically scroll.
Figure 2. Using Keyword Highlighting to quickly locate items in the log stream.
To highlight keywords that appear in your running Live Tail, click the A button. A dialog will open — enter the term you’d like to highlight. You may enter multi-term keywords separated by spaces. Hit enter to add additional keywords. The different keywords are then highlighted using different colors, so that they are easy to find on the screen. You can highlight up to eight keywords at a time.
Multi-tailing
A single log file doesn’t always give you a full view. Using the multi-tail feature, you can tail multiple logs simultaneously. For example, after a database reboot, you can check if it was successful by validating that the application is querying the database. But if there’s an error on one server, you’ll need to check the other servers to see if they may be affected.
You can start a second Live Tail session from the Live Tail page, or from the Search page, and the browser opens in split-screen mode, and streams 300 – 400 messages per minute. You can also open, or “pop out” a running Live Tail session into a new browser window. This way, you can move the new window to another screen, or watch it separately from the browser window where Sumo Logic is running.
Figure 3. Multi-tailing in split screen mode
Launch In Context
One of the highlights of Sumo Logic Live Tail is the ability to launch in context, which allows you to seamlessly alternate between Sumo Logic Search and Live Tail in browser mode. For example, when you are on the search page and need to start tailing a log file to view the most recent log files coming in (raw log lines), you click on a button to launch the Live Tail page from Search and the source name gets carried forward automatically. If you are looking to perform more advanced operations like parsing, using operators or increasing the time range for the previous day, simply click “Open in Search”. This action launches a new search tab which automatically includes the parameters you entered on the Live Tail page. There is no delay to re-enter the parameters.
For more information about using Live Tail, check out the documentation in Sumo Logic Help.
