How to Monitor MongoDB Logs

MongoDB is a leading NoSQL database that has proven itself through time as a stable and easy-to-use system of great flexibility. There are numerous tutorials and real-world success stories that show how MongoDB has helped with supporting the database needs of certain organizations and projects. It’s fair to say that using MongoDB proves to be a solid choice for integrating with web-based applications and microservices serving as a database system or even as a monitoring aggregator itself.

Let’s take a short tour as we explain in this article the importance of monitoring MongoDB clusters, as well as best practices and recommendations for monitoring data that we collect from MongoDB.

What is MongoDB?

MongoDB is technically classified as a NoSQL document oriented database. This means that we use documents to describe models in a schema. Additionally, its main selling points are: easy indexing, replication, ad-hoc queries, schemaless, cross-platform support and high performance.

You can think of a document as an object that has keys and values all together under the same classification. For example, here is a document model encoded as JSON for a typical user:

{
    "username": "Theo",
    "email": "email@gmail.com"
    "enrolled_at": "2020-10-02T15:43:21Z",
}

MongoDB innovates here by using this structure and adopting it into a new format called BSON, which is a binary representation of JSON data. Doing that optimizes the speed, schema flexibility and governance which, in turn, allows for better scalability. MongoDB performs data partitioning with chunking larger shards into smaller ones. This can be done automatically or manually, based on some criteria.

We have options for hosting MongoDB. First and foremost, the open source self-hosted option is the quickest way to run a server for experimentation. MongoDB also offers a Cloud-based option with MongoDB Atlas; and there are plenty of other providers that host MongoDB, with various degrees of service levels.

All of these operations need to be captured and analyzed for opportunities for improvement. This is where monitoring comes in handy.

Why Use Automated Monitoring Tools?

The following advice does not only cover MongoDB, but every critical infrastructure component.

You need monitoring tools just as much as you need database data. The simple reason is to understand the behavior and performance of the cluster as a whole, and to determine if there are missed opportunities for improvement. Usually monitoring tools work for you by inspecting the live stream of logs and alerting you if something goes wrong. This is a typical example of automation in practice. You don’t need to have an army of human operators checking the logs for suspicious events; it only takes one to manage dozens of clusters.

Thus, you save money and effort and give a better experience for your end-users. A top-notch monitoring setup enables you to be on top of any potential risks and hazards that can affect the availability and performance of the cluster.

Monitoring tools come in all shapes and sizes. They can be self-served, offered as part of a service as a turn-key solution, or custom-made on premises. Either way, the principles are the same: to ensure accurate systematic gathering of data that can be further analyzed and organized into valuable information.

What Do I Monitor in MongoDB?

MongoDB offers several strategies to monitor the cluster. You can configure the log granularity levels from simple entry to very detailed, suitable for development and debug sessions.

However not all metrics are suitable in a production environment and finding the most useful ones requires a case-by-case analysis.

At a basic level you want to pay attention to the following metrics:

• Health checks: For uptime and availability counters
• Current load and limits: For checking against upper limits. For example, reaching a connection limit or a storage limit
• Collection level performance: For checking how much time the server spent on queries (read or write) for particular collections
• Server/Shard performance: For inspecting the actual performance of the server daemon and cluster components.

For advanced cases you may have to utilize the command line to extract further information. You can reconfigure the profiling level to perform performance investigations while making sure the overhead is manageable. You can also use the currentOp command to inspect the current operation based on certain criteria; for example, when it’s running for more than three seconds. This is useful for on-the-spot investigations and queries. If you want to automate the process, you can use the respective MongoDB drivers to issue client commands, capture the results and send them programmatically to an external service provider.

How Do I Monitor My MongoDB?

Monitoring a MongoDB cluster is actually a fairly streamlined process. In fact, you have several options for how to do it.

• You have the free monitoring option that you can enable, which offers an option to have a unique URL to inspect the cluster metrics.

• If you are using MongoDB Atlas, you have a free monitoring option in the dashboard where you can inspect the status log and perform granular queries for network or connection counters.

• At the basic level, you have the option to configure the log verbosity levels, and inspect information that is logged there. Then you can specify a systemlog destination where the logs will be written to.

• At the most advanced level you have the option to query the server and capture the logs in the command line. You can either use the serverStatus command for basic health check reports or the mongotop tool that provides statistics per connection level.

In any case you can extract plenty of information in regard to query performance, connection information or errors happening inside the cluster. That information needs to be readily available for further process.

Monitor MongoDB with Sumo Logic

Instead of setting up a complete monitoring stack from scratch by yourself, you can leverage Sumo Logic pre-built apps to handle this process for you. In return, you get advanced operational intelligence controls and customized dashboards that integrate well with most services.

Sumo Logic supports adding collectors for both MongoDB and MongoDB Atlas. The process is divided into three steps:

1. First, you need to install a collector into the system that runs the database using this guide.
2. Next, you need to configure a local file source inside the system that the collector runs. This is the location of the log files with the messages that the collector will use to scan and publish to Sumo Logic service for consumption.
3. Lastly, you need to use the app dashboard to install the MongoDB app that will enable the dashboard. Then, once the events are triggered, you will be able to see the live activity of them.

After that process you have the flexibility to create custom queries and window configurations that suit your environment. You can even connect events with alerts that will help you find problematic queries and performance bottlenecks that occur on your MongoDB cluster.

To help get you started, you can leverage the generous free trial period and test how Sumo Logic MongoDB monitoring works for your organization.