Why we required structured logging?

Why we required structured logging?

Logging is one of the important cross-cutting concerns in software development and operations. It helps to debug or investigate an issue, create business intelligence on user activities and track the state and health of the system. So, it is very important to ensure we have enough and context full logs in place to make developers and operational engineers life easier.

Broadly, application logs are categorized into two and it is purely based on how a log item formatted and stored. The first one is the unstructured log and the second one is the structured log.

What is Unstructured logs?

An unstructured log item is an arbitrary text, it stored and retrieved as a whole string to/from a log store. It might provide the meaning full to the human reader but it is not understandable by a machine. Correlation of related log entries is a complex process or almost not possible. Most of the time we use unstructured logs, especially when we writing application logs and these logs are useless or complex to process.

Example:

_logger.LogInformation("Product " + productCode + " added to inventory and the id is  " + productId);

It is important to understand what is a structured log and how going to rescue us from the above issues. In this post, we are going to discuss structure logs.

What is a Structured Log?

Unstructured log entries are present in a textual format that is easily readable for a human but difficult to correlate with related log entries. Sometimes, we may want to use automated processing to investigate log files, or we may want to use algorithms to categorize, index and search through log files based on specific parameters (by date, user, number, etc.). Unstructured logs are not suitable to do above processes since it is missing the contextual or associated state which helps to filter or group the log entries easily.

Structured Logging is the practice or method that uses the predefined message template for application logs which allows being treated as data sets rather than plain text. The idea of structured logging is to take an application log that is delivered as a string of text and convert it into a simple relational data set that can be more easily searched and analyzed.

Example:

_logger.LogInformation("Product {ProductCode} added to the inventor with id {ProductId}", productCode, productId);

Why we required a Structured Log?

One common problem with an unstructured log is correlating the related log entries or searching required log entries are not an easy task. In order to do the effective search or grouping the log entries, we need some additional data points to index or associate the logs. Structured logs are using the defined message template to add the required details to logs and it makes the logs can easily be searched and analyzed.

Structured logs address the following issues,

  • Unstructured logs are formatted arbitrarily, so it requires some custom parser to parse and search the data. We might end up to create a parser for each message format.
  • Correlating or grouping related logs from unstructured logs are a very complex task and it will increase the meantime to detect (MTTD) and mean time to resolve (MTTR) any runtime issue occurs during business operations.
  •  Whenever the message format changed, the underlying parser and depending application might have impacts

Structured logging is a practice rather than a process which will reduce the MTTD and MTTR time factors during the development and operations phases, so all developer should create a habit of using this practice in their everyday coding life where ever it is appropriate.

What is the need for a centralized log management system?

Another common issue in logging is storing the logs into multiple log stores (file, database, in memory, etc) across different layers or tiers of an application. Which completely isolates the log entries and almost impossible to correlate with each other.  This is due to lack of log management, in order to make developer life easier they need all log entries in one place. The centralized log management system brings the all log entries into one place and facilitates to retrieve the log entries easily and expressively.

The bottom line is, the centralized log management system will bring all application-related logs in one place and facilitates to retrieve easily. The structured logs will create required data points to create the relationship map across multiple log entries. So, we need both in place to do effective logging and analysis.

Log Store

Another important aspect of logging is log store, it stores and retrieves the log entries. Example log stores are files, databases, etc. However, these log stores should have the following capabilities to utilize structured log features.

  • Easy retrieval
  • Searching logs with filter
  • Correlating logs through query

Here is a list of some popular log stores,

Logging Framework

Logging frameworks are enablers to use the structured log. The supporting framework should have the following functionality to facilitate the structured log

  • Use the message template to format and capture data points
  • Capture the data points and rendered message separately
  • Include the message template within the log entry

Here is the list of some popular logging frameworks which supports for structured log in various programming languages,

In the upcoming article, I am planning to provide guidelines on how to effectively write the structured log entries in your application. So, please stay tuned and we will meet soon in another post.