Let’s check the logs together!

Written by Amer Hasanbegović, Software Developer at Softray Solutions

If you are a developer, there is no way you haven’t worked with logs or at least heard of them. In this article, we are going to take a look at what are logs and log management and why is it important. We will discuss ways of storing logs and get familiar with the cloud-based solutions that are becoming increasingly common in the IT world.

Logging and log management

Photo by Angie Muldowney on Unsplash

Before we go deeper into the subject, it has nothing to do with the image you just saw, or does it? First, we need to understand what a log is. Simply said, log or log file is a computer-generated file that records activity within the software application. It can capture informational messages, warnings, errors, file requests, transfers, etc. The good thing is that each logged activity is timestamped, so besides knowing what happened, we also know when it happened. Log management is continuously gathering, storing, processing, and analyzing log data to optimize system performance, identify issues, ease troubleshooting and increase security.

Why is logging important?

Logging is essential in software development because it helps troubleshoot bugs in different environments, improves monitoring in the production environment, and eases code debugging while in development.

1. Troubleshooting bugs

Most projects and applications have multiple environments: development, test, UAT, and production. Imagine that suddenly the application broke down in the production environment, and you need to take care of the problem as soon as possible. If you have logs, you can quickly check them and at least get an idea of what caused the problem and try to replicate the process and hopefully solve the issue.

2. Production monitoring

By adding logs to your application, you can monitor the execution time of background processes, measure response time and speed, and collect traffic data to get information on when the application was used by users. In case there is heavy traffic, for example, at a specific time, metrics based on log data can help develop new solutions or an infrastructure to respond to that traffic and ultimately improve stability and performance. Also, log data can help find and understand vulnerabilities in the code and possible security issues and help us improve in the right place.

3. Debugging

Sometimes, in the development process, logs can be helpful when an unexpected problem occurs, and you don’t have time or don’t want to debug each line of the code. Instead, check the logs and see what happened.

Log levels

Log levels are a way to classify the log messages in terms of severity. The most common log levels are: fatal, error, warn, info, and debug.

            try
            {
                // Do something
            }
            catch (Exception)
            {
                log.Fatal("Very severe error events that will presumably lead the application to abort.");
                log.Error("Error events that might still allow the application to continue running.");
                log.Warn("Designates potentially harmful situations.");
                log.Debug("Designates fine-grained informational events that are most useful to debug an application.");
                log.Info("Informational messages that highlight the progress of the application at coarse-grained level.");
            }

There are also levels like all (logs everything), off (doesn’t log anything), and trace (even more detailed than debug log level). Using log levels, we can control the amount of information in log files and easily filter out those we are looking for.

How to log?

First, it is vital to log at the proper level. The log levels provide information on how crucial the event is. The logged messages must be meaningful and human-readable since humans will read and use those messages and contain the necessary information to help understand the problem.

Also, logs must be created with proper context information that connects multiple application layers and points to the real problem.

Bad example:

Login unsuccessful.

Good example:

Login unsuccessful.
Database error: Error establishing a database connestion.

In this example, we logged a message that said login was unsuccessful, but that message doesn’t tell us the real problem, why was it unsuccessful? When we add additional information to that message that connects different layers of application, expanding the context and further describing the problem, then we know why we got that message in the first place.

Ways to store logs

There are multiple ways to store logs, each of which has pros and cons. Plain text files and databases are the most common ways to store log data. However, the new age brings one more option to the table, and that is third-party cloud-based log management systems.

By default, most servers store logs in plain text files on their local file system, which is the fastest way. Text files are easily accessed and readable but challenging to use, query, analyze, and generate metrics. Another problem with plain text files is that they can contain a massive amount of data, measured in hundreds of megabytes or even a few gigabytes depending on the frequency of logging. Just imagine having to search for something in a gigabyte text file.

The second option is to log into the database. The reason to do that is that you get an option to query the results. Searching for specific logs is way easier in the database, especially if you did a good job with log messages in terms of quality and contextual information. Accessing databases and checking logs instead of text files on the server that can be corrupted or even locked is also a great advantage.

Depending on the size of the application, traffic, and amount of log data, you may choose one of these options. If your application is small, for example, an informational website, and it doesn’t generate much log data, logging into text files is ideal. On the other hand, bigger applications and enterprise systems should implement databases or even cloud logging.

After all, it looks like it has something to do with that image from the beginning. You need a place to store them.

Modern log management solutions

In general, cloud computing and cloud solutions are becoming increasingly available and used in the IT world. Cloud providers like New Relic, LogDNA, Papertrail, Google, AWS, and many more offer various log management tools and services that take care of storing, securing, and analyzing log data.

Using the cloud provides enhanced flexibility and scalability, meaning that organizations can quickly expand or shrink storage and processing based on needs. In addition, log management tools enable powerful real-time querying, sorting, and analyzing of the log data, along with visualizing that data in a single place. All of this allows for finding and fixing problems faster.

Due to different business and compliance requirements, well-configured and maintained centralization of log data becomes very important in many cases. At first, it is easier and cheaper to keep data “in-house,” but eventually, the amount of data will grow, and storing it becomes a nightmare. Maintaining and keeping everything in order and secured can be very costly and time-consuming if we are talking about enterprise projects that generate huge amounts of log data. Using cloud tools and services from vendors can provide more security and reduce costs in the long run.

Conclusion

Logging and log management must be integral to every serious software development project. It helps developers and administrators prevent, troubleshoot, and quickly address issues. In addition, it provides an incredible source of data that can be analyzed and used to create new and better solutions for your product.

Logging is a great concept that is not hard to understand and easy to implement, and it comes with great benefits for everyone using or developing the application. The ultimate goal is to keep your customers happy and make the job easier for people working on your product.

I hope you learned something new and exciting and enjoyed reading this article. And always remember to check the logs first!

Resources

If you want to know more about logging and log management (which you definitely should!), check the links below:

https://www.section.io/engineering-education/how-to-choose-levels-of-logging/

https://newrelic.com/topics/what-is-log-management

https://www.youtube.com/watch?v=yp0EM4fz6F8

https://www.addictivetips.com/net-admin/cloud-logging-services/

https://scoutapm.com/blog/logging-monitoring

If you enjoyed reading this, click the clap button so others can find this post.