Log File is a file that contains all events that happened in the system. By simply reading the log file, Developers can understand what happened, who did something to the system, and many more. Some systems have their standard way to write logs like Apache, Nginx, Envoy, Etc. But how about the custom one? Of course, developers need to write the log file as descriptive as possible to read it easily.
Perhaps writing logs is not an issue, but reading it? Do developers review and read their logs? Do they can easily understand the logs? Do they know what happened to the system right now? Perhaps not. That’s where monitoring apps like Elastic or Grafana come to help parse and monitor the logs file.
Monitoring apps could help developers read the system logs, creating an alert if something went wrong. But they don’t understand what happened to the system. They only follow the rules created by the developers. What if the developers want to put a little
brain to the monitoring system so the monitoring apps could understand what happened? Sure by creating a
deep learning model to analyze the logs is more than a help. But before that, developers should make sure they can parse the logs.
In this blog post, you will create a simple logs parser using Go as a first step to understand the logs file better.
Define the Log Format
Let’s say there’s a single line of log formatted like this:
You could extract the data you want from that line, for example:
- HTTP Method
- Request Path
- Response Code
Then create the log format according to that line. Let’s say you want to name the
$time_stamp, and the unimportant data as
$_. Now you will have a formatted string like this:
So you can read your logs data like this:
Create the Parser
Let’s create a
main.go file with the logs data and the format. To be easily used by the regex, you should escape the special symbol in your format using
After define the format, adjust your
logFormat to a format that regex could read. Because your variable starts with
$ and only contains
underscore. You can match the variable using this regex
\$([\w_]*) then change all of the variables into a
named capturing group in regex. Which is
(?P<name>re). Because you want to
<name> to your defined variable name, you can modify the
named capturing group to
(?P<$1>.*). So if you put that in the code, it should be like this:
regexFormat looks like this:
Then compile your
regexFormat to find all data in the logs line.
matches should have all your matched data. Let’s print it.
The output should be like this:
After parsing a single logs line, you should be able to parse all your logs files. The only thing you need to do is define your logs file format. And then transform it into a human-readable format like the previous step.
Here is the complete code:
Thank you for reading!