Introduction

JSON/JSONL logs are, in my opinion, a good way to log from your application to a file. The structured nature makes them easily programatically parsed. This works great with tools like graphana, loki or Google Cloud Logs, however I'v noticed that parsing them locally isn't as straightforward as you might expect.

In this post, I'm going to go over the ways I view and process logs, locally, on my work machine.
I will be using CLI and TUI tools only.

Update 21/03/2024

A few weeks ago I have found an even better way of viewing jsonl logs using VisiData.

See my post about VisiData for more information.

What is VisiData

It's a TUI (terminal UI) application that looks a bit like your traditional spreadsheet application - Think LibreOffice Calc or Excel except that it's much faster and can open much bigger files in a flash. It also has native support for JSON and JSONL file formats.

The tools

There are many tools to visualise and navigate json or jsonl data:

• jless
• fx

And there are tools to filter json:

• jq
• ijq
• fx (again)

But to interactively do both, this is where it gets more tricky

Combining the tools

Here are some of the techniques I've come up with to combine the tools to get the best of both worlds.

jq + jless

jq is by far my favourite json filterer/processor. It's fast and well documented and supported.
jless is up until now, my favourite json viewer. I like it's documentation better than fx, and it's written in rust instead of javascript.

Here is a sample command to parse some jsonl logs:

cat application_name.log.jsonl | jq 'select(.level != "DEBUG")' | jless

I used cat to keep the file I'm passing in, nice and separate from the filter command, but it's also possible to pass the file as a 2nd parameter to jq and only have to pipe data once.

ijq + jless

jq can be a bit hard to learn at first, so an interactive approach is welcome. ijq allows us to interactively try filters and see the output change in real time. Once we are happy with the output, we can then send it to jless for further viewing

cat application_name.log.jsonl | ijq | jless

or the upgraded version which will also copy the filter to your clipboard (on macos only). If you want this to work on linux, you need to replace pbcopy with xclip -selection clipboard
This allows you to use the filter in jq later on if needed.

cat application_name.log.jsonl | ijq 2> >(pbcopy) | jless

The copy thing works because ijq outputs the filter to stderr and the json result to stdout.
We are are using process substitution to pass stderr to pbcopy (or xclip) and stdout to jless.