Querying data in Better Stack

There are three main ways of accessing logs in Better Stack: Live tail, Query API, and Dashboards.

Live tail

Live tail is your main window into logs you sent to Better Stack. We've optimized it to easily handle terabytes of logs, and we also made sure querying them is a breeze.

Query panel

The first thing you'll need to know about Live tail is the Query panel on top. This is how it looks, and we'll walk you through the individual parts you can find on it:

The first thing in the Query panel is the Source selector. You can mix & match which logs to show on Live tail, making it super easy to switch between apps, or to combine them to see patterns in multiple environments, to name a few use-cases.

Next comes the Query builder prompt, powered by the Live tail Query Language. This is where you filter your logs based on the content of the message, level, or any other attributes you send our way.

The Datetime picker next to the Query prompt gives you an easy way to specify the time range. You can use relative datetimes, like now-3h, that change automatically based on your time, or absolute times, like Monday at 3pm. The datetime picker (and the Scroll to field) also accept unix timestamps and ISO8601 strings, making it extremely straightforward to copy datetimes from your other sources, like error reporting.

You can use the bookmark icon to save views. Think of views as permanent filters unbounded by time - we store your query prompt and selected sources, and you can easily return to them. We found views very useful to keep track of common errors - we utilize them ourselves along with Alerts.

The last interesting action available in the Query panel is the Scroll to button. It accepts the same range of values as the datepicker (relative, absolute, or time described in words), and jumps your context to the given time, querying logs around it. It's very useful to see the context of what happend at a given time, and to drill down for some more context, you can apply additional filters.

Individual log lines

The next element you'll likely see a lot of is the individual log line. Here's an example, with expanded view of all the attributes:

There are two main parts - the primary fields rendered on top, and the attribute explorer you get when you click on a line.

You can control the row rendered by tweaking the primary fields in the source advanced settings. You may find that the default attributes we render are not the best match for you - feel free to tweak them as you like!

The rendered attributes give you an easy insight into the extra context you sent along with the log.

The entire log line is interactive, and there are multiple actions you can do. Here's a little gallery of the possibilities:

Query API

If you prefer to search for logs from your own systems, we have a Query API which you can use. It uses the same backend as the Live tail, so the options should feel familiar if you used it for querying logs.

Grafana (deprecated)

The third way of working with your logs is through the integrated Grafana.

You can either create dashboards with various panels, or use the Explore functionality to chart one-off queries.

We also automatically generate dashboards based on the source types and data structure you send our way - giving you a headstart! Note that the automated dashboards are re-created, so we advise against making them editable and changing them. Your changes won't be persisted in that way.

You can use the Clickhouse-flavor of SQL, but there's a little gotcha when you want to access individual columns. We store all your data in the json column, and you need to parse out the individual attributes out, if you want to select and query them. Luckily, it's very easy to do so, using native functions from ClickHouse!

Here's an example query:

SELECT JSONExtractString(json, 'level') AS level
FROM db_123.test_source
WHERE level IN ('ERROR', 'WARN')

As you can see, we use the JSONExtractString function and we alias the result as level, which makes it very easy to query with the attribute further. There are various JSON accessors like JSONExtractString, here's the full list of functions for working with JSON.

Materialized columns

When you find you're using a single column very often, either in Grafana or in Live tail, you can add it to the Materialized Columns section in the advanced settings of your source. The query will be simpler as well:

SELECT level
FROM db_123.test_source
WHERE level IN ('ERROR', 'WARN')

When you register a materialized column, we automatically optimize queries filtering by this column, leading to massive speed gains. Materialized columns pair very well with our Views, too!

Note that due to performance constraints, we have a limit on the number of materialized columns you can add to an individual source. Let us know at hello@betterstack.com if this is an issue for you!