# Live tail

The Live tail query language is what powers searching in Live tail. The query language is designed to be straightforward to use and powerful at the same time.

There are two main modes to use the query language in:

- [Simple](#simple-queries)
- [Compound](#compound-queries)

## Simple queries
Simple queries are what you're used to from search engines - queries like these are all supported.

- "hello world"
- Started
- 404
- "`https://example.com/abcd`"

When you use a simple query, we search all columns that match the query type with a fulltext query, case insensitive. This means that if you search for `Hello`, we can find logs like

```json
[label Matched log example]
{
  message_string: "Hello world!",
}
```

and

```json
[label Matched log example]
{
  article_published_at: "2021-05-26T13:32:01Z",
  article_title: "The band Hello announced a new tour!"
}
```

## Compound queries
Compound queries follow a more rigid structure, and allow for chaining query clauses in a single search. Compound queries are composed of clauses optionally chained with a conjunction operator. You can also use parentheses to give preference to the queries. 

```
[label Compound query - Match 4xx status codes]
vercel.proxy.status_code_integer>=400 AND vercel.proxy.status_code_integer<500
```

In the example query, there are two query clauses joined by an AND cojunction operator.

A compound query clause consists of a column, an operator, and a value. In the case of the previous query, the first clause has

- `vercel.proxy.status_code_integer` as the column,
- `>=` as the operator, and
- `400` as the value

### Query clause format
You can use any columns you sent us in a compound query clause.

This is the operator list currently supported:

* Operators applicable to all column types
    - `=` equals
    - `!=` not equals

* **String** operators
    -  `:` contains
    -  `!:` not contains
    -  `=~` matches regex
    -  `!~` doesn't match regex

* **Integer** and **Float** operators
    - `>=` greater than or equal
    - `<=` less than or equal
    - `>` greater than
    - `<` less than

The value of a query clause can be either a simple text, or a more complex text surrounded by quotes. Here are some examples:

- `hello`
- `"Hello World"`
- `500`

Note that query clause values don't support asterisks (`*`) or question marks (`?`) for regex matching. These queries are therefore not valid:

- `vercel.proxy.status_code_integer>=4??`
- `vercel.proxy.status_code_integer>=4*`

In most cases, you can achieve the expected result with query chaining, like

- `vercel.proxy.status_code_integer>=400 AND vercel.proxy.status_code_integer<500`

We support two conjunction operators: `AND` and `OR`. You can also query for compound queries without using either of those, in which case we assume the `AND` operator. This means that this query

```
[label Compound query - Match 4xx status codes]
vercel.proxy.status_code_integer>=400 AND vercel.proxy.status_code_integer<500
```

behaves the same as this one

```
[label Compound query - Implicit AND for 4xx status codes]
vercel.proxy.status_code_integer>=400 vercel.proxy.status_code_integer<500
```

Note that it's currently not possible to chain Simple and Compound queries - if you tried something like `vercel.proxy.status_code_integer>=400 AND "Hello World"`, we fallback to parsing this as a simple query, matching the entire query contents against your columns.

### Escaping dots in column names

When your log data contains keys with literal dots in their names, you need to escape them with backslashes and wrap the entire key in quotes to distinguish them from nested object notation.

```
[label Escaping dots in column names]
"resource.attributes.service\.name"=mysql
```

Without proper escaping, a query like `nested.key.with.dot=value` would be interpreted as accessing the `dot` property of the `with` object inside the `key` object of the `nested` object, rather than a single key literally named `nested.key.with.dot`.


### Fulltext column
Because chaining simple and compound queries is not supported, we have a special column to specify the fulltext search behavior present in simple queries: fulltext. The usage is the same as with other columns - the difference is that when we encounter this column, we search all columns in the same way as the simple queries do.

These queries are therefore equivalent:

- `Hello world`
- `fulltext:"Hello world"`

You can use `=`, `!=`, `:`, `!:`, `=~`, and `!~` operators with the `fulltext` column.

You can use the special `fullext` column to make wider searches while maintaining filters imposed by other clauses, for example

```
[label Example of using fulltext in compound query]
vercel.proxy.status_code_intetger>=400 AND \
  vercel.proxy.status_code_intetger>=400 AND \
  fulltext:"Record Not Found"
```

## Regular expressions

Query using regular expressions in Live tail to match more complex patterns within log entries.

There are three primary ways to use regex in your queries:

- **Fulltext search**: Applies the regex to the entire log message, searching across all columns (e.g., `/Started (GET|POST)/`).
- **Column search**: Applies the regex to specific columns using the `=~` and `!~` operators (e.g., `message=~/Failed (GET|POST)/`).
- **Case-insensitive matching**: Allows regex to ignore case sensitivity by appending an `i` after the expression, which requires slashes (e.g., `message=~/case insensitive/i`).

[info]  
**Want to match a range of numbers?**  
Regex is only supported for string-based columns. For numeric columns like `status_code`, use comparison operators such as `status_code>=400 AND status_code<500`.  
[/info]

### Fulltext search with regex

In a fulltext search, the regex is applied to the entire log message, searching across all columns. 

Filter logs where the log message contains "Started GET" or "Started POST":

```
[label Fulltext search: Match GET or POST requests]
/Started (GET|POST)/
```

Find logs containing the word "error" followed by a 4xx or 5xx status code:

```
[label Fulltext search: Match error followed by 4xx or 5xx status codes]
/error (4\d{2}|5\d{2})/
```

Match logs that include "OK in [some number] ms," where the number can have one or more digits and a decimal point:

```
[label Fulltext search: Match OK response time]
/OK in \d+\.\d+ms/
```

Match logs containing user login messages in the format "User [ID] logged in," where `[ID]` is a combination of alphanumeric characters and hyphens:

```
[label Fulltext search: Match user login by ID]
/"User [\w\-]+ logged in"/
```

### Column search with regex

When using regex with specific columns, use the `=~` and `!~` operators, and wrap multiple-word patterns in quotes.

Filter logs where the `message` column contains "Failed GET" or "Failed POST":

```
[label Column search: Match failed GET or POST requests]
message=~/Failed (GET|POST)/
```

Find logs where the `user_agent` column includes either "Chrome" or "Firefox":

```
[label Column search: Match Chrome or Firefox user agents]
user_agent=~/Chrome|Firefox/
```

Match logs where the `url` column follows the pattern `/api/v[version]/items`, useful for tracking API version usage:

```
[label Column search: Match API version in URL path]
url=~/\/api\/v\d+\/items/
```

### Case-insensitive matching with regex

To perform case-insensitive matching, append `i` at the end of the regex expression, and use slashes around the pattern.

Perform a case-insensitive search for "Kubernetes" or "Docker" within the `platform` column:

```
[label Case-insensitive match: Match Kubernetes or Docker]
platform=~/Kubernetes|Docker/i
```

Find logs where the `user_agent` column includes either "Chrome" or "Firefox":

```
[label Case-insensitive match: Match Chrome or Firefox user agents]
user_agent=~/Chrome|Firefox/i
```

## Pattern-based filtering

Every log entry in Live tail includes a special `_pattern` field that represents a normalized version of the message, with variable data like IDs, numbers, and timestamps removed. This helps group similar log entries and makes it easy to filter repetitive or noisy logs.

You can use the `_pattern` field in compound queries to include or exclude logs matching specific patterns.

To make this easier, just hover over a log in Live tail and click **Filter by similar pattern** or **Exclude this pattern**.

![Pattern-based filtering in Live tail](https://imagedelivery.net/xZXo0QFi-1_4Zimer-T0XQ/058eb0cf-d9be-4497-40db-6e4e7c0dc500/public =2034x848)

That will add a new filter into your current query, for example excluding entries that match a specific pattern:

```text
[label Exclude known health check pattern]
_pattern!=6476bd6e601f1d12aa91e82371c88066
```

Learn more about [how log patterns work](https://betterstack.com/docs/logs/using-logtail/using-pattern-field/).