Using built-in embeddings

Better Stack Warehouse allows you to generate text embeddings without calling any third party API.

Simply go to Sources -> Your sources -> Embeddings and choose what text field to generate embeddings from and what JSON attribute we should store it in.

Storing embeddings as time series

Go to Sources -> Your source -> Time series on NVMe SSDs and click + Time series. Use JSON dot notation to write the name of the target column from the previous step and chose BFloat16 and choose your Vector index.

Indexing vector columns

If you're working with hundreds of millions of events your queries might benefit from creating vector indices for your embeddings.

Querying embeddings in Warehouse

Better Stack Warehouse stores embeddings as vector columns that can be indexed and queried efficiently using ClickHouse’s vector type and similarity functions such as distance() or cosineSimilarity().

SELECT
  description,
  cosineDistance(embedding, embedding({{description}})) AS similarity
FROM {{source}}
ORDER BY similarity ASC
LIMIT 5;

This finds the events most semantically similar to a given description.