Explore documentation
  1. Welcome to Warehouse 👋
  2. Ingesting data
    1. Ingesting events
    2. S3 vs NVMe hosted data
    3. Host events in your S3 bucket
  3. Querying data
    1. Querying events & time series
    2. Writing ClickHouse SQL queries
    3. Query speeds
    4. Ad-hoc SQL API
    5. Generate client-facing APIs
    6. Visualizing data
  4. Vector embeddings
    1. Introduction to embeddings
    2. Using built-in embeddings
    3. Bringing your own embeddings
    4. Data structures and indices
  5. Use cases
    1. Retrieval Augmented Generation
  6. Management
    1. Controlling costs

Bringing your own embeddings

You can use any third-party API to generate embeddings for your events prior to sending them to Better Stack data warehouse.

Better Stack's built-in model embeddinggemma:300m should cover most use cases but if you want to use an external provider, we recommend OpenAI's embedding models for the best price/performance ratio.

Send us the embedding alongside the original text in a dedicated attribute as an array of floats.

Storing embeddings as a time series

Go to Warehouse -> Sources -> Your source -> Time series on NVMe SSDs and click + Time series. Use JSON dot notation to write the name of the target column from the previous step and choose BFloat16 and choose your Vector index.

Creating a time series for your embeddings

Querying embeddings in Warehouse

Better Stack Warehouse stores embeddings as vector columns that can be indexed and queried efficiently using ClickHouse’s vector type and similarity functions such as distance() or cosineSimilarity().

To find the closest match, generate embedding using the same model and use a Text query variable to send us the embedding:

Querying in time series
SELECT
  text_id, -- fetch raw texts from JSON events to minimize data on NVM
  cosineDistance(
    meta_embeddings_text,
    JSONExtract({{embedding}}, 'Array(Float32)')
  ) AS distance
FROM {{source}}
ORDER BY distance ASC
LIMIT 5

Previous article

Using built-in embeddings

Next article

Vector embedding data structures and indices