# Better Stack Kubernetes logging ## Start logging in 2 minutes How do you want to integrate Better Stack with your Kubernetes cluster? - [Helm](#helm) - [Kustomize](#kustomize) ## Helm ### 1. Add Helm repository Add Better Stack Helm repository: ```sh [label Add Helm repo] helm repo add betterstack-logs https://betterstackhq.github.io/logs-helm-chart helm repo update ``` ### 2. Set up Helm chart Create `values.yaml` config file with the following contents: [code-tabs] ```yaml [label Default] vector: customConfig: sinks: better_stack_http_sink: uri: "https://$INGESTING_HOST/" auth: strategy: "bearer" token: "$SOURCE_TOKEN" better_stack_http_metrics_sink: uri: "https://$INGESTING_HOST/metrics" auth: strategy: "bearer" token: "$SOURCE_TOKEN" ``` ```yaml [label Disabled TLS] # config with disabled TLS (e.g. for Minikube) vector: customConfig: sinks: better_stack_http_sink: uri: "https://$INGESTING_HOST/" auth: strategy: "bearer" token: "$SOURCE_TOKEN" better_stack_http_metrics_sink: uri: "https://$INGESTING_HOST/metrics" auth: strategy: "bearer" token: "$SOURCE_TOKEN" sources: better_stack_kubernetes_metrics_nodes: tls: verify_certificate: false verify_hostname: false better_stack_kubernetes_metrics_pods: tls: verify_certificate: false verify_hostname: false metrics-server: args: - --kubelet-insecure-tls ``` ```yaml [label GKE Autopilot] # config for GKE Autopilot cluster vector: customConfig: sinks: better_stack_http_sink: uri: "https://$INGESTING_HOST/" auth: strategy: "bearer" token: "$SOURCE_TOKEN" better_stack_http_metrics_sink: uri: "https://$INGESTING_HOST/metrics" auth: strategy: "bearer" token: "$SOURCE_TOKEN" sources: better_stack_kubernetes_metrics_nodes: endpoint: "https://metrics-server.kube-system/apis/metrics.k8s.io/v1beta1/nodes" tls: verify_certificate: false verify_hostname: false better_stack_kubernetes_metrics_pods: endpoint: "https://metrics-server.kube-system/apis/metrics.k8s.io/v1beta1/pods" tls: verify_certificate: false verify_hostname: false # Disable persistence using a volume mount persistence: hostPath: enabled: false # Override default volumes and mounts to ensure only /var/log is mounted defaultVolumes: - name: var-log hostPath: path: /var/log type: DirectoryOrCreate defaultVolumeMounts: - name: var-log mountPath: /var/log readOnly: true # Metrics server is already installed metrics-server: enabled: false ``` [/code-tabs] ### 3. Deploy the chart Install the Better Stack chart: [code-tabs] ```sh [label Install] helm install betterstack-logs betterstack-logs/betterstack-logs -f values.yaml ``` ```sh [label Install without metrics server] helm install betterstack-logs betterstack-logs/betterstack-logs -f values.yaml \ --set metrics-server.enabled=false ``` [/code-tabs] [info] #### Already have metrics server installed? Install Better Stack chart **without metrics server** using code block above. Alternatively, set `metrics-server.enabled=false` in `values.yaml`. [/info] You should see your logs in [Better Stack → Live tail](https://telemetry.betterstack.com/team/0/tail ";_blank"). Check out your metrics in the [Kubernetes dashboard](https://telemetry.betterstack.com/team/0/dashboards/platform/kubernetes ";_blank"). [warning] **Kubernetes version 1.15 or higher is required.** You can customize the release name `betterstack-logs` to meet your naming conventions, but calling it `vector` or ending with `-vector` may lead to internal naming conflicts. [/warning] ## Need help? Please let us know at hello@betterstack.com. We're happy to help! 🙏 ## Kustomize ### 1. Install Kubernetes Metrics Server Install [Kubernetes Metrics Server](https://github.com/kubernetes-sigs/metrics-server ";_blank") into your cluster: ```bash [label Install Metrics Server] kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.6.4/components.yaml ``` ### 2. Create namespace Create namespace configuration file: ```bash [label Create namespace] kubectl create namespace --dry-run=client -o yaml vector > namespace.yaml ``` ### 3. Set up Kustomization Save the following config as `kustomization.yaml`: ```yaml [label Kustomization config] apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization # Override the namespace of all of the resources we manage. namespace: vector images: # Override the Vector image to pin the version used. - name: timberio/vector newName: timberio/vector newTag: 0.33.0-distroless-libc resources: # The namespace previously created to keep the resources in. - namespace.yaml # RBAC config allowing Vector access to metrics and creating a Secret with the service account. - rbac.yaml # Include Vector recommended base (from git). - github.com/vectordotdev/vector/distribution/kubernetes/vector-agent configMapGenerator: # ConfigMap with Vector config. - name: vector behavior: replace files: - vector-agent.yaml patches: # Prevent data dir read-only volume mount issue: https://github.com/vectordotdev/helm-charts/issues/226 # Set volumeMount readOnly: false # Also, add vector SA token into an ENV variable, Vector will use it to connect to metrics-server - patch: |- - op: replace path: /spec/template/spec/containers/0/volumeMounts/3/readOnly value: false - op: add path: /spec/template/spec/containers/0/env/- value: name: SERVICE_ACCOUNT_TOKEN valueFrom: secretKeyRef: name: vector-service-account key: token target: group: apps version: v1 kind: DaemonSet name: vector ``` ### 4. Set up RBAC configuration Save the following config as `rbac.yaml` to allow Vector to access the Metrics Server: ```yaml [label RBAC config] apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: vector-metrics labels: app.kubernetes.io/name: vector app.kubernetes.io/instance: vector app.kubernetes.io/component: Agent rules: - apiGroups: - "metrics.k8s.io" resources: - nodes - pods verbs: - list --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: vector-metrics labels: app.kubernetes.io/name: vector app.kubernetes.io/instance: vector app.kubernetes.io/component: Agent roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: vector-metrics subjects: - kind: ServiceAccount name: vector namespace: vector --- apiVersion: v1 kind: Secret metadata: name: vector-service-account annotations: kubernetes.io/service-account.name: "vector" type: kubernetes.io/service-account-token ``` ### 5. Set up Vector Download Vector configuration for your source: ```bash [label Download Vector config] curl -o vector-agent.yaml \ https://telemetry.betterstack.com/vector-yaml/kubernetes/$SOURCE_TOKEN ``` ### 6. Reload & restart Reload Vector configuration and restart it: ```shell [label Reload and restart] kubectl apply -k . kubectl rollout restart -n vector daemonset/vector ``` You should see your logs in [Better Stack → Live tail](https://telemetry.betterstack.com/team/0/tail ";_blank"). Check out your metrics in the [Kubernetes dashboard](https://telemetry.betterstack.com/team/0/dashboards/platform/kubernetes ";_blank"). [warning] **Kubernetes version 1.15 or higher is required.** [/warning] [info] **Using Minikube to test the setup?** Disable certificate validation to prevent TLS errors: - Add `--kubelet-insecure-tls` to `args` config of `metrics-server` DaemonSet - Add `tls.verify_certificate = false` and `tls.verify_hostname = false` to all `http_client` sources in `vector-agent.yaml` [/info] ### Need help? Please let us know at hello@betterstack.com. We're happy to help! 🙏 ## Additional information Interested in reading about **installing and configuring Vector** in more detail? Check out the [official Vector Kubernetes installation guide](https://vector.dev/docs/setup/installation/platforms/kubernetes/). Want an overview of different tools for monitoring Kubernetes? Head to comparison of [best Kubernetes monitoring tools](https://betterstack.com/community/comparisons/kubernetes-monitoring-tools/). ### Filtering logs sent to Better Stack Customize your Vector configuration by adding a [Vector filter transform](https://vector.dev/docs/reference/configuration/transforms/filter/) between the `better_stack_http_sink` sink and its original inputs: [code-tabs] ```yaml [label values.yaml (Helm)] vector: customConfig: sinks: better_stack_http_sink: uri: "https://$INGESTING_HOST/" [highlight] inputs: [better_stack_http_filter] [/highlight] auth: strategy: "bearer" token: "$SOURCE_TOKEN" better_stack_http_metrics_sink: uri: "https://$INGESTING_HOST/metrics" auth: strategy: "bearer" token: "$SOURCE_TOKEN" [highlight] transforms: better_stack_http_filter: type: "filter" inputs: [better_stack_kubernetes_parser] condition: .kubernetes.container_name == "my_important_container" [/highlight] ``` ```yaml [label vector-agent.yaml (Kustomize)] sources: # ... transforms: better_stack_kubernetes_parser_$SOURCE_TOKEN: # ... [highlight] better_stack_http_filter_$SOURCE_TOKEN: type: "filter" inputs: [better_stack_kubernetes_parser_$SOURCE_TOKEN] condition: .kubernetes.container_name == "my_important_container" [/highlight] sinks: better_stack_http_sink_$SOURCE_TOKEN: type: "http" method: "post" uri: "https://$INGESTING_HOST/" encoding: codec: "json" compression: "gzip" auth: strategy: "bearer" token: "$SOURCE_TOKEN" [highlight] inputs: ["better_stack_http_filter_$SOURCE_TOKEN"] [/highlight] better_stack_http_metrics_sink_$SOURCE_TOKEN: # ... ``` [/code-tabs]