# Google Cloud Pub/Sub logging

## Start logging in 5 minutes

Forward logs from Google Cloud Platform to Better Stack.

You will need a **Pub/Sub Subscription** which you can find or create in [Google Cloud Console → Pub/Sub → Subscriptions](https://console.cloud.google.com/cloudpubsub/subscription).

Note the **Subscription name** for future use, in the format `projects/<YOUR_PROJECT>/subscriptions/<YOUR_SUBSCRIPTION>`.

## Deploy the Dataflow job using Web UI

1. Go to  [Google Cloud Console → Dataflow → Create job from template](https://console.cloud.google.com/dataflow/createjob).
2. Fill in **Job name** and **Region**.
3. Under **Dataflow template**, select **Custom template**.
4. As **Template path**, use `betterstack/pubsub-to-betterstack.json`
5. Fill in the **Required Parameters**:

```plain
[label Required parameters for Pub/Sub to Better Stack template]
Input Pub/Sub Subscription:  projects/$PROJECT/subscriptions/$NAME
Better Stack Source Token:   $SOURCE_TOKEN
Better Stack Ingesting Host: $INGESTING_HOST
```

Then, click **Run job** 🚀

You should see your logs in [Better Stack → Live tail](https://telemetry.betterstack.com/team/0/tail ";_blank").

[note]
Please note it may take a few minutes for the job to initialize.
[/note]

## Deploy the Dataflow job using CLI (alternative)

If you prefer using `gcloud` CLI, you can run the job using the following command in your active project and region:
   
```bash
[label Run the Dataflow job using template]
PROJECT="$(gcloud config get-value project)"
REGION="$(gcloud config get-value compute/region)"
SUBSCRIPTION="projects/$PROJECT/subscriptions/<YOUR_SUBSCRIPTION_NAME>"
gcloud dataflow flex-template \
  run "pubsub-to-betterstack-$(date +%Y%m%d-%H%M%S)" \
  --template-file-gcs-location="gs://betterstack/pubsub-to-betterstack.json" \
  --parameters input_subscription="$SUBSCRIPTION" \
  --parameters better_stack_source_token="$SOURCE_TOKEN" \
  --parameters better_stack_ingesting_host="$INGESTING_HOST" \
  --region="$REGION"
```

You should see your logs in [Better Stack → Live tail](https://telemetry.betterstack.com/team/0/tail ";_blank").

[note]
Please note it may take a few minutes for the job to initialize.
[/note]

## Need help?

Please let us know at **hello@betterstack.com**.  
We're happy to help! 🙏


## Additional information

### Managing cost

When not specified, the streaming job uses the default machine type for Google Compute Engine instances used in your pipeline execution. For example, `n1-standard-1`.

You can customize the used machine type in **Optional Parameters** section while creating the job. Uncheck the **Use default machine type** and pick a different type. Alternatively, you can use `--worker-machine-type` CLI parameter.

You can also customize the autoscaling options or the used zone for running your Dataflow job.

You can read more on [Pricing in official Google Cloud docs](https://cloud.google.com/dataflow/pricing).

### Specifying a VPC network

By default, the job is created in the `default` network.

This may lead to errors while initializing the job, such as:

```plain
[label Example error due to network misconfiguration]
Failed to start the VM used for launching because of status code: INVALID_ARGUMENT,
reason: Invalid Error: Message: Invalid value for field
‘resource.networkInterfaces[0].network’: ‘global/networks/default’.
The referenced network resource cannot be found.
```

You can customize the used network or subnetwork in **Optional Parameters** section while creating the job. Alternatively, if you're using the CLI you can use parameters `--network` or `--subnetwork`.

[note]
The Better Stack ingesting host must be routable from the VPC that the job runs in.
[/note]

You can read more about how to [Specify a network and subnetwork in Dataflow official docs](https://cloud.google.com/dataflow/docs/guides/specifying-networks).

### Need to fine-tune the log forwarding?

Supply the following option in the **Optional Parameters** section in Web UI, or via `--parameters` in the CLI command.

- `batch_size` - Number of messages to batch before sending. Default: `100`
- `window_size` - Window size in seconds for batching messages. Default: `10`
- `max_retries` - Maximum number of retry attempts for failed requests. Uses exponential backoff between retries. Default: `3`
- `initial_retry_delay` - Initial delay in seconds between retries. The delay doubles with each retry attempt. Default: `1`

You can also fork the [open-source repository on Github](https://github.com/BetterStackHQ/gcp-dataflow-pubsub-to-betterstack) and fully customize the template.