Archive your logs in AWS S3 or a compatible storage service.

To preserve your logs even after the retention period is over, we highly recommend setting up S3-compatible storage, which will allow you to store your logs safely for later use.

How does it work you say? We have the backup scheduled every hour since the S3-compatible storage is connected, to ensure the steady flow of the logs and ease the follow-up work with those archived logs.

We then save the file in the format of #{database}/#{table}/#{from.strftime("%Y-%m")}/#{from.iso8601}-#{to.iso8601}.#{FILE_EXTENSION}".

The file extension we use is zstd. Now let's share how to get started with some most common storages, as well as the other ones.

Archiving with AWS S3

1. In the AWS Management Console
   1. Create a new S3 Bucket and write down its name and region.
   2. Create a new user in IAM with Programmatic access and configure the permissions to have read and write access to your new bucket. Write down the access key ID and the access secret key.
   3. Make sure your bucket is not publicly accessible.
2. In the Better Stack Integrations section
   1. Add new AWS S3-compatible archive integration.
   2. Give your integration a name.
   3. Fill in the bucket name and region from step 1.1
   4. Fill in your credentials from step 1.2
   5. If you entered all information correctly and if you set up the access privileges correctly, you'll see a confirmation message and you will find a betterstack_access_test text file in your Bucket.
3. In the advanced settings of your source, you can now select the integration under the Long-term log archive setting

Please make sure that you have the following required permissions for the AWS S3 Bucket:

 

Copied!

"Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket_name>",
                "arn:aws:s3:::<bucket_name>/*"
            ]
        }
    ]

Archiving with DigitalOcean Spaces

1. In the DigitalOcean console
   1. Create a new Space with** Restrict File Listing** and write down its name *and *region.
   2. In the API section generate a new key. Write down the *Key *and *Secret *of the new key.
2. In the Logs Integrations section
   1. Add new AWS S3-compatible archive integration.
   2. Give your integration a name.
   3. Select "DigitalOcean Spaces".
   4. Fill in the bucket *field with the *space name and also set the DigitalOcean region
   5. Fill in your credentials from step 1.2
      • Key *corresponds to *Access Key ID
      • Secret **corresponds to Secret Access Key**
   6. If you entered all information correctly and if you set up the access privileges correctly, you'll see a confirmation message and you will find a betterstack_access_test text file in your Space.
3. In the advanced settings of your source, you can now select the integration under the Long-term log archive setting

Archiving with Backblaze B2

1. Log in to your Backblaze B2 account
   1. Create a new Private *bucket and write down its *name *and *region
   2. In the App Keys section add a new application key with read and write access to the new bucket. Write down *keyID *and *applicationKey *(secret).
2. In the Logs Integrations section
   1. Add new AWS S3-compatible archive integration
   2. Give your integration a name
   3. Select "Backblaze B2"
   4. Fill in the bucket name and the *region *(include region number, e.g. us-west-002)
   5. Fill in your credentials from step 1.2
      • keyID *corresponds to *Access Key ID
      • applicationKey *corresponds to *Secret Access Key
   6. If you entered all information correctly and if you set up the access privileges correctly, you'll see a confirmation message and you will find a betterstack_access_test text file in your Bucket.
3. In the advanced settings of your source, you can now select the integration under the Long-term log archive setting

Archiving with other S3-compatible storage services

1. Create a new bucket in your S3-compatible service of choice
2. Create an access key and write down the access key ID and the** secret key**
3. In the Better Stack Integrations section
   1. Add new AWS S3-compatible archive integration
   2. Give your integration a name
   3. Select "Other"
   4. Fill in the credentials (access key ID and secret access key)
   5. In "Advanced settings" fill in the *endpoint *of your bucket in the path-style format (e.g., https://<region>.digitaloceanspaces.com/<bucket-name>).
4. In the advanced settings of your source, you can now select the integration under the Long-term log archive setting

Contact us at hello@betterstack.com if you're having trouble setting up archiving to your S3-compatible storage service.

---