MinIO Storage Full and GCS Logging UI Issue – Seeking Solutions

[karolur]

Hi Airbyte Community,

I’m currently managing an Airbyte deployment on Kubernetes using the Helm chart, and I’ve been running into challenges with log storage and visibility:

MinIO Storage Issue:

• I previously configured Airbyte to use MinIO for log storage. However, I consistently encounter a Storage backend has reached its minimum free drive threshold. Please delete a few objects to proceed. error in MinIO nearly every week.

• To resolve this, I’ve been manually cleaning up the logs and, unfortunately, reinstalling the Helm chart to clear space. I’d like to avoid the full reinstall and manage cleanup more effectively.

• Is there a recommended way to configure MinIO to periodically clean logs without requiring a full reinstallation?

Switching to GCS for Logs – No UI Visibility:

• Today, I tried switching to GCS to handle log storage more seamlessly. While logs are indeed being saved in GCS, they are no longer visible in the Airbyte UI. With MinIO, I could see logs populating live in the UI.

• I understand that GCS doesn’t support the same live log streaming as MinIO, but is there any workaround to make GCS logs visible in the Airbyte UI?

Any insights or best practices would be greatly appreciated. If anyone has tackled similar issues or has tips for optimizing MinIO log management, I’d love to hear your approach. Thanks in advance for the help!

Karol U.

[joeybenamy]

We use AWS and had the same issue with Minio in Kubernetes, so we switched to AWS S3 storage for logs and state. We are still able to see the logs in the Airbyte UI. We have Minio fully disabled. If you post your Helm values (with sensitive information obfuscated), we can compare with ours.

[karolur]

hi @joeybenamy thank you for helping me with this.
These are the values I tried today with the help of this site airbyte logs storage:

global:
  storage:
    type: "GCS"
    storageSecretName: "airbyte-storage-secret"
    bucket:
      log: "airbyte_logs_bucket"
      state: "airbyte_logs_bucket"
      workloadOutput: "airbyte_logs_bucket"
    gcs:
      projectId: ***
      credentialsJson: "/secrets/gcs-log-creds/gcp.json"

How were you able to disable Minio ? I thought that just by changing the storage "type" it would be enough.

If you spot something that I might me missing I would greatly appreciate it if you would point it out.

[joeybenamy]

hi @joeybenamy thank you for helping me with this. These are the values I tried today with the help of this site airbyte logs storage:

global:
  storage:
    type: "GCS"
    storageSecretName: "airbyte-storage-secret"
    bucket:
      log: "airbyte_logs_bucket"
      state: "airbyte_logs_bucket"
      workloadOutput: "airbyte_logs_bucket"
    gcs:
      projectId: ***
      credentialsJson: "/secrets/gcs-log-creds/gcp.json"

How were you able to disable Minio ? I thought that just by changing the storage "type" it would be enough.

If you spot something that I might me missing I would greatly appreciate it if you would point it out.

I think the most obvious next step is to disable Minio and see if that fixes streaming logs in the UI from object storage. In our Helm values:

airbyte:
...
  minio:
    enabled: false
...

We also have workloadOutput set to airbyte-storage:

global:
...
  storage:
    type: s3
    bucket:
      workloadOutput: airbyte-storage
...

[justbeez]

@karolur We're also using GCS for logs, and it populates in the UI just fine . . . just keep in mind that it won't move the old MinIO logs for you (so those won't work going forward).

Here's a gist of what we're using in our values.yaml (excuse my use use of YAML anchors—I just like to keep my YAML D.R.Y. when possible. These can be substituted for static values if you prefer):

minio:
  enabled: false

global:
  storage:
    type: GCS
    storageSecretName: your-secret-name
    bucket:
      log: &storage-bucket your-storage-bucket
      state: *storage-bucket
      workloadOutput: *storage-bucket
      activityPayload: *storage-bucket
    gcs:
      projectId: your-project-id
      bucket: *storage-bucket
      credentialsJson: [redacted]

I believe that MinIO is automatically disabled by the global.type selection, but I haven't gotten around to removing it from my config yet.

Make sure you let everything re-start and then run a fresh job run before you try to check the logging in the UI.

If it isn't working, check the logs from both bootloader and elsewhere to ensure you aren't getting an error authenticating to GCS (which is a common issue I see in the Slack channel)

[karolur]

I finally figured it out, it was an issue with permissions to the credentials that I was passing to the helm value. Thank you very much none the less for the quick help and specially for confirming that it was possible to have live logs with other types of storage providers.