Skip to content

fix(managed object storage): page through the bucket listing - #1049

Open
alisharpasand wants to merge 1 commit into
UpCloudLtd:mainfrom
alisharpasand:fix/paginate-object-storage-bucket-lookup
Open

fix(managed object storage): page through the bucket listing#1049
alisharpasand wants to merge 1 commit into
UpCloudLtd:mainfrom
alisharpasand:fix/paginate-object-storage-bucket-lookup

Conversation

@alisharpasand

Copy link
Copy Markdown

Problem

getBucket passes nil params to ListObjectStorageBucketMetrics:

apiResp, err := client.ListObjectStorageBucketMetricsWithResponse(ctx, svcUUID, nil)

The API returns 10 buckets when no limit is given, so on a service with more than 10 buckets everything past the first page is invisible to the provider.

The failure is quiet, and then unrecoverable:

  1. Read finds no match, getBucket returns nil, and the resource calls resp.State.RemoveResource(ctx) — the bucket silently leaves state, with no error or warning.
  2. The next apply tries to create it and the API returns 400, because the name is still in use.
  3. terraform import shares this same lookup, so the documented recovery reports Cannot import non-existent remote object.

The bucket is healthy and serving traffic throughout. It simply cannot be managed by Terraform any more, and which bucket is affected depends only on where the page boundary happens to fall.

Reproduction

On a service with 11 buckets, the eleventh in lexical order became unmanageable the moment it was pushed off page one:

GET /1.3/object-storage-2/{uuid}/buckets                    -> 10 buckets  (the 11th absent)
GET /1.3/object-storage-2/{uuid}/buckets?limit=100          -> 11 buckets  (all present)
GET /1.3/object-storage-2/{uuid}/buckets?limit=10&offset=10 -> the 11th alone on page 2

The API is behaving correctly and paginating as documented; only the provider's read is unpaginated.

Fix

Page with an explicit limit, stopping on a short page, instead of assuming a single request returns everything.

Paging rather than passing one large limit: the page size is the API's to decide, and a service may eventually hold more buckets than any single response returns.

Notes

  • go build, go vet and go test ./internal/service/managedobjectstorage/ all pass.
  • The redundant StatusNotFound branch is deliberately left in place to keep the semantic diff small.
  • No acceptance test added — reproducing this needs a service with more buckets than the default page size, which seemed likely to be unwelcome in CI. Happy to add one if you would like it.
  • Changelog entry added under [Unreleased].

getBucket passed nil params to ListObjectStorageBucketMetrics, so it only ever
saw the first page. The API returns 10 buckets by default, which made every
bucket past the tenth invisible to the provider.

The failure is quiet and then unrecoverable. Read finds no match and calls
RemoveResource, so the bucket silently leaves state; the next apply tries to
create it and the API returns 400 because the name is still in use; and
terraform import shares this lookup, so the documented recovery path reports
"Cannot import non-existent remote object". Meanwhile the bucket is healthy
and serving traffic.

Observed on a service with 11 buckets, where the eleventh in lexical order
became unmanageable the moment it was pushed off page one:

    GET /object-storage-2/{uuid}/buckets                -> 10 buckets
    GET /object-storage-2/{uuid}/buckets?limit=100      -> 11 buckets
    GET /object-storage-2/{uuid}/buckets?limit=10&offset=10 -> the 11th

Now pages with an explicit limit and stops on a short page, rather than
assuming one request returns everything.
@alisharpasand
alisharpasand requested a review from a team as a code owner August 12, 2026 07:57
@villevsv-upcloud

Copy link
Copy Markdown
Contributor

Hello and thank you for the contribution and finding the issue.

There might be some additions coming up the to API that could change how we can fetch the bucket details. Due to that, we need to verify the situation before further review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants