MANTA-5503 buckets-mdapi should have batch metadata updates#65
Conversation
cneira
force-pushed
the
MANTA-5484
branch
2 times, most recently
from
31f930e to
e01c570
Compare
danmcd
left a comment
danmcd
left a comment
There was a problem hiding this comment.
Quick pass 0. Missing MPL 2.0 headers everywhere and a copyright bump that shouldn't be in an otherwise unchanged file.
Pass 1 on Friday.
New Fast RPC endpoint that accepts multiple UpdateObjectPayload items, groups them by vnode, and executes each vnode group within a single PostgreSQL transaction. Objects in the same vnode succeed or fail atomically. === AI-METADATA === AI-Role: code generation Human-Role: code review, implementation guidance, design decisions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
danmcd
left a comment
danmcd
left a comment
There was a problem hiding this comment.
I'm worried the stated performance bounds in the comments are incorrect.
This is an incremental. Only rpc_handlers.rs remain for me to review.
danmcd
left a comment
danmcd
left a comment
There was a problem hiding this comment.
Pass 1 complete. Will return after blocking comments addressed.
Fix syntax error (missing dot and semicolon) on the 'which pg_tmp' expect call and replace generic "failed to execute process" messages with descriptive ones that name the command and suggest what to check. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
danmcd
left a comment
danmcd
left a comment
There was a problem hiding this comment.
Missing one unresolved in rpc_handlers.rs .
danmcd
left a comment
danmcd
left a comment
There was a problem hiding this comment.
Please address or dismiss my last comment? It's the only thing blocking +1 from me here.
This has been fixed. |
danmcd
left a comment
danmcd
left a comment
There was a problem hiding this comment.
Look forward to seeing your why-not-Rust-1.63 survey. Please file a ticket if it's possible to fix this, or at least report somewhere visible why not.
3 participants
Today, manta-rebalancer does not acknowledge manta-buckets-api objects, which means those are not rebalanced/evacuated to the appropriate storage nodes.
To evacuate an object, the following events must happen:
[This is similar to what was done in ] (https://github.com/cneira/monitor-reef/blob/15383e082579a663ca16a4a4b79e1e46b30e3e21/libs/moray/src/objects.rs#L282)
This PR deals with (1). This new RPC endpoint helps to minimize network round-trips by updating objects via batches. Consumers of this new call will receive an object of type
BatchUpdateObjectsResponse, which contains the failed payload of objects, to be able to be resubmitted again.Current update object functionality in mdapi has added a
sharksfield as optional; this was intentional to not break current consumers of mdapi. If the shark metadata is not present on the request, then the object's shark metadata will not be touched.The only consumer of this feature today will be manta-rebalancer.