Add claim admission, fairness, and resource-governance hooks

[ccarvalho-eng]

Problem

Squidie lets host workers provide capacity through execute_next/1, and workflow-level queue routing was established in #136. The runtime has no standard admission/fairness contract for preventing one partition, workflow, or resource class from monopolizing claims or overwhelming a downstream service.

Partitions isolate identity but do not solve noisy-neighbor behavior. Requiring every host to invent claim ordering, concurrency accounting, and rate-budget integration weakens the operational story.

Reference: https://www.inngest.com/docs/functions/concurrency

Proposed solution

Add a backend-neutral claim admission and fairness policy boundary:

config :squidie,
  admission_policy: MyApp.SquidieAdmission

defmodule MyApp.SquidieAdmission do
  @behaviour Squidie.Runtime.AdmissionPolicy

  def classify(attempt, context) do
    %{
      priority: 50,
      concurrency_keys: [
        {:partition, context.partition, 10},
        {:workflow, context.workflow, 25},
        {:resource, "payments-api", 5}
      ],
      rate_keys: [{"payments-api", 100, :second}]
    }
  end
end

Squidie owns durable attempt/claim truth and calls a trusted host policy. Concrete global counters, token buckets, or backend leases may be implemented by a capability adapter; workflow modules remain backend-neutral.

Semantics

• Classifications are derived from trusted registry/configuration, never arbitrary persisted module names or request callbacks.
• Priority has bounded documented ordering and cannot bypass visibility time or dependencies.
• Admission denial leaves work visible with a bounded next-check time/reason; it is not failure or retry and does not consume attempt budget.
• Concurrency keys count active claimed execution, not sleeping/waiting runs.
• Claim acquisition and concurrency-slot acquisition must have one repairable/fenced protocol; leaked slots are reclaimed after lease expiry.
• Fairness prevents perpetual starvation across eligible classes, even with priorities.
• Partition/workflow/resource values used for metrics remain bounded or host-normalized.
• Rate/concurrency policy changes do not rewrite workflow definitions or journal history.
• When no policy/adapter is configured, current claim behavior remains unchanged.

Implementation plan

• Define admission classification, decision, denial reason, and capability contracts.
• Add trusted policy configuration and validation.
• Integrate admission with dispatch candidate selection and claim fencing.
• Define concurrency-slot lifecycle across claim, heartbeat, completion, failure, expiry, and cancellation.
• Add bounded priority plus anti-starvation/fair selection.
• Define adapter contracts for distributed concurrency and rate budgets.
• Keep external backend leases separate from Squidie journal claim leases.
• Expose admission state through inspection, explanation, status, and telemetry.
• Add partition-aware examples after partition support lands.
• Document when to use queue routing, worker pool sizing, concurrency, rate limits, and downstream idempotency.

Acceptance criteria

• A configured partition/workflow/resource limit prevents claims above capacity across nodes.
• Waiting/sleeping/manual runs consume no execution slots.
• Completion, failure, cancellation, and lease expiry release or recover capacity.
• Crash windows cannot permanently leak capacity or create an unfenced double claim.
• Low-priority eligible work eventually runs under sustained high-priority load.
• Admission denial is inspectable and does not consume retry budget.
• Policy failure fails safely without claiming work under unknown rules.
• Default behavior is unchanged when the feature is not configured.
• Workflow definitions remain backend-neutral.

Verification

• Multi-node concurrency, lease-expiry, crash-window, cancellation, fairness, priority, rate-budget, policy-failure, and telemetry tests.
• Bedrock reference-adapter smoke coverage where applicable.
• mix precommit
• Multi-node example-host verification.

Out of scope

• Authorization or billing quotas.
• A mandatory rate-limiter dependency in Squidie core.
• Exactly-once external side effects.
• Letting workflow-authored or request data choose executable policy modules.
• Replacing backend-owned delivery leases with admission slots.

[ccarvalho-eng]

I'll pick this one

[ccarvalho-eng]

This seems a lot more complex than I anticipated and I wonder if it's worth doing now.