Repository navigation
feat(llm): carry the cache-write TTL split (5m/1h) into telemetry attributes - #2
Merged
Conversation
…ributes Producer half of the TTL-exact cache-write metering contract (consumer: synapse-runtime accounting-tap PR agentgateway#39, fields cache_write_5m/1h_tokens). Bedrock Converse: parse TokenUsage.cacheDetails[] (AWS CacheDetail {ttl: 5m|1h, inputTokens}) — buffered responses via ProviderUsage and all three ConverseStream metadata paths — into LLMResponse cache_creation_5m/1h_input_tokens. Aggregate-only responses (older models) leave the split None; unknown TTL classes count under neither canonical class. Anthropic native: read usage.cache_creation.ephemeral_5m/1h_input_tokens from the pass-through rest JSON (Usage::cache_ttl_split) so the client-facing wire stays byte-identical — no re-parsed typed field, no key reordering (golden passthrough snapshot unchanged except the telemetry stanza). Split overlays via overlay_upstream_usage like every other native usage dimension. Telemetry: new span attributes gen_ai.usage.cache_creation.input_tokens.5m / .1h, emitted only when the provider reported the split — absent means unknown, never zero, and the accounting consumer fails closed on ambiguous aggregates rather than approximating. Client protocol semantics unchanged: no client-facing struct or translation was modified. Tests: Bedrock cacheDetails parse (5m-only/1h-only/mixed/aggregate-only), per-class mapping with unknown-class isolation, buffered Converse → LLMResponse end-to-end, Anthropic-native split extraction + aggregate-only fallback, span-attribute emission (split present / aggregate-only absent). agent-llm 423/423, agentgateway 2229/2229; fmt + clippy clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Producer half of the TTL-exact cache-write metering contract for Synapse Provider Pricing Governance V1. Consumer: namirialgroup/synapse-runtime#39 (accounting-tap
cache_write_5m_tokens/cache_write_1h_tokens), which reads exactly these attribute names:gen_ai.usage.cache_creation.input_tokens.5mgen_ai.usage.cache_creation.input_tokens.1hBedrock Converse (the Synapse Bedrock leg): parses
TokenUsage.cacheDetails[](AWSCacheDetail {ttl: 5m|1h, inputTokens}) — buffered viaProviderUsageand all threeConverseStreammetadata paths. Aggregate-only responses leave the splitNone; unknown TTL classes count under neither canonical class.Anthropic native: reads
usage.cache_creation.ephemeral_5m/1h_input_tokensfrom the pass-throughrestJSON (Usage::cache_ttl_split) so the client-facing wire stays byte-identical — no typed re-parse, no key reordering (the golden passthrough snapshot changes only in its telemetry stanza). Split overlays viaoverlay_upstream_usagelike every other native usage dimension.Telemetry: attributes emitted only when the provider reported the split — absent = unknown, never zero; the accounting consumer fails closed (
cache_write_ttl_ambiguous) on ambiguous aggregates rather than approximating. Client protocol semantics unchanged: no client-facing struct or translation modified.Tests: Bedrock
cacheDetailsparse (5m-only / 1h-only / mixed / aggregate-only), per-class mapping with unknown-class isolation, buffered Converse → LLMResponse end-to-end, Anthropic-native split extraction + aggregate-only fallback, span-attribute emission (split present / aggregate-only absent).agent-llm423/423,agentgateway2229/2229; fmt + clippy clean.Note for reviewers: cross-linked Synapse delivery = namirial-synapse
feat/provider-pricing-governance-v1(Bitbucket PR) + namirialgroup/synapse-infra PR agentgateway#263 (CronJob ownership).