[Scheduled Callbacks] Access API queries & subscriptions by Callback ID

[jribbink]

Problem

Developers need a first-class way to query the results of scheduled transactions by callbackID, analogous to how native transactions are queried by txID. Without this, callback observability depends on an external indexer to locate the block which a given callback was included in.

Rough API Sketch

gRPC (AccessAPI)

• Lookup (single response):
  • GetScheduledTransactionResult(GetScheduledTransactionRequest) -> TransactionResultResponse
  • GetScheduledTransaction(GetScheduledTransactionRequest) -> TransactionResponse
  • message GetScheduledTransactionRequest { uint64 id = 1; }
• Subscription (streaming):
  • SubscribeScheduledTransactionResults(SubscribeScheduledTransactionRequest) -> (stream TransactionResultResponse)
  • message SubscribeScheduledTransactionRequest { uint64 id = 1; }

REST (HTTP)

• Lookup:
  • GET /v1/scheduled_transactions/{callback_id}/result
  • GET /v1/scheduled_transactions/{callback_id}/transaction
• WebSocket (/v1/ws):
  • Topic: scheduled_transactions_results
  • Arguments: callback_id
  • Payload: same shape as transaction result messages

Acceptance Criteria

• Given a valid callbackID, a client can:
  • Fetch TransactionResult without an indexer
  • Subscribe and receive the result at execution
  • Retrieve the associated system Transaction (or learn it's not yet available)

Future Scope

Including more verbose endpoints relating to scheduled transactions (e.g. lookup CallbackData for a currently scheduled transaction that has not yet executed) would offer a useful & more direct API for clients. However, these are not must-have items and can be derived via script execution if necessary.

The primary gap at the moment is that there is no mapping for callback ID <-> TX ID once executed, meaning that clients must manually index ALL callback Executed/PendingExecution events to locate their results.

[peterargue]

I updated the description to use transaction instead of callback to match the new naming.

I like this approach. some general thoughts on what will be needed on ANs:

ANs will need to maintain an index of scheduled callbackID to transaction ID (or at least blockID). Since this is a new index that requires execution state data, it will only be available on ANs with indexing enabled. It will not be possible to query execution nodes for this info.

I think as a first pass, we could implement support for the polling endpoints, then add grpc/websockets streaming afterwards.

[peterargue]

Opened onflow/flow#1638 for the protobuf and openapi spec. Opted for a slightly different REST endpoint scheme that more closely matches what we do for transactions and results today. Adding these endpoints:

gRPC:

• GetScheduledTransaction - returns the transaction body of the scheduled transaction by callbackID
• GetScheduledTransactionResult - returns the transaction result of the scheduled transaction by callbackID

REST:

• /scheduled_transactions/{id} - returns the transaction body of the scheduled transaction by callbackID
• /scheduled_transaction_results/{id} - returns the transaction result of the scheduled transaction by callbackID

adding these:

• /system_transactions/{id} - returns the system transaction body for the requested transaction ID
• /system_transaction_results/{transaction_id} - returns the system transaction result for the requested transaction ID

[devbugging]

@peterargue I've raised this concern before but am also sharing it here because it feels like a more appropriate place.

I strongly believe we should include system transaction retrieval in the existing GetTransactionByID endpoint.

I've experienced the confusion myself already when I was developing a tool for scheduled transactions, and I was trying to fetch transaction which executed the scheduled tx, and I couldn't find it. it took me a minute to realize I need to use the GetSystemTxByID endpoint instead, others won't have that much insight.