Skip to content

Global gRPC Deadline #1020

@venilinvasilev

Description

@venilinvasilev

Summary

Currently, SDKs do not consistently handle gRPC request timeouts across implementations.
While the Java SDK includes a configurable per-request grpcDeadline, other SDKs either lack this feature or require it to be manually set per transaction (e.g., via Transaction.setGrpcDeadline()).

This results in inconsistent behavior and potentially poor user experience - especially in cases where a node becomes unresponsive or experiences long response times (e.g., Cloudflare 522 timeout scenarios).

The goal of this issue is to introduce a global, SDK-wide grpcDeadline setting that defines a maximum timeout per gRPC request, after which the SDK marks the node as unhealthy and automatically rotates to the next available node.


Background

  • When a node is slow or unresponsive, the SDK can currently hang for extended periods (e.g., 30–40 seconds) before failing.
  • Inconsistent timeout handling across SDKs leads to different behavior between languages, making it harder for developers to reason about network reliability and retry behavior.
  • The Java SDK already provides a Client.setGrpcDeadline(Duration) setting, which applies a consistent timeout to all requests.
  • Other SDKs rely on per-transaction configuration or lack timeout enforcement entirely.

Proposal

Introduce a global grpcDeadline configuration at the client level for all SDKs:

const client = Client.forMainnet();
client.setGrpcDeadline(5_000); // 5 seconds per gRPC request

Behavior

  • Applies to all outgoing gRPC requests (queries, transactions, etc.).
  • If a request exceeds this deadline:
    • The SDK aborts the request.
    • Marks the node as temporarily unhealthy.
    • Rotates to the next healthy node automatically.
  • Can be overridden per transaction or query, maintaining backward compatibility with Transaction.setGrpcDeadline() (or equivalent).

Relationship with requestTimeout

The SDK already defines a requestTimeout property on the client, which controls the total time budget for a complete Transaction or Query execute operation — including:

  • Attempting the first node.
  • Failing due to a node error.
  • Applying retry/backoff logic.
  • Retrying with another healthy node.

To ensure consistent and predictable behavior:

  • requestTimeout must be greater than or equal to the per-request grpcDeadline.
  • The SDK should enforce this constraint at runtime or initialization (e.g., throw or adjust automatically if misconfigured).
  • A sensible default should be introduced for requestTimeout (e.g., 2 minutes) if not already defined.

This ensures that the SDK does not abort the overall operation prematurely while still respecting per-node timeouts.


Defaults and Recommendations

Parameter Description Default Notes
grpcDeadline Global timeout per gRPC request 10 seconds Per-node timeout
requestTimeout Maximum total duration for a Transaction or Query execute operation (including retries) 2 minutes Must be ≥ grpcDeadline

Benefits

  • Consistent timeout handling across SDKs.
  • Prevents long “hang” periods caused by slow or unresponsive nodes.
  • Improves UX by failing fast and rotating predictably.
  • Aligns with Java SDK’s existing behavior.
  • Introduces clear relationship between per-request and total operation timeouts.

Tasks

  • Define grpcDeadline property on client config for all SDKs.
  • Enforce requestTimeout >= grpcDeadline.
  • Ensure all gRPC-based requests respect the global deadline.
  • Implement consistent error classification and node rotation behavior.
  • Add defaults for both settings (grpcDeadline and requestTimeout).
  • Update SDK reference docs and examples.

Acceptance Criteria

  • All SDKs have a global grpcDeadline configuration.
  • SDK enforces requestTimeout >= grpcDeadline.
  • Timeout logic is consistent across SDKs.
  • SDK rotates to next node on timeout.
  • Documentation clearly explains both timeouts and how to override them.

Labels: enhancement, cross-sdk, network, timeout, ux

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions