Memory Leak when setting `options.timeout` per subgraph in transportEntries

[jaffemd]

Versions:

node: 22.10.0
"@graphql-hive/gateway": "1.6.6"
"@graphql-hive/gateway-runtime": "1.3.13"

Hi! We currently have a hive gateway setup to serve a supergraph.

We recently tried adding timeout settings per subgraph. Below is our config:

const allowedHeaders = [
  //...
];

const subgraph1Location = process.env.SUBGRAPH_1_LOCATION;
const subgraph1Timeout = process.env.SUBGRAPH_1_TIMEOUT;

const subgraph2Location = process.env.SUBGRAPH_2_LOCATION;
const subgraph2Timeout = process.env.SUBGRAPH_2_TIMEOUT;

export const gatewayConfig = defineConfig({
  supergraph: "supergraph.graphql",
  port: parseInt(process.env.PORT || "8000"),
  cors: {
    origin: "*",
    methods: ["GET", "POST", "PUT", "DELETE", "OPTIONS"],
    allowedHeaders,
  },
  landingPage: false,
  executionCancellation: true,
  upstreamCancellation: true,
  disableIntrospection: true,
  batching: false,
  pollingInterval: 31_556_952_000,
  maskedErrors: false,
  graphiql: false,

  transportEntries: {
    'subgraph1': {
      location: subgraph1Location,
      options: {
        timeout: subgraph1Timeout,
      },
    },
    'subgraph2': {
      location: subgraph2Location,
      options: {
        timeout: subgraph2Timeout,
      },
    },
  }
});

After a couple of days, we observed a memory leak and traced it to this code change on our side.

After trying several things, the single change that fixed our memory leak was removing the options.timeout setting in transportEntries. We moved the per subgraph timeout to a single setting on the newly added requestTimeout option, and the memory leak was immediately fixed.

  requestTimeout: process.env.REQUEST_TIMEOUT,
  
  transportEntries: {
    'subgraph1': {
      location: subgraph1Location,
    },
    'subgraph2': {
      location: subgraph2Location,
    },

[jaffemd]

Hi! I saw the new upstreamTimeout feature you just implemented and it's exactly what we're looking for! Unfortunately, we're seeing the same memory leak issue with this as we were seeing with our previous implementation. Below is the our memory usage graph before and after turning on this feature.

Here's our implementation:

const UPSTREAM_TIMEOUTS: Record<string, number> = {
  //[subgraph1]: 2000
  //...
}

 upstreamTimeout: ({ subgraphName }: TimeoutFactoryPayload): number | undefined {
    if (!subgraphName) { return requestTimeout; }

    return UPSTREAM_TIMEOUTS[subgraphName] || 10000;
  },

[ardatan]

Thanks for creating the issue and the feedback!
Could you test with the alphas here?
#407 (comment)

[jaffemd]

@ardatan Thank you for the proposed fix! However, we just tested the alpha release for a couple hours and noticed that it didn't help, and actually might have increased the rate of leaking memory.

[enisdenjo]

Can you try the latest LTS Node (v22.12.0)? Just to make sure it's not a Node quirk (which we've had happen before 😅).

[jaffemd]

@enisdenjo Thanks for the suggestion, but we're already using node v22.12.0.

[enisdenjo]

Ah ok cool, because I see v22.10.0 in the issue description.

[jaffemd]

Ah, we must've upgraded to 22.12.0 since I initially logged the issue. But I did verify that we're currently on 22.12.0

[dotansimha]

Hi @jaffemd, is there any chance we can get your help with reproducing this issue somehow? We've tried to reproduce it, but it does not seem to leak.

[dotansimha]

Also, @jaffemd , is it possible for you to share a memory snapshot?