Skip to content

Memory leak during reconnection storm test  #514

@dkropachev

Description

@dkropachev

While testing #473 I have found that in scenaior when driver can't properly initialize connections it start eating memory, in my case session endup consuming twice more memory than usual.

Repro:
In order to test both PRs I have setup an environment with a cluster of 2 nodes, 16 smp each.
And a swarm of client containers, each container holds 200 sessions.

Test scenario:

  1. Start cluster of 2 nodes, 16 smp each
  2. Populate the swarm of clients up to 30 clients:
    2.1. Start with 0 containers
    2.2. Grow the number of containers by 10
    2.3. Wait till all shard connections are created
    2.4. Repeat until reach target limit
  3. Restart one of the nodes
  4. Monitor number of connections in the swarm

You need to run it on scylladb/scylla:2025.2.1 with uninitialized_connections_semaphore_cpu_concurrency high enough to let scylla lag and fail connections during initialization.

Most-likely connections are leaking and stay in memory.

Repro code is attached.
test.zip

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions