Skip to content

Conversation

@Renizmy
Copy link

@Renizmy Renizmy commented Oct 26, 2025

Proposed changes

  • Add a new function to offer capacity to to consume events by configurable batches
  • Introduce parameters to control batch size, batch timeout (maximum wait time before sending a batch), and the maximum number of batches that can be sent per minute.

Checklist

  • I consider the submitted work as finished
  • I tested the code for its functionality
  • I wrote test cases for the relevant uses case
  • I added/update the relevant documentation (either on github or on notion)
  • Where necessary I refactored code to improve the overall quality

Further comments

Some stream connectors face limitations due to low rate limits imposed by certain SIEM providers (for example, Azure Sentinel).
This change allows events to be processed and synchronized in configurable batches, improving throughput and simplifying the integration of connectors with constrained ingestion rates.

Future improvements:

Add a parameter to handle deduplication of events related to the same object, so that only the latest event is kept and processed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant