Configuring Benthos Stream Shutdown

[AlmogAlmado]

Hi Guys,

I am currently using Benthos stream also for files yamls (not API), particularly for multi-job/batch operations.
I wanted to inquire if there is a configuration option available to gracefully shut down the stream once all inputs, along with their respective processing, have been completed.

*I'm using Close input like read_until & generate

Tx,
Almog

[mihaitodor]

Hey @AlmogAlmado have a look at shutdown_timeout and shutdown_delay here. If all inputs shut down, then Benthos shuts down, but will wait for shutdown_delay to pass if configured. shutdown_timeout configures how long Benthos will wait before shutting down if it receives a signal to do so even if some inputs are still active.

[AlmogAlmado]

Hi @mihaitodor 👋,

I hope you're doing well! 🚀

I wanted to share some details about my setup. I have several configurations (YAMLs), and when I run in normal mode, the shutdown process works perfectly. However, in Streaming mode, the streams are always up, which makes perfect sense. Nevertheless, it would be helpful for me if there was an option to shut down once all stream inputs are finished.

I'm aware of the force shutdown option, but I'd prefer to avoid it. Do you think it would be elegant or efficient to have a feature like this?

Thanks

[mihaitodor]

I'll leave that up to @Jeffail, but I think it would be rather confusing as the default behaviour. Might be worth proposing an advanced flag or something that can enable this behaviour explicitly.

[AlmogAlmado]

Thanks @mihaitodor for your input! Currently, I'm using a workaround with a custom image that loops the streams over with a regular launch.

@Jeffail, could you please share your point of view?

[Jeffail]

@AlmogAlmado I'd say that if both of the following conditions are true:

1. Benthos streams mode is run with the --no-api flag (disabling the REST endpoints)
2. All loaded streams have shut themselves down

Then it'd be reasonable for benthos to exit similar to when you're running a single config. I can't say off the top of my head how difficult that would be to implement but I'm happy to take a PR for it as long as it doesn't uproot too much of the internals.

[AlmogAlmado]

@Jeffail

I fully agree, and this aligns perfectly with what I had in mind. I'll create a pull request once I observe increase in my usage. For now, my custom image, although not ideal, gets the job done.

Thanks a lot to both of you, @mihaitodor and @Jeffail!