Error - nimble too many files open

[federicolepera]

Hi,
I have installed 3.0.2 hpe-csi-driver with helm on ocp with nodes that have 256 CPU each.
The pod nimble-csp fails with these logs:

LogManager error of type OPEN_FAILURE: Failed to set log file
java.io.FileNotFoundException: /var/log/nimble-csp.log
at com.oracle.svm.core.posix.PosixUtils.fileOpen(PosixUtils.java:235)
at java.io.FileOutputStream.open(FileOutputStream.java:580)
at java.io.FileOutputStream.(FileOutputStream.java:213)
at org.jboss.logmanager.handlers.FileHandler.setFile(FileHandler.java:148)
at org.jboss.logmanager.handlers.SizeRotatingFileHandler.setFileInternal(SizeRotatingFileHandler.java:252)
at org.jboss.logmanager.handlers.SizeRotatingFileHandler.setFile(SizeRotatingFileHandler.java:144)
at io.quarkus.runtime.logging.LoggingSetupRecorder.configureFileHandler(LoggingSetupRecorder.java:214)
at io.quarkus.runtime.logging.LoggingSetupRecorder.initializeLogging(LoggingSetupRecorder.java:100)
at io.quarkus.deployment.steps.LoggingResourceProcessor$setupLoggingRuntimeInit35.deploy_0(LoggingResourceProcessor$setupLoggingRuntimeInit35.zig:122)
at io.quarkus.deployment.steps.LoggingResourceProcessor$setupLoggingRuntimeInit35.deploy(LoggingResourceProcessor$setupLoggingRuntimeInit35.zig:36)
at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:141)
at io.quarkus.runtime.Application.start(Application.java:87)
at io.quarkus.runtime.Application.run(Application.java:210)
at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:41)
10:26:25 ERROR [io.qu.application]] (main) Failed to start application: java.lang.IllegalStateException: failed to create a child event loop
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:88)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47)
at io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59)
at io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:78)
at io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:73)
at io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:60)
at io.vertx.core.net.impl.transport.Transport.eventLoopGroup(Transport.java:148)
at io.vertx.core.impl.VertxImpl.(VertxImpl.java:143)
at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92)
at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40)
at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32)
at io.vertx.core.Vertx.vertx(Vertx.java:85)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder.initializeWeb(VertxCoreRecorder.java:105)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder.initializeWeb(VertxCoreRecorder.java:87)
at io.quarkus.deployment.steps.VertxCoreProcessor$buildWeb36.deploy_0(VertxCoreProcessor$buildWeb36.zig:88)
at io.quarkus.deployment.steps.VertxCoreProcessor$buildWeb36.deploy(VertxCoreProcessor$buildWeb36.zig:36)
at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:189)
at io.quarkus.runtime.Application.start(Application.java:87)
at io.quarkus.runtime.Application.run(Application.java:210)
at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:41)
Caused by: io.netty.channel.ChannelException: failed to open a new selector
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:181)
at io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:147)
at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:138)
at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84)
... 20 more
Caused by: java.io.IOException: Too many open files
at com.oracle.svm.core.posix.PosixUtils.newIOExceptionWithLastError(PosixUtils.java:216)
at sun.nio.ch.IOUtil.makePipe(IOUtil.java:507)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:65)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:179)
... 24 more

Exception in thread "main" java.lang.RuntimeException: Failed to start quarkus
at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:275)
at io.quarkus.runtime.Application.start(Application.java:87)
at io.quarkus.runtime.Application.run(Application.java:210)
at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:41)
Caused by: java.lang.IllegalStateException: failed to create a child event loop
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:88)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47)
at io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59)
at io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:78)
at io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:73)
at io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:60)
at io.vertx.core.net.impl.transport.Transport.eventLoopGroup(Transport.java:148)
at io.vertx.core.impl.VertxImpl.(VertxImpl.java:143)
at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92)
at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40)
at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32)
at io.vertx.core.Vertx.vertx(Vertx.java:85)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder.initializeWeb(VertxCoreRecorder.java:105)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder.initializeWeb(VertxCoreRecorder.java:87)
at io.quarkus.deployment.steps.VertxCoreProcessor$buildWeb36.deploy_0(VertxCoreProcessor$buildWeb36.zig:88)
at io.quarkus.deployment.steps.VertxCoreProcessor$buildWeb36.deploy(VertxCoreProcessor$buildWeb36.zig:36)
at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:189)
... 3 more
Caused by: io.netty.channel.ChannelException: failed to open a new selector
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:181)
at io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:147)
at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:138)
at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84)
... 20 more
Caused by: java.io.IOException: Too many open files
at com.oracle.svm.core.posix.PosixUtils.newIOExceptionWithLastError(PosixUtils.java:216)
at sun.nio.ch.IOUtil.makePipe(IOUtil.java:507)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:65)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:179)
... 24 more

Same installation on a cluster with 128 CPU has no problem

[federicolepera]

Hi,
I've discovered that with 256 cpu nodes.. in you put this env var QUARKUS_VERTX_EVENT_LOOPS_POOL_SIZE at the value of 338 the pod nimble-csp works... if you put 339 crash with the posted logs

[datamattsson]

I've discovered that with 256 cpu nodes.. in you put this env var QUARKUS_VERTX_EVENT_LOOPS_POOL_SIZE at the value of 338 the pod nimble-csp works... if you put 339 crash with the posted logs

Thanks for reporting this. I'm surprised we haven't seen this before. The driver has been tested on systems with way more cores than 256 on ARM systems.