Random occurrence of "Failed to upload files to FTP server"

[axbazin]

Hello,

I've met a very similar problem to #141 multiple times also, and with webin-cli 8.1.1 as well as 8.2.0. However OP seems to think it comes from their institution firewall, so writing a new issue as this may be another problem with similar log (since it definitely does not come from my institution firewall).

The webin processes go silent for a while, then at some point the FTP connection gets lost, and the process crashes.

The log is the following:

2025-07-01T09:10:55 INFO : Submission has not been validated previously.
2025-07-01T09:11:39 INFO : Submission(s) validated successfully.
2025-07-01T09:11:40 INFO : Connecting to FTP server : webin2.ebi.ac.uk
2025-07-01T09:11:41 INFO : Uploading file: REDACTED
2025-07-01T11:25:08 WARN : Retrying file upload to FTP server.
2025-07-01T11:25:08 INFO : Connecting to FTP server : webin2.ebi.ac.uk
2025-07-01T11:25:42 WARN : Retrying file upload to FTP server.
2025-07-01T11:25:42 INFO : Connecting to FTP server : webin2.ebi.ac.uk
2025-07-01T11:26:00 WARN : Retrying file upload to FTP server.
2025-07-01T11:26:00 INFO : Connecting to FTP server : webin2.ebi.ac.uk
2025-07-01T15:00:07 WARN : Retrying file upload to FTP server.
2025-07-01T15:00:07 INFO : Connecting to FTP server : webin2.ebi.ac.uk
2025-07-01T15:00:23 WARN : Retrying file upload to FTP server.
2025-07-01T15:00:23 INFO : Connecting to FTP server : webin2.ebi.ac.uk
2025-07-01T15:00:38 WARN : Retrying file upload to FTP server.
2025-07-01T15:00:38 INFO : Connecting to FTP server : webin2.ebi.ac.uk
2025-07-01T15:01:09 ERROR: Failed to upload files to FTP server. Failed to upload files to server because of a system error.
uk.ac.ebi.ena.webin.cli.WebinCliException: Failed to upload files to FTP server. Failed to upload files to server because of a system error.
	at uk.ac.ebi.ena.webin.cli.WebinCliException.error(WebinCliException.java:89)
	at uk.ac.ebi.ena.webin.cli.WebinCli.submit(WebinCli.java:371)
	at uk.ac.ebi.ena.webin.cli.WebinCli.execute(WebinCli.java:292)
	at uk.ac.ebi.ena.webin.cli.WebinCli.__main(WebinCli.java:122)
	at uk.ac.ebi.ena.webin.cli.WebinCli.main(WebinCli.java:90)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: uk.ac.ebi.ena.webin.cli.WebinCliException: Failed to upload files to FTP server.
	at uk.ac.ebi.ena.webin.cli.WebinCliException.systemError(WebinCliException.java:69)
	at uk.ac.ebi.ena.webin.cli.upload.FtpService.storeFile(FtpService.java:266)
	at uk.ac.ebi.ena.webin.cli.upload.FtpService.upload(FtpService.java:94)
	at uk.ac.ebi.ena.webin.cli.WebinCli.submit(WebinCli.java:362)
	... 11 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:288)
	at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:314)
	at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
	at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
	at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
	at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
	at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1465)
	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
	at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)
	at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)
	at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)
	at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
	at java.base/java.io.BufferedReader.fill(BufferedReader.java:162)
	at java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)
	at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)
	at org.apache.commons.net.ftp.FTP.getReply(FTP.java:565)
	at org.apache.commons.net.ftp.FTP.getReply(FTP.java:556)
	at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:1222)
	at org.apache.commons.net.ftp.FTPSClient.sendCommand(FTPSClient.java:894)
	at org.apache.commons.net.ftp.FTPSClient.openDataSecureConnection(FTPSClient.java:825)
	at org.apache.commons.net.ftp.FTPSClient._openDataConnection_(FTPSClient.java:281)
	at org.apache.commons.net.ftp.FTPClient._storeFile(FTPClient.java:947)
	at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:3275)
	at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:3296)
	at uk.ac.ebi.ena.webin.cli.upload.FtpService.lambda$storeFile$13(FtpService.java:253)
	at uk.ac.ebi.ena.webin.cli.upload.FtpService.lambda$executeWithReconnect$15(FtpService.java:297)
	at uk.ac.ebi.ena.webin.cli.utils.RetryUtils.lambda$executeWithRetry$0(RetryUtils.java:69)
	at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:329)
	at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:209)
	at uk.ac.ebi.ena.webin.cli.utils.RetryUtils.executeWithRetry(RetryUtils.java:63)
	at uk.ac.ebi.ena.webin.cli.upload.FtpService.executeWithReconnect(FtpService.java:289)
	at uk.ac.ebi.ena.webin.cli.upload.FtpService.storeFile(FtpService.java:246)
	... 13 common frames omitted
2025-07-01T15:01:09 ERROR: Some or all submissions failed. Please see application logs.
uk.ac.ebi.ena.webin.cli.WebinCliException: Some or all submissions failed. Please see application logs.
	at uk.ac.ebi.ena.webin.cli.WebinCliException.systemError(WebinCliException.java:81)
	at uk.ac.ebi.ena.webin.cli.WebinCli.submit(WebinCli.java:413)
	at uk.ac.ebi.ena.webin.cli.WebinCli.execute(WebinCli.java:292)
	at uk.ac.ebi.ena.webin.cli.WebinCli.__main(WebinCli.java:122)
	at uk.ac.ebi.ena.webin.cli.WebinCli.main(WebinCli.java:90)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)

There were 2 surprising things to me:

• The timestamps, this happens at 11:26, then somehow the next failure it at 15:00. I guess it got the connection back, but still did nothing?
• The processes status in the mean time were permanently silenced everytime I checked on them.

For this log specifically, the file I was uploading was 500Mo. Other files of the same size took barely a minute or two and worked perfectly. There is no problem of disk or network or anything like this in the meantime.

In my previous attempts, the FTP failures times were random. Crashes were random and impossible to replicate, sometimes a file would work first time, something it would need more tries. It seemed vaguely linked to file size as files of bigger size (8 Go needed 6 attempts) seem more prone to failure.

If you need anything else to track down the issue don't hesitate.

Have a nice day,

Adelme

[corneliusroemer]

Yep, we're also getting these and it's definitely not firewall related as I commented on the linked issue.

This error has happened frequently to us (Loculus/Pathoplexus) since at least May 19 (before then the Github actions logs are gone): https://github.com/loculus-project/loculus/actions/runs/15257676433/job/42909071358#step:8:130

It's temporary: the error occurs for a few hours, then it's ok again. I wonder if it's related to ENA's dev instance being down when it pulls the latest changes from prod.

@axbazin can you also include the exact webin-cli command you run? Is it for -test or prod? We do most runs on -test and that's where we've seen the failures. We only rarely use prod and I don't think we've ever seen the failure there for the ~10 times we've used prod in the past few months.

[axbazin]

@corneliusroemer This log is for prod. I has not happened to me when uploading to the test instance, but I only do tests with small-ish batches.

If needed, the full webin-cli command I use is the following:

java -jar ~/bin/webin-cli-8.2.0.jar \
        -submit \
        -context="reads" \
        -userName=REDACTED \
        -passwordFile=REDACTED \
        -inputDir=$inputfilepath/$fastqbase/ \
        -manifest=$MANIF \
        -sampleUpdate

The content of the manifest looks like this:

STUDY	PRJEB90659
SAMPLE	ERS25116345
NAME	WGS run 001 mut0254_001
INSTRUMENT	Illumina NovaSeq X
LIBRARY_SOURCE	GENOMIC
LIBRARY_SELECTION	RANDOM
LIBRARY_STRATEGY	WGS
FASTQ	mut0254_1.fq.gz
FASTQ	mut0254_2.fq.gz

Happened today again at 2025-07-03T09:09:41 (CET), after working normally for almost 4h.

[corneliusroemer]

On dev, we're now getting a spurious FTP login error 530 (definitely wrong error message, we're using valid credentials):

DEBUG    ena_deposition.ena_submission_helper:ena_submission_helper.py:594 Invoking webin-cli with args: ['ena-webin-cli', '-username=***', "-centername='Test institution'", '-submit', '-context=genome', "-manifest='/tmp/tmptrzzzvo7.tsv'", "-outputdir='/tmp/tmp732cm6n5'", '-test', '-***']
ERROR    ena_deposition.ena_submission_helper:ena_submission_helper.py:650 Webin CLI command succeeded but did not return ERZ accession.
INFO : Your application version is 8.2.0
INFO : Submission has not been validated previously.
INFO : Creating report file: /tmp/tmp732cm6n5/./webin-cli.report
INFO : Submission(s) validated successfully.
INFO : Connecting to FTP server : webin2.ebi.ac.uk
ERROR: Last received FTP Reply. ReplyCode : 530, ReplyStrings : [530 Login incorrect.]
ERROR: FTP service authentication error. Invalid submission account user name or password. Please try enclosing your password in single quotes. Failed to upload files to server because of a user error.
INFO : This was a TEST submission(s).

[axbazin]

Still facing this problem one month later.

I've chose to implement an additional retry logic on top of webin-cli's own retry logic, since literally all I can do is to relaunch the command.

If those server issues are there to stay, maybe it would be a good idea to let users set up parameters for retries themselves @haseeb-gh ? The current 'default' retries of webin-cli are too close to each other in time for my use case and rarely manage to connect back to the FTP server, while relaunching a few minutes later often does work. Maybe letting us customize the number of retries and delay between them would make webin-cli more resilient to those server issues ?

[haseeb-gh]

We're deeply sorry for these issues. Here at EBI we are constantly trying to sort these problems out for good. FTP availability is definitely an issue. On top of that, the authentication system in place to let users upload data using their webin credentials some times acts up too which confuses users because the error they see tells them there is something wrong with their username or password.

This is what you guys @axbazin and @corneliusroemer and most probably lots of other users are experiencing. We continue to work as hard as we can to fix the issues so please bear with us.

@axbazin your suggestion on making retry settings configurable is something we can definitely explore if you think it will help. The current retry configuration is old and was designed this way based on how severe the problem was back then. And it worked well at the time but maybe now it needs to be revised. Leave it with us. We will review this and come up with a plan as soon as possible.

Thank you for your patience.

[axbazin]

No worries, it is perfectly understandable that things are not perfect for such a service that is constantly under constraint. Thank you for trying to make things better.

It would definitely help in the context I'm working with (big fastq files, not-so-fast bandwidth), as some of the files take multiple attemps over days to finally get properly uploaded to ENA servers.

Thank you very much for your time and answer @haseeb-gh.

[pontushojer]

Hi @axbazin, I was also experiencing these issues a lot, trying to upload large paired FASTQs over several days but failing multiple times due to server errors. Today I changed to use the Aspera CLI and it has worked great for the files I have run so far, much more stable. If you have the ability to install Aspera that might be a good option.

[axbazin]

Hi @pontushojer , alright I will try this for my next run, thank you for the tip.

[corneliusroemer]

For visibility into how common or not this is, we've got this error again just now on prod:

2025-11-05T14:04:07 INFO : Submission(s) validated successfully.
2025-11-05T14:04:07 INFO : Connecting to FTP server : webin2.ebi.ac.uk
2025-11-05T14:04:11 ERROR: Last received FTP Reply. ReplyCode : 530, ReplyStrings : [530 Login incorrect.]
2025-11-05T14:04:11 ERROR: FTP service authentication error. Invalid submission account user name or password. Please try enclosing your password in single quotes. Failed to upload files to server because of a user error.
uk.ac.ebi.ena.webin.cli.WebinCliException: FTP service authentication error. Invalid submission account user name or password. Please try enclosing your password in single quotes. Failed to upload files to server because of a user error.

We haven't changed anything on our end.

[axbazin]

On my side, I bypassed this problem using a moderately aggressive crontab:

• Every hour, a job checks the upload statuses:
  • If everything has been uploaded successfully (information gotten from the xlm files webin-cli produces), then fine, stop everything.
  • If a job is running, don't do anything.
  • If the job failed, relaunch it.

This goes on forever until all files are properly uploaded (or my disk eventually gets filled with log files of all my jobs, whichever comes first).
A bit hacky, but at least this problem stopped bothering me everytime I need to upload data to ENA.

For the Aspera CLI solution sadly I did not have the ability to install it in the end.

[haseeb-gh]

@corneliusroemer Thank you for letting us know about this. We have started investigating the problem. It seems to something that keeps occurring from time to time. I would suggest that you keep retrying as often as you can. Once resolved, your submissions will go through automatically without making any kind of change.

[haseeb-gh]

@corneliusroemer everything seems to be ok on our side, were you able to try again successfully? If you are still experiencing the same problem then please reach out to us through:
https://www.ebi.ac.uk/ena/browser/support

We will need some more information from you e.g. Webin account name to investigate this further.

Thank you for your patience

• Haseeb