Cannot connect to the server

[serwansj]

Actions before raising this issue

• I searched the existing issues and did not find anything similar.
• I read/searched the docs

Steps to Reproduce

Whenever i am trying to access the CVAT server on my host, I encounter the error Cannot connect to the server (see image below). Specifically, it is the request to /api/server/health/?format=json&org= that fails as can be seen in the network logs.

Executing docker exec -t cvat_server python manage.py health_check on the machine that runs the CVAT server also hangs. I also checked docker logs and didn't spot any hints to the issue.

This only happens after the CVAT server has been running for a couple of days. If I restart the server, everything works fine again for a couple of days.

Expected Behavior

Having access to the CVAT UI

Possible Solution

No response

Context

No response

Environment

No response

[antortjim]

I have the exact same problem, and I am trying with multiple versions of the repo (master, develop and a couple of tags). I come from an existing installation that used to work but after a while not using it, now I needed to get back to it and I am encountering this issue.

[antortjim]

This screenshot shows the errors I get with master (a33f7f57088744bab61f18e8a8cf6528a0c22fd2).

[antortjim]

The problem is my root partition was >90% used. I saw the message in the logs, but didn't realize it was an error (thought it was just a warning). I am not sure this is the desired behavior anyway.

[serwansj]

unfortunately, on my end that is not the issue. it also stops working even if the machine is not being used at all.

[eliabruni]

Same problem here!

[bsekachev]

Hello, please provide output of 'docker ps' and 'docker logs cvat_server'
Also the version you are using and OS

[antortjim]

unfortunately, on my end that is not the issue. it also stops working even if the machine is not being used at all.

I meant it seems that if 90% of the storage is used, cvat won't work (even if the computer is not busy or not being used at all like you said). So it's not RAM, it's storage.

[bsekachev]

I meant it seems that if 90% of the storage is used, cvat won't work

That is true

[serwansj]

@antortjim i got that, however, what i meant was that even if nothing is changed in the storage it also stops working.

@bsekachev i will post the outputs and logs once it stops working again. just restarted the server after i opened this issue, so i will probably have to wait a couple of days

[stone100010]

事情没有大家想象的那么麻烦，重启一下你的服务器即可，一切都会恢复如常。祝各位好运~

[serwansj]

issue came up again. here are the logs:

• docker ps:

b0b7dd6ee112   cvat/ui:dev                                "/docker-entrypoint.…"   2 weeks ago    Up 2 weeks    80/tcp                                                                                                                                                           cvat_ui
a05539e659f8   cvat/server:dev                            "./backend_entrypoin…"   2 weeks ago    Up 2 weeks    8080/tcp, 0.0.0.0:9095->9095/tcp, :::9095->9095/tcp                                                                                                              cvat_worker_analytics_reports
4d0d39ea5d11   cvat/server:dev                            "./backend_entrypoin…"   2 weeks ago    Up 2 weeks    8080/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp                                                                                                              cvat_server
b92136d521a4   cvat/server:dev                            "./backend_entrypoin…"   2 weeks ago    Up 2 weeks    8080/tcp                                                                                                                                                         cvat_worker_webhooks
d5cffa902e7e   cvat/server:dev                            "./backend_entrypoin…"   2 weeks ago    Up 2 weeks    8080/tcp, 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                                                                                                              cvat_worker_export
42a44c76d241   cvat/server:dev                            "./backend_entrypoin…"   2 weeks ago    Up 2 weeks    8080/tcp, 0.0.0.0:9094->9094/tcp, :::9094->9094/tcp                                                                                                              cvat_worker_quality_reports
5877770cc680   cvat/server:dev                            "./backend_entrypoin…"   2 weeks ago    Up 2 weeks    8080/tcp, 0.0.0.0:9091->9091/tcp, :::9091->9091/tcp                                                                                                              cvat_worker_annotation
a6679a1ecbce   cvat/server:dev                            "./backend_entrypoin…"   2 weeks ago    Up 2 weeks    8080/tcp                                                                                                                                                         cvat_utils
539b7da21db7   cvat/server:dev                            "./backend_entrypoin…"   2 weeks ago    Up 2 weeks    8080/tcp, 0.0.0.0:9093->9093/tcp, :::9093->9093/tcp                                                                                                              cvat_worker_import
e003873070e5   timberio/vector:0.26.0-alpine              "/usr/local/bin/vect…"   2 weeks ago    Up 2 weeks                                                                                                                                                                     cvat_vector
c9a18a20c542   clickhouse/clickhouse-server:22.3-alpine   "/entrypoint.sh"         2 weeks ago    Up 2 weeks    8123/tcp, 9000/tcp, 9009/tcp                                                                                                                                     cvat_clickhouse
5bd40f9cd48a   grafana/grafana-oss:10.1.2                 "sh -euc 'mkdir -p /…"   2 weeks ago    Up 2 weeks    3000/tcp                                                                                                                                                         cvat_grafana
1c104405c418   eqalpha/keydb:x86_64_v6.3.2                "docker-entrypoint.s…"   2 weeks ago    Up 2 weeks    6379/tcp                                                                                                                                                         cvat_redis
ca21f853a558   postgres:15-alpine                         "docker-entrypoint.s…"   2 weeks ago    Up 2 weeks    0.0.0.0:5432->5432/tcp, :::5432->5432/tcp                                                                                                                        cvat_db
5bc5bcfeb2b0   traefik:v2.9                               "/entrypoint.sh --pr…"   2 weeks ago    Up 2 weeks    0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp   traefik
a9e0a69fd0cf   openpolicyagent/opa:0.45.0-rootless        "/opa run --server -…"   2 weeks ago    Up 2 weeks    0.0.0.0:8181->8181/tcp, :::8181->8181/tcp                                                                                                                        cvat_opa

• docker logs cvat_server (the following keeps repeating a bunch, probably because the server keeps attempting to build a connection?):

2024-02-12 01:27:22,106 DEBG 'uvicorn-1' stdout output:
INFO:     192.168.80.5:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

[bsekachev]

Please, provide all logs from cvat_server, cvat_traefik and cvat_redis.

Also, please try to upgrade, we got rid of keydb storage (in cvat_redis container) because it was not stable.

[devwasabi]

Hey guys. I am having the same issue as the guy who raised this issue. I've always had memory issues with cvat, since I'm using my local machine as storage. I am using cvat within wsl. Yesterday I decided to move cvat to my 2nd drive which has more storage (around 1 terabyte). Cvat was working seamlessly until I started to export my datasets.

Thereafter, +- 300 gigs of extra memory suddenly got decreased to a few gigs. Now, all of a sudden I have only 6 gigs left. I have already checked the logs for cvat_server, and it's here below:
[2024-02-20 07:34:29,941] ERROR health-check: warning: 603a12ee426c 92.9% disk usage exceeds 90%
It's clear that the issue is memory, it's just that I am not sure how to fix it now, since i am using local storage instead of cloud storage. My annotations are probably a lot. Is there a way to stop the exportation and restart it. I think it may have been that I made multiple export requests, because I wasn't getting a quick response (the downloading of the dataset). I feel like I may be suffering the repurrcussions of exporting multiple datasets simultaneously, and now my storage is taking the lagging backlash.

[serwansj]

@devwasabi this is not the same issue as i was having since mine was not related to running out of storage