fix: missing -dir flag for seaweedfs

[aldy505]

Closes #3990

Data won't be persisted otherwise. Folks needs to re-create the volume.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.49%. Comparing base (0ed3569) to head (d134098).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3991   +/-   ##
=======================================
  Coverage   99.49%   99.49%           
=======================================
  Files           3        3           
  Lines         197      197           
=======================================
  Hits          196      196           
  Misses          1        1           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mjonasz]

can confirm - this PR fixes the issue with broken events details after sentry restart

[cyfran]

I edited section seaweedfs: added "-dir=/data" and rebuild seaweedfs and run systemctl restart docker.service, but error for request "GET /api/0/organizations/.../events/api:ae32c0b82b3c4027a08b76785ee9b279/ HTTP/1.0" 500 72" still persist.
Do I need to rebuild all the containers?

[mjonasz]

Do I need to rebuild all the containers?

probably not, but please note that events before adding this fix, won't be recovered (all events details before fix will be lost as they were stored in /tmp inside seaweedfs container); only new events will be accessible

you may verify whether seaweedfs is started correctly using something like (look for added -dir line) :

> docker container inspect sentry-self-hosted-seaweedfs-1  | grep Cmd -A10
            "Cmd": [
                "server",
                "-dir=/data",
                "-filer=true",
                "-filer.port=8888",
                "-filer.port.grpc=18888",
                "-filer.defaultReplicaPlacement=000",
                "-master=true",
                "-master.port=9333",
                "-master.port.grpc=19333",
                "-metricsPort=9091",

[aldy505]

I edited section seaweedfs: added "-dir=/data" and rebuild seaweedfs and run systemctl restart docker.service, but error for request "GET /api/0/organizations/.../events/api:ae32c0b82b3c4027a08b76785ee9b279/ HTTP/1.0" 500 72" still persist. Do I need to rebuild all the containers?

No need to rebuild, and everything before this patch is lost. One thing that we can do is to find /tmp -maxdepth 1 -type f -name "*.dat" -exec mv -t /data/ {} + first thing before docker compose down is executed by the install script

[cyfran]

I edited section seaweedfs: added "-dir=/data" and rebuild seaweedfs and run systemctl restart docker.service, but error for request "GET /api/0/organizations/.../events/api:ae32c0b82b3c4027a08b76785ee9b279/ HTTP/1.0" 500 72" still persist. Do I need to rebuild all the containers?

No need to rebuild, and everything before this patch is lost. One thing that we can do is to find /tmp -maxdepth 1 -type f -name "*.dat" -exec mv -t /data/ {} + first thing before docker compose down is executed by the install script

I did not find any *.dat files in /tmp and /data directory

/data # ls -l
total 8
drwxr-xr-x 10 root root 122 Sep 22 12:34 filerldb2
drwxr-xr-x 30 root root 4096 Oct 2 12:29 idx
drwxr-xr-x 3 root root 61 Oct 8 13:22 m9333
-rw-r--r-- 1 root root 36 Oct 8 12:31 vol_dir.uuid
/data # cd /tmp/
/tmp # ls -la
total 260
drwxrwxrwt 1 root root 4096 Oct 8 13:31 .
drwxr-xr-x 1 root root 86 Oct 8 13:18 ..
srwxr-xr-x 1 root root 0 Oct 8 13:22 seaweedfs-filer-8888.sock
srwxr-xr-x 1 root root 0 Oct 8 13:22 seaweedfs-s3-8333.sock
-rw-r--r-- 1 root root 69814 Oct 8 13:57 weed.816565a05250.root.log.ERROR.20251008-133103.1
-rw-r--r-- 1 root root 12301 Oct 8 13:20 weed.816565a05250.root.log.INFO.20251008-131844.1
-rw-r--r-- 1 root root 78966 Oct 8 13:57 weed.816565a05250.root.log.INFO.20251008-132212.1
-rw-r--r-- 1 root root 412 Oct 8 13:20 weed.816565a05250.root.log.WARNING.20251008-131845.1
-rw-r--r-- 1 root root 69902 Oct 8 13:57 weed.816565a05250.root.log.WARNING.20251008-132213.1
lrwxrwxrwx 1 root root 50 Oct 8 13:31 weed.ERROR -> weed.816565a05250.root.log.ERROR.20251008-133103.1
lrwxrwxrwx 1 root root 49 Oct 8 13:22 weed.INFO -> weed.816565a05250.root.log.INFO.20251008-132212.1
lrwxrwxrwx 1 root root 52 Oct 8 13:22 weed.WARNING -> weed.816565a05250.root.log.WARNING.20251008-132213.1
/tmp # cd /data/
/data # find . -name "*.dat"
/data #

As I see service is running with parameters:

1 root      0:02 weed server -dir=/data -filer=true -filer.port=8888 -filer.port.grpc=18888 -filer.defaultReplicaPlacement=000 -master=true -master.port=9333 -master.port.grpc=19333 -metricsPort=9091 -s3=true -s3.port=8333 -s3.port.grpc=18333 -volume=true -volume.dir.idx=/data/idx -volume.index=leveldbL

In weed.ERROR now I see:

E1008 14:14:28.394891 stream.go:107 request_id:09cf168c-ff00-4c40-8366-0c9847624658operation LookupFileId 3,702d12a1c7f0aa failed, err: urls not found
E1008 14:14:28.394927 filer_server_handlers_read.go:262 request_id:09cf168c-ff00-4c40-8366-0c9847624658 failed to prepare stream content /buckets/nodestore/nodestore/9af719e79db68f2cb4c16dee6e725a1d: operation LookupFileId 3,702d12a1c7f0aa failed, err: urls not found
E1008 14:14:28.394954 common.go:310 ProcessRangeRequest: operation LookupFileId 3,702d12a1c7f0aa failed, err: urls not found
E1008 14:14:31.668020 stream.go:107 request_id:8f817688-f1d7-417a-bb99-90bb7a303051operation LookupFileId 3,702d12a1c7f0aa failed, err: urls not found
E1008 14:14:31.668042 filer_server_handlers_read.go:262 request_id:8f817688-f1d7-417a-bb99-90bb7a303051 failed to prepare stream content /buckets/nodestore/nodestore/9af719e79db68f2cb4c16dee6e725a1d: operation LookupFileId 3,702d12a1c7f0aa failed, err: urls not found
E1008 14:14:31.668055 common.go:310 ProcessRangeRequest: operation LookupFileId 3,702d12a1c7f0aa failed, err: urls not found

How I can resolve this? Only purge volume with all data? ):

volumes:
  - "sentry-seaweedfs:/data"

[aldy505]

I edited section seaweedfs: added "-dir=/data" and rebuild seaweedfs and run systemctl restart docker.service, but error for request "GET /api/0/organizations/.../events/api:ae32c0b82b3c4027a08b76785ee9b279/ HTTP/1.0" 500 72" still persist. Do I need to rebuild all the containers?

No need to rebuild, and everything before this patch is lost. One thing that we can do is to find /tmp -maxdepth 1 -type f -name "*.dat" -exec mv -t /data/ {} + first thing before docker compose down is executed by the install script

I did not find any *.dat files in /tmp and /data directory

You didn't find any because you've rebuilt the Docker container and re-run it with the correct flag (by including the -dir flag).

In weed.ERROR now I see:

E1008 14:14:28.394891 stream.go:107 request_id:09cf168c-ff00-4c40-8366-0c9847624658operation LookupFileId 3,702d12a1c7f0aa failed, err: urls not found E1008 14:14:28.394927 filer_server_handlers_read.go:262 request_id:09cf168c-ff00-4c40-8366-0c9847624658 failed to prepare stream content /buckets/nodestore/nodestore/9af719e79db68f2cb4c16dee6e725a1d: operation LookupFileId 3,702d12a1c7f0aa failed, err: urls not found E1008 14:14:28.394954 common.go:310 ProcessRangeRequest: operation LookupFileId 3,702d12a1c7f0aa failed, err: urls not found E1008 14:14:31.668020 stream.go:107 request_id:8f817688-f1d7-417a-bb99-90bb7a303051operation LookupFileId 3,702d12a1c7f0aa failed, err: urls not found E1008 14:14:31.668042 filer_server_handlers_read.go:262 request_id:8f817688-f1d7-417a-bb99-90bb7a303051 failed to prepare stream content /buckets/nodestore/nodestore/9af719e79db68f2cb4c16dee6e725a1d: operation LookupFileId 3,702d12a1c7f0aa failed, err: urls not found E1008 14:14:31.668055 common.go:310 ProcessRangeRequest: operation LookupFileId 3,702d12a1c7f0aa failed, err: urls not found

How I can resolve this? Only purge volume with all data? ):

Probably, I have no idea as well, this is the first time I encounter this issue.

Taking everything aside, what's your SENTRY_RETENTION_DAYS value?

[cyfran]

But If I remove/wipe all /data and rebuild - new alerts will be writing and accessible? At now new alerts also can not find in storage:

E1008 14:58:09.051937 stream.go:107 request_id:54639a84-dd72-4e78-9345-dfc5e96d4c9coperation LookupFileId 5,70ff67d48924d4 failed, err: urls not found
E1008 14:58:09.051958 filer_server_handlers_read.go:262 request_id:54639a84-dd72-4e78-9345-dfc5e96d4c9c failed to prepare stream content /buckets/nodestore/nodestore/dbac2b5fcba7c7d23c6f33ea7fd437b8: operation LookupFileId 5,70ff67d48924d4 failed, err: urls not found
E1008 14:58:09.051973 common.go:310 ProcessRangeRequest: operation LookupFileId 5,70ff67d48924d4 failed, err: urls not found

[root@log self-hosted]# cat .env
COMPOSE_PROJECT_NAME=sentry-self-hosted
# Set COMPOSE_PROFILES to "feature-complete" to enable all features
# To enable errors monitoring only, set COMPOSE_PROFILES=errors-only
# See https://develop.sentry.dev/self-hosted/experimental/errors-only/
COMPOSE_PROFILES=feature-complete
SENTRY_EVENT_RETENTION_DAYS=90
# You can either use a port number or an IP:PORT combo for SENTRY_BIND
# See https://docs.docker.com/compose/compose-file/#ports for more
SENTRY_BIND=9000
# Set SENTRY_MAIL_HOST to a valid FQDN (host/domain name) to be able to send emails!
# SENTRY_MAIL_HOST=example.com
SENTRY_IMAGE=ghcr.io/getsentry/sentry:25.9.0
SNUBA_IMAGE=ghcr.io/getsentry/snuba:25.9.0
RELAY_IMAGE=ghcr.io/getsentry/relay:25.9.0
SYMBOLICATOR_IMAGE=ghcr.io/getsentry/symbolicator:25.9.0
TASKBROKER_IMAGE=ghcr.io/getsentry/taskbroker:25.9.0
VROOM_IMAGE=ghcr.io/getsentry/vroom:25.9.0
UPTIME_CHECKER_IMAGE=ghcr.io/getsentry/uptime-checker:25.9.0
HEALTHCHECK_INTERVAL=30s
HEALTHCHECK_TIMEOUT=1m30s
HEALTHCHECK_RETRIES=10
HEALTHCHECK_START_PERIOD=10s
HEALTHCHECK_FILE_INTERVAL=60s
HEALTHCHECK_FILE_TIMEOUT=10s
HEALTHCHECK_FILE_RETRIES=3
HEALTHCHECK_FILE_START_PERIOD=600s
# Set SETUP_JS_SDK_ASSETS to 1 to enable the setup of JS SDK assets
# SETUP_JS_SDK_ASSETS=1
[root@log self-hosted]#

config.yml:

# Uploaded media uses these `filestore` settings. The available
# backends are either `filesystem` or `s3`.

filestore.backend: 'filesystem'
filestore.options:
  location: '/data/files'
dsym.cache-path: '/data/dsym-cache'
releasefile.cache-path: '/data/releasefile-cache'

# filestore.backend: 's3'
# filestore.options:
#   access_key: 'AKIXXXXXX'
#   secret_key: 'XXXXXXX'
#   bucket_name: 's3-bucket-name'

In other words, what can be done to ensure that new "events" arrive and are read correctly, even if old ones are unavailable?