Address already in use (os error 98)

[hightwoltt]

When trying to configure Vector, I encounter an error.
2025-01-15T06:32:04.648465Z INFO vector: Vector has started. debug="false" version="0.43.1" arch="x86_64" revision="e30bf1f 2024-12-10 16:14:47.175528383" 2025-01-15T06:32:04.651295Z ERROR vector::internal_events::socket: Error binding socket. error=error creating server listener: Address already in use (os error 98) error_code="socket_bind" error_type="io_failed" stage="receiving" mode=tcp internal_log_rate_limit=true 2025-01-15T06:32:04.651461Z ERROR vector::app: An error occurred that Vector couldn't handle: error creating server listener: Address already in use (os error 98). 2025-01-15T06:32:04.651511Z INFO vector: Vector has stopped.
I can only see this error if I go into the container and type the "Vector" command, when starting the container itself or when trying to debug using docker logs vector -f this error is not displayed.

Vector runs in a Docker container, which is running in a single instance. There are no more containers or Vector instances on the host.

The container was started using:
docker run -d \ --name vector \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /etc/vector/vector.yaml:/etc/vector/vector.yaml:ro \ timberio/vector:0.43.1-debian \ --config /etc/vector/vector.yaml

I tried reinstalling the container.
Can you tell me how to solve this problem? how do I see what makes Vector think it's already running on the host?

[jszwedko]

Hi @hightwoltt ,

Can you share your configuration?

[hightwoltt]

In my infrastructure has 2 Vector:
Both of them are deployed from Dokker, and they both have errors with the address already in use.
Vector agent(On the host where I monitor anything), and intermediate Vector. Vector agents send data to an intermediate vector, which in turn sends it to VMetrics.

Vector agent configuration:

data_dir: /var/lib/vector

api:
   enabled: true

sources:

  host_metrics:
    type: host_metrics
    scrape_interval_secs: 1
    collectors:
      - cpu
      - memory
      - disk
      - network
      - filesystem
      - process

transforms:
  add_host:
    type: remap
    inputs:
      - host_metrics
    source: |
      .host = "10.20.0.13"

sinks:

  intermediary_vector:
    type: vector
    inputs:
      - add_host
    address: "10.15.1.102:9020"

    batch:
      max_bytes: 1048576
      timeout_secs: 3

    healthcheck:
      enabled: true
    acknowledgements: true

Intermediate Vector:

# /etc/vector/vector.yaml

data_dir: /var/lib/vector

api:
   enabled: true

sources:

  intermediary_vector:
    type: vector
    address: "0.0.0.0:9020"

transforms:
  prepare_metrics:
    type: remap
    inputs:
      - intermediary_vector
    source: |
      .labels = {"source": "vector"}
      .value = 1.0
      .timestamp = now()

sinks:

  # VictoriaMetrics VM-Insert
  vm_insert:
    type: prometheus_remote_write
    inputs:
      - prepare_metrics
    endpoint: 'http://1.1.1.103:8480/insert/0/prometheus/api/v1/write'

    auth:
      strategy: "basic"
      user: "vector_user"
      password: "passwd"

    batch:
      max_bytes: 1048576
      timeout_secs: 3

    healthcheck:
      enabled: false

[jszwedko]

Interesting, nothing is immediately jumping out from your config. Are you sure nothing else is listening on the ports you are binding to? You could use lsof to check.

[hightwoltt]

Here is lsof | grep TCP command output:

vector       1               root    9u     IPv4          570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1               root   10u     IPv4          570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1   7 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1   7 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1   8 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1   8 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1   9 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1   9 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1  10 vector-wo root    9u     IPv4    570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1  10 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1  12 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1  12 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1  13 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1  13 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1  14 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1  14 vector-wo root   10u     IPv4    570656580       0t0       TCP localhost:8686 (LISTEN)
vector       1  15 vector-wo root    9u     IPv4     570657634       0t0       TCP 33874bfe0d95:43626->10.15.1.102:9020 (ESTABLISHED)
vector       1  15 vector-wo root   10u     IPv4     570656580       0t0       TCP localhost:8686 (LISTEN)

[hightwoltt]

@jszwedko
Do you have any ideas for solving this problem? I still haven't been able to figure out what the reason is, is it normal that "lsof" shows such a number of processes??

[pront]

I cannot see the headers, but typically the second column is the PID so it looks like there's only one process. We don't have access to the environment but it looks like a problem with your deployment. Especially since you are running multiple docker instances, you might not be configuring the ports correctly. I suggest you trying to make this work locally on the same host to verify that the configurations are correct and then move them to docker, instance by instance.