Alternatives to lua for my use case

[giacom0c]

Question

Hello, I opened a discussion some time ago about my vector config (see #24635). This week we finally went to production and we found out some problems not seen in our test environment.

Namely, it seems we had some issues for the exec source, which is a bash script that just dumps metadata from K8s API server and fuels our enrichment_table. We run the script manually inside vector pods and it run fast with no error (returned 0), so probably that wasn't the root cause.
After more troubleshooting we found out we had a lot of jq stuck processes. Since it was used inside the lua script, we tried removing the entire lua from the transforms section of vector config. Turns out, after that everything worked fine and timeout errors disappeared.

So my reasoning is, our current lua script is not optimized enough with our current load. You can see it in the section below. Vector's lua does not support HTTP external libraries, right? So the only way I have to fetch the application name for what is not present in the enrichment_table is spawning a curl process.

Do you think I have other alternatives if I want to try to remediate a missing app name when the record is not present in the enrichment table? In general this seems to happen to less of 1% of our logs, so it's relatively rare, but on absolute numbers it happens thousands of times.

Thank you in advance.

Note the vector config below only includes what I think is in scope for this question.

Vector Config

enrichment_tables:
  k8s_metadata:
	type: "memory"
	ttl_seconds: 1000
	# The table reads events directly from this transform
	inputs: ["app_enricher"]

sources:
  vector_metrics:
	type: internal_metrics
  kube:
	type: "exec"
	command:
	  - "/etc/vector/scripts/get_k8s_label.sh"
	mode: "scheduled"
	scheduled:
	  exec_interval_secs: 30
	decoding:
	  codec: "json"

  logs:
	type: "file"
	include: ["/var/log/containers/*.log"]
	read_from: end
	ignore_older_secs: 20
	offset_key: offset

transforms:
  # Prepare data for the enrichment table
  app_enricher:
	type: "remap"
	inputs: ["kube"]
	source: |
	  # Reshape the event so the KEY is the container_id and the VALUE is the app_name.
	  # The memory table interprets { "key": "value" } as: lookup("key") -> "value"

	  # We want to store: container_id -> { app_name: "..." }
	  # So we create a dynamic key using the container_id

	  id = .container_id
	  app = .app_name

	  # Clear the event and set the new structure
	  . = {}

	  # Set the dynamic key.
	  #set!(., [id], app)
	  . = set!(., [id], { "app_name": app })
	  
  router:
	type: route
	inputs:
	  - set_app
	reroute_unmatched: false
	route:
	  needs_enrichment:
		source: "!exists(.app_name)"
		type: vrl
	  ok:
		source: "exists(.app_name)"
		type: vrl

  lua_enricher:
	type: lua
	inputs:
	  - router.needs_enrichment
	version: "2"
	hooks:
	  process: |-
		function (event, emit)
		  -- Hardcoded target (as in your curl example)
		  local namespace = event.log.namespace
		  local pod       = event.log.pod_name

		  local cmd = table.concat({
			"curl -s",
			"https://endpoint.example.com",
			"| jq -c '.metadata.labels.\"app.kubernetes.io/name\"'",
			"2>&1",
		  }, " ")

		  local handle = io.popen(cmd)
		  if not handle then
			event.log = event.log or {}
			event.log.k8s_label_error = "io.popen failed"
			emit(event)
			return
		  end

		  local out = handle:read("*a") or ""
		  local ok, _, exit_code = handle:close()

		  -- Trim output
		  out = out:gsub("^%s*(.-)%s*$", "%1")

		  event.log = event.log or {}

		  if ok and out ~= "" and out ~= "null" then
			-- out is a JSON string like: "shared-shake"
			-- store it as-is or strip quotes:
			local unquoted = out:gsub('^"(.*)"$', "%1")
			event.log.app_name = unquoted
			event.log.k8s_label_source = "curl_apiserver"
			event.log.enrichment_source = "table"
		  else
			event.log.k8s_label_error = out
			event.log.k8s_label_exit_code = exit_code
		  end

		  emit(event)
		end

Vector Logs

2026-05-07T06:53:45.169858Z ERROR source{component_kind="source" component_id=kube component_type=exec}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Source send cancelled."
2026-05-07T06:53:45.169949Z ERROR source{component_kind="source" component_id=kube component_type=exec}: vector::internal_events::exec: Timeout during exec. command=/etc/vector/scripts/get_k8s_label.sh elapsed_seconds=30 error=deadline has elapsed error_type="timed_out" stage="receiving"
2026-05-07T06:53:48.642950Z ERROR vector::internal_events::exec: Receive channel closed, unable to send. error_type="command_failed" stage="receiving"
2026-05-07T06:53:48.642972Z ERROR vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Receive channel closed, unable to send."

[agrovr]

You should not need Lua, curl, jq, the scheduled exec source, or the memory table for this label. The value fetched by the fallback is exactly a Pod label, and Vector's kubernetes_logs source already watches Pods through the Kubernetes API and adds their labels to each log event under .kubernetes.pod_labels.

Replace the generic file source with kubernetes_logs and read the label in VRL:

sources:
  logs:
    type: kubernetes_logs
    self_node_name: ${VECTOR_SELF_NODE_NAME}
    # Keep metadata longer for logs drained after a Pod is deleted.
    delay_deletion_ms: 300000

transforms:
  set_app:
    type: remap
    inputs:
      - logs
    source: |
      if exists(.kubernetes.pod_labels."app.kubernetes.io/name") {
        .app_name = .kubernetes.pod_labels."app.kubernetes.io/name"
        .enrichment_source = "kubernetes_logs"
      } else {
        .k8s_label_error = "app.kubernetes.io/name is absent from Pod labels"
      }

The source needs list/watch access to Pods (and, for the standard enrichment, Namespaces and Nodes). Vector's Kubernetes Helm/agent manifests normally provide the required host mounts, downward-API node name, and RBAC.

This removes a synchronous child process from the per-event hot path. Thousands of io.popen("curl ... | jq ...") calls can accumulate processes and backpressure the topology; an external API lookup should not run once per log event. For genuinely unlabeled Pods, I would route the small missing-label stream to a fallback/dead-letter sink and fix the workload labels rather than querying the API synchronously.

If the missing events mainly occur after Pods terminate, tune delay_deletion_ms: Vector keeps deleted-Pod metadata in its cache for this reason (the documented default is 60 seconds). If they occur on live Pods, first verify that those Pods actually have app.kubernetes.io/name and that the Vector service account can watch Pods.

References:

• kubernetes_logs enrichment, RBAC, and metadata cache: https://vector.dev/docs/reference/configuration/sources/kubernetes_logs/
• VRL quoted paths: https://vector.dev/docs/reference/vrl/expressions/#quoted-path

This assumes Vector runs as a node agent/DaemonSet and the endpoint is only returning that Pod label, as the posted curl | jq shows. If it applies additional business mapping, that mapping would still need a separate asynchronous enrichment path.