[8.19] (backport #20464) add tbs storage limit and disk-related metrics

[mergify]

Motivation/summary

Expose a new set of metrics to enhance TBS observability. The metric fields in the index data are tested in this PR, while the mappings are tested in the corresponding linked PRs. Changes in the elastic/elasticsearch#138131 PR are tested in conjunction with the changed in this PR.

See #15533 (comment) for the detailed overview.

Depends on PR:

• apm-server: update monitor mappings to include new TBS metrics elasticsearch#140700
• metricbeats: update monitor mappings to include new TBS metrics beats#48432
• elastic-agent: update monitor fields to include new TBS metrics integrations#16560

Checklist

• Update CHANGELOG.asciidoc
• Documentation has been updated

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

Step 1: Ensure Elasticsearch & Kibana is running

Depends on elastic/elasticsearch#138131 with updates to monitoring-beats.json.

1. Build Docker image with changes

> cd elasticsearch

# Build the ES image with the added metric mappings.
> ./gradlew buildAarch64DockerImage --rerun-tasks

2. Edit apm-server/docker-compose.yml and change the ES image.

elasticsearch:
 image: docker.elastic.co/elasticsearch/elasticsearch:9.3.0-custom-SNAPSHOT
 # ... rest of config

3. Spin up the required services.

> cd apm-server
> docker-compose up elasticsearch kibana

Step 2: Create APM Server config

apm-server:
  host: "127.0.0.1:8200"

output.elasticsearch:
  enabled: true
  hosts: ["http://localhost:9200"]
  username: "admin"
  password: "changeme"

monitoring.enabled: true

monitoring.elasticsearch:
  protocol: "http"
  hosts: ["http://localhost:9200"]
  username: "admin"
  password: "changeme"

Step 3: Start APM Server

Run APM Server binary directly:

> cd apm-server
> ./apm-server -e -v -c apm-server.yml

Step 6: Verify Data

Verify data in index .monitoring-beats-7-*

GET .monitoring-beats-7-2026.01.13/_search
{
  "_source": ["beats_stats.metrics.apm-server.sampling.tail"],
  "query": {
    "exists": {
      "field": "beats_stats.metrics.apm-server.sampling.tail"
    }
  },
  "size": 1
}

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 31,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".monitoring-beats-7-2026.01.13",
        "_id": "ApzqgpoBYbFQC8_sQ5JQ",
        "_score": 1,
        "_source": {
          "beats_stats": {
            "metrics": {
              "apm-server": {
                "sampling": {
                  "tail": {
                    "storage": {
                      "value_log_size": 0,
                      "storage_limit": 0,
                      "disk_used": 309071659008,
                      "disk_total": 994662584320,
                      "disk_usage_threshold_pct": 80.2,
                      "lsm_size": 8891
                    }
                  }
                }
              }
            }
          }
        }
      }
    ]
  }
}

Verify mappings for index monitoring-beats-7-*

GET .monitoring-beats-7-2026.01.13/_mapping?filter_path=**.storage

{
  ".monitoring-beats-7-2026.01.13": {
    "mappings": {
      "properties": {
        "beats_stats": {
          "properties": {
            "metrics": {
              "properties": {
                "apm-server": {
                  "properties": {
                    "sampling": {
                      "properties": {
                        "tail": {
                          "properties": {
                            "storage": {
                              "properties": {
                                "disk_total": {
                                  "type": "long"
                                },
                                "disk_usage_threshold_pct": {
                                  "type": "float"
                                },
                                "disk_used": {
                                  "type": "long"
                                },
                                "lsm_size": {
                                  "type": "long"
                                },
                                "storage_limit": {
                                  "type": "long"
                                },
                                "value_log_size": {
                                  "type": "long"
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Related issues

Part of #15533

---

This is an automatic backport of pull request #20464 done by Mergify.

[mergify]

Cherry-pick of 60e89a9 has failed:

On branch mergify/bp/8.19/pr-20464
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 60e89a99.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   internal/beater/monitoringtest/opentelemetry.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   x-pack/apm-server/main_test.go
	both modified:   x-pack/apm-server/sampling/eventstorage/storage_manager.go
	both modified:   x-pack/apm-server/sampling/processor_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 1d596f3

Failed CI Steps

• Package x86_64
• Package x86_64
• Package x86_64
• Package x86_64
• Package aarch64
• Package aarch64
• Package aarch64
• Package aarch64

History

cc @ericywl