Restructure of update health monitor inventory representation

[karencfv]

Background

When I initially started working on the health monitor for update, we landed on a design for the inventory API where each sled agent had a designated health-monitor object which would contain different health checks. Each health check is done asynchronously every minute as a background job:

$ curl -H "api-version: 16.0.0"  http://[::1]:41364/inventory | jq
<...>
  "health_monitor": {
    "smf_services_in_maintenance": {
      "ok": {
        "services": [
          {
            "fmri": "svc:/site/fake-service2:default",
            "zone": "global"
          },
          {
            "fmri": "svc:/site/fake-service:default",
            "zone": "global"
          }
        ],
        "errors": [],
        "time_of_status": "2026-01-19T06:03:13.713176608Z"
      }
    },
    "unhealthy_zpools": {
      "ok": {
        "zpools": [
          {
            "zpool": "fakepool1",
            "state": "degraded"
          },
          {
            "zpool": "fakepool2",
            "state": "degraded"
          }
        ],
        "errors": [],
        "time_of_status": "2026-01-19T06:03:13.690492960Z"
      }
    }
  }
<More health checks>
}

This approach made sense from a usability standpoint for a few reasons:

• All health check data is contained in one place and with a clear purpose, which makes it easy for consumers to ingest.
• Empty health check lists mean all components checked are healthy based on the parameters we have, so it's very easy to deduct whether there is a problem or not.

Unfortunately, there are a few issues with this approach:

• For some components that will be checked for health, we are already collecting data, like zpools in the reconciler. This is a problem. For example, the health monitor could end up reporting on different zpools than the reconciler. The reconciler reports data on demand, whereas the health monitor polls every minute.
• Not all health checks will be run on a "per sled" basis, like stale sagas. Having a single health monitor object per sled would be confusing and impractical.
• Some health checks report on the state of some component that is reported by itself, like services in maintenance or zpool health, but others will be determined by self imposed thresholds like stale sagas, dataset usage etc. Having all of the health checks lumped in a "health-monitor" would not make these nuances obvious.

Proposal

Instead of having a single health-monitor object per sled, add additional health status/information to every component or create a new entry for components that aren't listed yet. This approach will also allow new components that have no data recorded in inventory yet, to contain more information than just health.

Taking SMF services as an example, with the current approach, we only list services in maintenance. Only listing services in maintenance could prove to be insufficient information to gauge a system's health (e.g. #9855 (comment)), we would have more information to go on by.

In addition to health information/status, each item should have a timestamp of when the sample was taken. This will help with accuracy and avoid relying on assumptions.

It will be the job of the consumer to decide whether a status meets the criteria of a failed health check or not.

I have also assessed the current manual health check script, and have a list of what health status information we'll want to include. It'd be great to have input from others on this:

Health check	Scope	Necessary?
Uptime	per sled	no
Kernel low memory scan	per sled	maybe? Would need to have a threshold to compare the result against. May need to run the command on a loop for a bit as well.
SMF service health	per sled/zone	yes - priority
Core file or kernel core file	per sled	no, There could be leftover core files that don’t mean anything
NVME firmware version	per sled	possibly necessary, but not urgent
expected physical disk count	per sled	yes
zpool health	per sled	yes- priority
unmounted dataset	per sled	yes. Will need to make sure the datasets are ones that matter (e.g. not support bundles)
expected crucible zone count	per rack	yes
dataset usage over 80% of quota or root dataset with <150 GiB avail	per sled	yes
crucible free space when total capacity minus total used < 5TiB	per rack	yes
dimm expected memory (1 or 2 TB)	per rack	yes
missing components for MGS driven updates	per rack	yes
stale sagas - not on the script but we want to add this check	per rack	yes - priority

Open questions

To avoid triggering a bunch of expensive calls when an inventory collection is created, @jgallagher suggested we use the intermittent polling background tasks in the existing health monitor. This could also have the unintended consequence of data being older than the inventory collection itself. Is this acceptable? How would this impact the reconciler?

[karencfv]

Would love to have feedback from @davepacheco @jgallagher @smklein @andrewjstone and anyone else interested in the topic!

[davepacheco]

Thanks @karencfv. I think we're on the same page about this. To play it back:

• Move per-sled zpool health information that's in the sled agent inventory from the health_monitor container to where the zpools are already (elsewhere in the inventory structure).
• If we choose to return potentially old, cached information in the inventory, add additional timestamps so that consumers of this information know how old all of it is. The main case I'm thinking of is if we grab zpool information asynchronously and provide cached values in the inventory, then I could see having a separate timestamp for that so that consumers know those values were collected potentially before the inventory collection even started.
• Report facts in the inventory with the understanding that it may be somewhat complex for different consumers to decide if what they care about is healthy or not.
• The priority is on SMF service health and zpool health.

Instead of having a single health-monitor object per sled, add additional health status/information to every component or create a new entry for components that aren't listed yet. This approach will also allow new components that have no data recorded in inventory yet, to contain more information than just health.

I misread this a few times. I think what you're saying is: assuming there's some new hardware or software component that we want to add health information for, the design pattern is to add a set of records about it to the top level of the sled agent inventory rather than grouping health-only information under a health_monitor container. That seems fine. But I can't tell if you're also saying that if we're going to do this, we should list all instances of the component along their health (e.g., all SMF services, not just those in maintenance). I wouldn't go that far. In the case of SMF services: I wouldn't include all the healthy instances. There are quite a lot of them and most of their information isn't relevant to the control plane unless they're unhealthy.

I gather you're trying to create a general rule that we can follow as we add more of these. I'd recommend against trying to generalize more than we need to right now. I don't think we have enough information about our future needs to know what's going to work. But that's okay because we can also change all of this pretty easily later (merge conflicts aside 😬). In the long-term, health monitoring is the responsibility of the fault management subsystem and that project may have different approaches for collecting this information. For example, your entry for "crucible free space when total capacity minus total used < 5TiB" -- that could also be implemented by reporting usage to Oximeter and having FM monitor metric data from Clickhouse. In that case, effort we spend here trying to incorporate that into inventory would be a waste. I think if we know how we're expressing SMF service health and zpool health in the per-sled inventory structure, we're good for now.

Open questions

To avoid triggering a bunch of expensive calls when an inventory collection is created, @jgallagher suggested we use the intermittent polling background tasks in the existing health monitor. This could also have the unintended consequence of data being older than the invenrtory collection itself. Is this acceptable? How would this impact the reconciler?

Thinking out loud a bit: I'd say the only constraint here is that making multiple concurrent requests for inventory must not spawn multiple processes collecting the same information concurrently. You shouldn't be able to cause sled agent to fork a thousand of these concurrently by making the request a thousand times concurrently.

That aside, there are pros and cons of doing the work inline with the request:

• pro: simple implementation
• pro: the data is not collected when it's not needed
• pro: the data is always as up-to-date as it could be. A not-so-obvious consequence of this: suppose we don't do it inline, but in a background task every 5 minutes, but a consumer knows they just did something that would have changed it and wants to force sled agent to really recollect the data. With the background thing, it can't (unless we also provide a "force" flag or something, which I feel is a strong sign we've made the wrong choice).
• con: without more work, it doesn't meet the above constraint
• con: in the common case, the request will take a lot longer
• con: if we can't collect the information for some reason, we can't provide anything useful to the caller. The background collection solution would. But you could as well argue that if the desired behavior is to have older data when current data is unavailable, that's the consumer's responsibility and they can already do that by looking at the previous collection.

Yet another option here would be to have a mechanism that forks off a command to collect this data, but ensures that (1) no more of these commands is ever outstanding at a time and (2) the consumer always gets results from a collection that started after their request was made. The Nexus background task subsystem looks like this. Something like this:

• request R1 arrives and kicks off zpool list Z1
• requests R2, R3 arrive. They get in line.
• When Z1 finishes, R1 gets the results of Z1. Another collection Z2 gets kicked off.
• When Z2 finishes, R2 and R3 both get the result of Z2.
• No more zpool lists are kicked off until another request comes in.

I may well be overthinking this and am happy to discuss further.

I think there are two principles in tension here:

• possible principle 1: an inventory collection request should always be quick and report the latest information available. This is compelling in a different way: it means we can always quickly and easily check the latest status of the system.
• possible principle 2: inventory collection should be simple; freshness, frequency, caching, etc. should be under the control of the caller. This is the Prometheus approach for metrics and it feels like a better primitive for reasons like the above. I believe that topo enumeration in the OS also works this way.

[karencfv]

The priority is on SMF service health and zpool health.

Eek! Looks like I forgot to add stale sagas to that list (#9412). I only added what was in the script, I'll fix.

Instead of having a single health-monitor object per sled, add additional health status/information to every component or create a new entry for components that aren't listed yet. This approach will also allow new components that have no data recorded in inventory yet, to contain more information than just health.

I misread this a few times. I think what you're saying is: assuming there's some new hardware or software component that we want to add health information for, the design pattern is to add a set of records about it to the top level of the sled agent inventory rather than grouping health-only information under a health_monitor container. That seems fine. But I can't tell if you're also saying that if we're going to do this, we should list all instances of the component along their health (e.g., all SMF services, not just those in maintenance). I wouldn't go that far. In the case of SMF services: I wouldn't include all the healthy instances. There are quite a lot of them and most of their information isn't relevant to the control plane unless they're unhealthy.

Reading this back again I can see that it's just bad English on my part 😄 Yeah, what you understood is what I meant. I agree that listing all SMF services is overkill, what I meant is that we could have more information about each SMF service than just it's health/zone this way. I do want to clear up what unhealthy means in the context of SMF services though. In #9855 (comment) Alan was able to see something was wrong by services that were offline. If we stick to just listing SMF services in maintenance then a scenario like what Alan encountered would have been marked as healthy. Perhaps we list all "oxide" SMF services? That seems like too much. Or perhaps list all services that are not "online"? I'm not sure here tbh.

I do believe that the level of detail or number of components listed should be on a case by case basis.

I gather you're trying to create a general rule that we can follow as we add more of these. I'd recommend against trying to generalize more than we need to right now. I don't think we have enough information about our future needs to know what's going to work. But that's okay because we can also change all of this pretty easily later (merge conflicts aside 😬).

Yeah, fair. My main point was about just adding general information to a higher level object rather than having a dedicated health monitor object, I might have got carried away.

For example, your entry for "crucible free space when total capacity minus total used < 5TiB" -- that could also be implemented by reporting usage to Oximeter and having FM monitor metric data from Clickhouse. In that case, effort we spend here trying to incorporate that into inventory would be a waste. I think if we know how we're expressing SMF service health and zpool health in the per-sled inventory structure, we're good for now.

In this case what I assumed we'd do is only report crucible's free/used space. The consumer would take that information and make a decision (is it less than 5 TiB?) on whether it was enough free space to mark the rack as healthy or not. But yeah, this item is not on the priority list so we can punt on this anyway.

Yet another option here would be to have a mechanism that forks off a command to collect this data, but ensures that (1) no more of these commands is ever outstanding at a time and (2) the consumer always gets results from a collection that started after their request was made.

In this scenario, the correct approach would be to use this mechanism for every inline request triggered by an inventory collection yes? At this point, it kind of feels like a task that is not necessarily part of the health monitoring project but rather a refactoring of inventory collection to make it more resilient?

That said, I do like your approach. I didn't entirely love the background tasks for all of inventory, but I couldn't articulate exactly why. Your pros and cons list made it a lot clearer.

Regarding the two principles in conflict, thinking about principle 1:

possible principle 1: an inventory collection request should always be quick and report the latest information available. This is compelling in a different way: it means we can always quickly and easily check the latest status of the system.

I don't think your proposal clashes with this principle entirely. The latest information available will always be the tiniest bit out of date by the time a consumer receives it. With your proposal we may be adding a little bit of time if a large amount of requests are triggered concurrently, but I don't think it will make much of a difference. In addition, with the timestamps, the consumer will know exactly how "out of date" these values are.

[davepacheco]

👍 to most of that.

I do want to clear up what unhealthy means in the context of SMF services though. In #9855 (comment) Alan was able to see something was wrong by services that were offline. If we stick to just listing SMF services in maintenance then a scenario like what Alan encountered would have been marked as healthy. Perhaps we list all "oxide" SMF services? That seems like too much. Or perhaps list all services that are not "online"? I'm not sure here tbh.

I expect we want to use the same criteria that SMF uses to determine whether the service shows up in svcs -x. I suspect it's something like "in maintenance state or in state offline but enabled". I'd take a look at what svcs -x does (let me know if you want help). Regardless, I don't think we want to limit it to "oxide" services nor include "oxide" services that are healthy.

Yet another option here would be to have a mechanism that forks off a command to collect this data, but ensures that (1) no more of these commands is ever outstanding at a time and (2) the consumer always gets results from a collection that started after their request was made.

In this scenario, the correct approach would be to use this mechanism for every inline request triggered by an inventory collection yes? At this point, it kind of feels like a task that is not necessarily part of the health monitoring project but rather a refactoring of inventory collection to make it more resilient?

I'm not totally sure what you mean here. Yeah, the suggestion here is that in sled agent, when it receives an HTTP request to report its inventory: if the command already outstanding, it waits for it to finish. Regardless of whether it waited, it then forks off the command to fetch the information it needs and reports it. To be clear, this is all within sled agent. (I usually think of "inventory collection" as the Nexus side that's making these requests, so just being clear that that's not what we mean, right? That part is already resilient to sled agents that are failing or slow.)

If you're asking "is this refactor a blocker for the health checking work" -- I guess that depends on whether the changes we're already doing here increase the risk.

Regarding the two principles in conflict, thinking about principle 1:

possible principle 1: an inventory collection request should always be quick and report the latest information available. This is compelling in a different way: it means we can always quickly and easily check the latest status of the system.

I don't think your proposal clashes with this principle entirely. The latest information available will always be the tiniest bit out of date by the time a consumer receives it. With your proposal we may be adding a little bit of time if a large amount of requests are triggered concurrently, but I don't think it will make much of a difference. In addition, with the timestamps, the consumer will know exactly how "out of date" these values are.

There's an important difference between:

• the information I got was out of date when I received it [because something else happened after I made my request]
• the information I got was out of date even when I sent the request [because it was collected previously]

Here's the example I have in my head: a client makes a change to the sled's disk config and then wants to fetch inventory and verify that the change has happened.

This client doesn't care about changes made to the config after its inventory request. But it does very much care that the information it got back actually reflects the change that it had made previously. With "approach 2", where the data is collected inline with the request, it's easy for the client to reason about things, and the reasoning matches intuition: if you collect the inventory after you make the change, then the inventory will reflect the change. With "approach 1", where the data was always collected before your request was even received, it's harder to reason about. You have to know to look at the timestamps, and as I write that, I realize that even that's fraught because distributed time is complicated.

Then there's the question of: what do you do if the inventory you got back doesn't reflect the change? You have to keep polling. How long? As long as whatever interval we've used for the background collection. That means the whole operation the client is doing could take as long as whatever this interval is. That usually sucks. So people usually wind up adding to these APIs switches to say force_refresh=true or invalidate_cache=zpool or whatever. This is maximally flexible, but it also tends to be a source of bugs. Now the request is sometimes fast and sometimes not, sometimes has the latest data and sometimes not, and operations can seem to work when they're working by accident (because they technically require a force_refresh but the intervals happen to work right most of the time).

So why would you go with approach 1 at all? It better deals with data collection operations that are slow or even hang. Although it's not common, I've seen it a lot over the years. (zfs list at least used to take several minutes with thousands of filesystems. svcs -Z has to make RPC calls to configd in all of the zones on the system, and any of those being stuck will cause it to hang. zpool list could take a minute if some disks are not responding due to failures.) The idea behind approach 1 is: these events can always happen, and when they do, we still want to be able to instantly see the latest information. Concretely: suppose the zpool list is taking 5 minutes. Approach 1 says you should still be able to run omdb to get the inventory from a sled, have it respond immediately (not block for 5 minutes), and the response should show that the latest zpool list is much older than expected (and maybe even show that it's been running for 5 minutes).

I do waffle a bit on this but in my previous comment I was coming around to approach 2 just because it's so much simpler to think about and hard to introduce weird races. The runtime risk is real but maybe something that can be mitigated in other ways.

[karencfv]

I expect we want to use the same criteria that SMF uses to determine whether the service shows up in svcs -x. I suspect it's something like "in maintenance state or in state offline but enabled". I'd take a look at what svcs -x does (let me know if you want help). Regardless, I don't think we want to limit it to "oxide" services nor include "oxide" services that are healthy.

Fair. I took a look at what the docs say for svcs -x:

   -x
                     Displays explanations for service states.

                     Without arguments, the -x option explains the states
                     of services which:

                        o      are enabled, but are not running.

                        o      are preventing another enabled service
                               from running.

It wasn't super clear what states that meant, so I set up a few fake services in my local machines and only the ones that were "offline", "maintenance" or "degraded" showed up. It seems reasonable to me to show services in this state. I presume we'll want to keep using svcs -a -o state,fmri,zone instead of svcs -x because of how annoying it is to parse the response from that.

To be clear, this is all within sled agent. (I usually think of "inventory collection" as the Nexus side that's making these requests, so just being clear that that's not what we mean, right? That part is already resilient to sled agents that are failing or slow.)

Ahhh aha! Yup, I had been thinking of this in Nexus the whole time.

If you're asking "is this refactor a blocker for the health checking work" -- I guess that depends on whether the changes we're already doing here increase the risk.

I didn't think of it so much as a blocker, but rather that we should not stop at just taking this approach for the calls related to health reporting. Also, that this work didn't necessarily have to be part of the "health monitor" work.

I do waffle a bit on this but in my previous comment I was coming around to approach 2 just because it's so much simpler to think about and hard to introduce weird races. The runtime risk is real but maybe something that can be mitigated in other ways.

I agree that approach 2 is simpler to reason about, and it's easier to make assumptions about. I have a remaining concern though.

I remember when we talked about this at some point we said that if taking an inventory collection took longer than necessary we could always just get the latest one instead. The latest collection could be up to 10 minutes old if I remember correctly. In 10 minutes, there is a big chance the health status will be completely out of date. I wouldn't want to make decisions based off of an inventory collection that old. I'm guessing we could set a max threshold of how old the data is and if it's older that that report the system as unhealthy? Or try retrieving a new inventory collection a few times before reporting the system as unhealthy?

Something that I like about approach 1 is that you can make decisions about the validity of the data on a per-component level

[jgallagher]

The latest collection could be up to 10 minutes old if I remember correctly. In 10 minutes, there is a big chance the health status will be completely out of date. I wouldn't want to make decisions based off of an inventory collection that old. I'm guessing we could set a max threshold of how old the data is and if it's older that that report the system as unhealthy? Or try retrieving a new inventory collection a few times before reporting the system as unhealthy?

I don't think I'd frame this as a question about timeliness of inventory (at least not yet), because inventory is always out of date by the time you look at it. E.g., I'm not sure "10 minute old inventory" is noticeably worse than "3 minute old inventory" the majority of the time - either way lots of things may have changed.

Instead, I think I'd ask this from the point of view of the consumer of inventory. Who or what is making decisions based on it, and do they have enough information to make informed choices knowing that inventory is inherently stale? E.g. for the blueprint planner, I know we've talked about refusing to plan until the latest inventory is at least as new as the parent blueprint, because any older than that guarantees any changes made by that blueprint aren't yet reflected. But I'm not sure who/what is going to consume this health info. If it's solely for humans, at least at first, a timestamp seems like it could be sufficient - a human can decide whether they care that they're looking at N minute old info and wait for a newer one. If it's for support bundle inclusion, I don't think the time scale we're talking about here matters basically at all, because opening a support case, collecting a bundle, copying it off the rack, etc., is a lot longer time period than any timeouts we'd consider here.

[karencfv]

Instead, I think I'd ask this from the point of view of the consumer of inventory. Who or what is making decisions based on it, and do they have enough information to make informed choices knowing that inventory is inherently stale? E.g. for the blueprint planner, I know we've talked about refusing to plan until the latest inventory is at least as new as the parent blueprint, because any older than that guarantees any changes made by that blueprint aren't yet reflected. But I'm not sure who/what is going to consume this health info.

The idea here would be that the consumer triggers an inventory collection to get the latest information. If it cannot fetch a new inventory for whatever reason (we'd probably have to have some sort of timeout here I guess), it would attempt to use the latest one. We'd probably have to have a threshold of how old we allow the inventory to be. Or perhaps just report that we were not able to retrieve the information within a time limit, and that itself is a problem?

But I'm not sure who/what is going to consume this health info. If it's solely for humans, at least at first, a timestamp seems like it could be sufficient - a human can decide whether they care that they're looking at N minute old info and wait for a newer one. If it's for support bundle inclusion, I don't think the time scale we're talking about here matters basically at all, because opening a support case, collecting a bundle, copying it off the rack, etc., is a lot longer time period than any timeouts we'd consider here.

We'll have 2 consumers:

• A mechanism that evaluates the necessary data in inventory to determine whether the system is "healthy/ready to start an update". This mechanism will be used in two places.
  • In the "update_status" external API that will only report that the system is not in an updateable state aka unhealthy, but not give any details to the end user as none of this information is actionable for them. They will only be told to contact support if they believe the system should be healthy but it's not. I phrase it like this because a customer could potentially call the API during an update and it has the potential of reporting problems, when these are not expected.
  • Within reconfigurator to block an update if the system is not in an "updateable/healthy" state
• As part of inventory in the support bundles. This is where I think timestamps would be very handy. Having an exact time of when a sample was taken will be very useful in the context of forming a timeline of events