[Feature]: ResourceStatusSystem (RSS) in diracx

[fstagni]

User Story

As a diracx developer
I want to port the minimal subset of RSS in diracx
So that several other developments can proceed, as RSS is central to many of them

As a Dirac(X) administrator
I do not need nodes information
As Sites and resources are enough

As a diracx developer
I want to port the policy decisions and enforcement of RSS in diracx
So that I can remove it from legacy DIRAC

As a Dirac(X) administrator
I want to enable RSS within diracx
So that I can disable RSS from legacy DIRAC

As a Dirac(X) administrator
I want to keep the possibility of enabling policies or not
So that I can keep the system as simple as possible

As a developer of a diracx extension
I want to define my own policies
So that I can customize the list of them

As a Dirac(X) administrator
I want to manually insert downtime schedule
So that I can insert also those that do not appear on GOCDB

As a Dirac(X) administrator
I want sensible defaults
So that I do not have to bother with configuration if I do not want to

As a Dirac(X) administrator
I want to see a clear relation between sites and resources

As a Dirac(X) administrator
I want job draining before a downtime, rather than sudden banning

As a Dirac(X) administrator
I want to see pledged resources, maximum resource allowance, resource usage

Feature Description

This feature request should be considered as an "epic" from which several tasks will be derived. The user stories are derived from questionnaire https://docs.google.com/forms/d/e/1FAIpQLSd3r-ysd-pvneMK-aRm8aAtOjXg-qFiZvyLlyTzBKReWt1byQ/viewform?usp=sharing&ouid=117871612314528955780

Definition of Done

No response

Alternatives Considered

No response

Related Issues

No response

Additional Context

No response

[aldbr]

Proposed Plan: RSS Migration to DiracX

Following up on the questionnaire results and internal discussions, here's a proposed breakdown of the RSS migration into phases. This is meant as a starting point for discussion. Please comment with questions or disagreements.

Note: This plan is intentionally high-level. We follow an agile process: phases will be refined as we go, story point estimates will be revisited at the start of each sprint, and priorities may shift based on what we learn. The goal here is to align on direction and identify open questions — not to commit to a fixed roadmap.

Context and constraints

• ~2.5 FTEs allocated to RSS (~22 SP/sprint)
• No diracx-tasks available in the short term — legacy DIRAC agents continue running during transition
• No DB schema changes until migration is fully complete
• Existing ResourceStatusDB (SiteStatus, ResourceStatus, Log, History tables) is shared between DIRAC and diracx during transition

Architecture overview

The migration follows a read-first, write-later strategy. diracx reads from the existing DB while legacy agents keep writing to it. Consumers are switched to the diracx API early to maximize value.

graph TB
    subgraph "Phase 1-2: Read path"
        C1[DIRAC consumers<br/>ResourceStatus singleton<br/>SiteStatus singleton] -->|REST API| DX1[diracx RSS router<br/>read-only]
        DX1 --> DB1[(Existing<br/>ResourceStatusDB)]
        LA1[Legacy DIRAC agents<br/>SiteInspector, ElementInspector<br/>CacheFeeder] -->|direct DB write| DB1
    end

    subgraph "Phase 3-4: Read + Write"
        C2[DIRAC consumers] -->|REST API| DX2[diracx RSS router<br/>read + write]
        A2[Admins / CLI] -->|REST API| DX2
        DX2 --> DB2[(Existing<br/>ResourceStatusDB)]
        LA2[Legacy DIRAC agents<br/>still running] -->|direct DB write| DB2
    end

    subgraph "Phase 5-6: Full migration"
        C3[DIRAC consumers] -->|REST API| DX3[diracx RSS router<br/>full]
        A3[Admins / CLI] -->|REST API| DX3
        DT3[diracx-tasks<br/>policy engine] -->|REST API| DX3
        DX3 --> DB3[(Existing<br/>ResourceStatusDB)]
    end

Loading

Phase dependencies

graph LR
    P1[Phase 1<br/>Read-only API<br/>~27 SP] --> P2[Phase 2<br/>DIRAC integration<br/>~24 SP]
    P1 --> P3[Phase 3<br/>Write API + Admin<br/>~23 SP]
    P3 --> P4[Phase 4<br/>History + Downtimes<br/>~23 SP]
    P3 --> P5[Phase 5<br/>Policy engine<br/>~44 SP]
    P5 --> P6[Phase 6<br/>External integrations<br/>~28 SP]

    DT[diracx-tasks<br/>available] -.->|required| P5

    style P2 fill:#b3d9ff
    style P3 fill:#b3d9ff
    style DT fill:#ffe0b3,stroke-dasharray: 5 5

Loading

Phases 2 and 3 can run in parallel once Phase 1 is done.

Phase 1 — Read-only REST API (~27 SP)

Goal: diracx can serve resource/site status queries from the existing DB.
Milestone: API specification review (before implementation)

Task	SP	Notes
Define REST interface specification (OpenAPI)	3	Gate: team review before implementation proceeds
SQLAlchemy models for existing ResourceStatusDB tables	5	Map existing schema, no changes. Can start in parallel with spec review
Pydantic response models	3	Can start in parallel with spec review
POST /api/rss/{vo}/status/search (bulk read, SearchParams pattern)	5	Consistent with existing diracx bulk search endpoints
POST /api/rss/{vo}/status/summary (aggregation)	3	Supports getUsableSites use case (group by status, filter Active/Degraded)
RSSAccessPolicy (read: any authenticated user)	3	Following diracx access policy pattern
Unit + integration tests	5

Phase 2 — DIRAC legacy integration (~24 SP)

Goal: DIRAC's ResourceStatus and SiteStatus singletons consume from diracx instead of the legacy RSS service.

Task	SP	Notes
Patch ResourceStatus singleton to call diracx REST API	8	Replace ResourceStatusClient (RPC) backend. Cache layer can remain initially
Patch SiteStatus singleton (getUsableSites, getSiteStatuses)	5
Feature flag: switch between legacy/diracx per VO	3	Gradual rollout, fallback to legacy
Integration tests (both paths)	8	Verify identical behavior

After this phase: the legacy ResourceStatus/ResourceStatus and ResourceStatus/Publisher services can be switched off.

Phase 3 — Write API + Admin interface (~23 SP)

Goal: Administrators can set statuses and manage tokens through diracx.

Task	SP	Notes
PATCH /api/rss/{vo}/status (set status with reason)	5	Automatic token assignment (24h ownership)
Token management (ownership model, expiry logic)	5	rs_svc default, user takes temporary ownership
Automatic status logging on change (write to Log tables)	3	Matches current addOrModify behavior
CLI commands (diracx rss list-status, diracx rss set-status)	5
RSSAccessPolicy extension (write: SiteManager equivalent)	2
Tests	3

Phase 4 — History + Downtimes (~23 SP)

Goal: Historical visibility and downtime management.

Task	SP	Notes
History query endpoint (POST /api/rss/{vo}/status/history/search)	5	Useful for debugging
Log summarization logic (compress logs into history)	5	Requires diracx-tasks
Downtime CRUD endpoints	8	Supports manual insertion (user story: "insert downtimes not on GOCDB")
Tests	5

Phase 5 — Policy engine (~44 SP)

Goal: Automated status evaluation in diracx. Requires diracx-tasks.

Task	SP	Notes
Port state machine (status ordering and transitions)	3	Error > Banned > Probing > Degraded > Active > Unknown
Port PDP (policy matching, evaluation, combination logic)	8
Port PEP (enforcement orchestration)	5
Entry-point policy registration (diracx.rss.policies)	5	Extensions add their own policies
Ship DowntimePolicy + FreeDiskSpacePolicy as defaults	5	Sensible defaults out of the box (user story)
Draining support (gradual transition before downtime)	5	User story: "job draining rather than sudden banning"
Actions: LogStatusAction + notification action	5
Tests	8

Until diracx-tasks are ready, legacy agents (SiteInspectorAgent, ElementInspectorAgent, CacheFeederAgent) continue running against the same DB — no gap in automated evaluation.

Phase 6 — External integrations + cleanup (~28 SP)

Goal: Replace remaining legacy agents, full decommission.

Task	SP	Notes
GOCDB downtime sync	8	diracx-task
Rucio RSE status sync (replaces RucioRSSAgent)	5	diracx-task
Status change notifications	5	See open question below
Finer StatusType granularity for storage	5	Beyond Read/Write/Check/Remove if needed
Decommission legacy agents + migration guide	5

Rough timeline

At ~22 SP/sprint with 2.5 FTEs (this is indicative — we'll refine sprint by sprint):

Sprint	Work	Cumulative
1-2	Phase 1 (27 SP)	Read-only API available
3-4	Phase 2 + 3 in parallel (47 SP)	DIRAC consumers migrated, admin write API
5	Phase 4 (23 SP)	History + downtimes
6-7	Phase 5 (44 SP, needs diracx-tasks)	Policy engine in diracx
8	Phase 6 (28 SP)	Legacy fully decommissioned

Total: ~169 SP, ~8 sprints (~4 months) — assuming RSS is the primary focus for the allocated FTEs.

What we drop

• Node element type: already removed from DIRAC
• AccountingCache: already commented out in legacy code
• ResourceManagement as a separate service: no need for a dedicated service. Cache tables used by policies (DowntimeCache, etc.) can live in the main ResourceStatusDB. Others (JobCache, PilotCache, ...) can be replaced by on-demand fetching in policy commands, since the CacheFeederAgent was essentially pre-fetching to avoid hammering external sources — a concern better addressed by rate limiting or short-lived caches.
• Publisher service: replaced by the REST API directly

Architectural decisions for discussion

1. Read path: REST API only vs. config projection (hybrid)

One option discussed was projecting resource statuses into the diracx config service, so consumers read status from config (already loaded in every process) rather than making an API call.

Option A — REST API only (simpler)

• Consumers call GET /api/rss/{vo}/status/search with client-side caching (300s TTL, like today)
• Standard pattern, no new infrastructure needed
• Extra API call per cache miss

Option B — Hybrid: DB + config projection (better read performance)

• DB remains the source of truth (writes, history, tokens)
• On every status change, a projection is pushed into the diracx config:

resource_status:
  lhcb:
    Storage:
      CERN-DST: {Read: Active, Write: Active, Check: Banned, Remove: Banned}
    Compute:
      ce05.cern.ch: Active
    FTS:
      CERN-FTS: Active

• Consumers just read config.resource_status: zero-cost, already distributed, atomic snapshot
• But: config update mechanism for dynamic data doesn't exist yet in diracx

This is a significant architectural choice that affects phases 1-2. Thoughts?

2. Catalog element type

Still referenced in code but no policies operate on it, and the downtime command support is commented out. Do we carry it forward or drop it?

3. Notification mechanism

The legacy system uses email aggregation (EmailAgent). Modern alternatives worth considering:

• Monitoring integration (e.g. Grafana alerting on the status DB) — operators already watch dashboards, status changes become just another alert rule
• Webhook-based — generic, can integrate with any notification system
• Web UI notifications — status changes visible in the diracx web interface

This doesn't need to be decided now (it's Phase 6), but it's worth thinking about early.

4. Policy defaults

Which policies should ship enabled out of the box? Current candidates:

• DowntimePolicy (GOCDB downtimes -> Probing/Banned)
• FreeDiskSpacePolicy (storage space -> Degraded/Banned)

Others currently exist in DIRAC but may not be used in practice. Which ones are actually running in production?

---

For the discussion next week, I'd suggest devs focus on:

1. The architectural choice (Option A vs B for the read path)
2. Whether the phase ordering and SP estimates feel right
3. Any missing use cases from the questionnaire that aren't covered

After the discussion, we can create sub-issues for each phase and refine the estimates for the first sprint.

[fstagni]

Some comments on the above:

• In general, I think that all suggestions (from LLM) about tasks and SP should be rewritten/re-decided. The phase ordering seems fine.

• I would suggest the following change to the first diagram for phase 5-6:

graph TB
    subgraph "Phase 1-2: Read path"
        C1[DIRAC consumers<br/>ResourceStatus singleton<br/>SiteStatus singleton] -->|REST API| DX1[diracx RSS router<br/>read-only]
        DX1 --> DB1[(Existing<br/>ResourceStatusDB)]
        LA1[Legacy DIRAC agents<br/>SiteInspector, ElementInspector<br/>CacheFeeder] -->|direct DB write| DB1
    end

    subgraph "Phase 3-4: Read + Write"
        C2[DIRAC consumers] -->|REST API| DX2[diracx RSS router<br/>read + write]
        A2[Admins / CLI] -->|REST API| DX2
        DX2 --> DB2[(Existing<br/>ResourceStatusDB)]
        LA2[Legacy DIRAC agents<br/>still running] -->|direct DB write| DB2
    end

    subgraph "Phase 5-6: Full migration"
        C3[DIRAC consumers] -->|REST API| DX3[diracx RSS router<br/>full]
        A3[Admins / CLI] -->|REST API| DX3
        DT3[diracx-tasks<br/>policy engine] -->|direct diracx DB write| DB3
        DX3 --> DB3[(Existing<br/>ResourceStatusDB)]
    end

Loading

• For the tasks in phase 1 (I did not carefully check the other ones):

  • you wrote 2 POST methods, I guess you meant GET.
  • the task "RSSAccessPolicy (read: any authenticated user)" is unclear to me what it means.

• I think we should not "drop ResourceManagement as a separate service". And I do not see tasks for it (not urgent yet).

• On the architecture decisions:

1. A vs B: B looks more promising, but means first developing the update mechanism.
2. Drop: the survey we did showed no usage
3. Too early to decide
4. from the survey, the ones used are AlwaysBanned (and AlwaysAllowed), FreeDiskSpace, Downtime