Pilot Migration

[Robin-Van-de-Merghel]

Context

Legacy pilots: Pilots sent by DIRAC with a proxy embedding a DiracX user token

DiracX pilots: Pilots sent by DiracX with a secret to exchange for a DiracX pilot token

DIRAC Migration

We have to migrate some functionalities from DIRAC to DiracX:

Functions not developped

These functions are only used in DIRAC Web, and won't be migrated to DiracX:

• countPilots(countDict, *older, *newer, *timestamp)
• getPilotStatistics(attribute, selectDict)
• getPilotMonitorSelectors(selectDict, sortList, startItem, maxItems)
• [ ] getPilotMonitorWeb(selectDict, sortList, startItem, maxItems) (<-> getPilotSummary?)
• [ ] getPilotSummaryWeb(selectDict, sortList, startItem, maxItems) (<-> getPilotSummary?)
• [ ] getGroupedPilotSummary(selectDict, sortList, startItem, maxItems) (<-> getPilotSummary?)
• [ ] deletePilots

Not distributed functions:

• getCurrentPilotCounters(*attrDict)

Function not adapted:

• killPilot(pilotRefList), see Moved killPilot to diracAdmin itself DIRAC#8203
• getPilotLoggingInfo(pilotReference)
• storePilotOutPut(pilotReference, output, error)
• getRemotePilotOutput(pilotReference, pilotDict)
• getPilotOutput(pilotReference, pilotDict)

Function developped

We kept features from DIRAC, and added a working logging system.

Pilot management

Pilot management functionnalities only reproduces what's being done in DIRAC (pilot creation, deletion, patch, ...). It is mostly done in:

1. DiracX for the logic, endpoints: Add pilot management: create/delete/patch and query #570
2. DIRAC for the legacy adaptor (forwarding requests to DiracX): [9.0] feat: Migrating pilot client toward DiracX DIRAC#8233

Organization

We can decompose all features as such (POST/PATCH/... ... are DiracX endpoints, what's underneath are DIRAC features):

• POST /api/pilots/summary

  • getPilotSummary

• POST /api/pilots/

  • addPilotReferences

• PATCH /api/pilots/fields

  • Needs tests and documentation
  • setPilotBenchmark
  • setAccountingFlag
  • setPilotStatus

• PATCH /api/pilots/jobs

  • setJobForPilot

• GET /api/pilots/search

  • Needs documentation
  • getPilotInfo(pilotReference)
  • getPilots
  • selectPilots

Pilot auth

Pilot auth is pretty straightforward and follows the security model (https://diracx.io/en/latest/admin/reference/security_model/): A pilot will exchange a secret (string) for a pilot token.

A pilot token is like a user DiracX token, but without properties (roles), and with a stamp in it. It helps to differenciate a user from a pilot (whereas with legacy pilots, they have the same token as a user except for roles).

It is mostly done in #421.

Logging system

Logging system is divided in multiple PRs and POCS (proofs of concepts).

DiracX pilots

Current plan is only DiracX pilots can send logs:

1. They arrive with a secret
2. They exchange it for a token
3. They call an endpoint to send logs
4. They can refresh their pilot tokens on a pilot-dedicated endpoint

Done in:

1. Pilot side
   1. Pilot auth: feat: Adding JWT support alongside X509 auth Pilot#248
   2. UPDATE(factorized with legacy pilots) Pilot logs: feat: Add pilot pilot logging (legacy and DiracX) Pilot#260
2. DiracX side
   1. Pilot auth: Adding pilot registrations and authentification (Router) #421
   2. Pilot logs: Add pilot logging #550

Legacy pilots

Update after further discussions, legacy pilots could send logs:

1. They arrive with a proxy with an user token embedded in it (as currently)
2. They extract this token
3. They call an endpoint to send logs
4. They can refresh their user tokens on a user-dedicated endpoint

Done in:

1. Pilot logs: feat: Add pilot pilot logging (legacy and DiracX) Pilot#260
2. DiracX side: [POC] Pilot legacy logging #596

Tested in a DIRAC PR where I create the right environment for legacy pilots to send logs to DiracX: DIRACGrid/DIRAC#8268

New Pilot Features in DiracX

• Pilot Auth
  • Secret
    • Multi-VO
    • Limited / Unlimited in time
    • Count Max
    • Vacuum case
    • Flexible constraints
  • Token
  • Refresh Token
  • Login (POST /api/auth/secret-exchange)
• Pilot resources
  • Access Policies
  • Pilot Management
    • Basic Restrictions
    • [ ] Owner-Based Restrictions (No owner)
  • Job Access in Splitted the job endpoints to separate Pilots and Users #468 : Improving wmsjobpolicy to contain pilots
  • Log Access
  • Read CS (not implemented in DiracX yet)
• DiracX Site director (to deliver pilots with secret)

[fstagni]

Partially related: DIRACGrid/DIRAC#8190 (moving few WebApp-only related RPC calls to separate handler)

[Robin-Van-de-Merghel]

All PRs for auth:

---
config:
  logLevel: 'debug'
  theme: 'base'
---
gitGraph
    commit
    commit
    branch "Pilot management"
    checkout "Pilot management"
    commit
    commit
    commit
    branch "Pilot auth"
    checkout "Pilot auth"
    commit
    commit
    commit
    branch "Pilot logging"
    checkout "Pilot logging"
    commit
    commit
    commit
    checkout "Pilot auth"
    branch "Splitting jobs endpoints"
    checkout "Splitting jobs endpoints"
    commit
    commit
    commit

Loading

[Robin-Van-de-Merghel]

Description refreshed