About porting the TSCatalogClient from DIRAC

[arrabito]

I raise the question here if in the context of the migration of the TS to diracx we should keep the TSCatalog interface functionality by porting the TSCatalogClient from DIRAC:
https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/Resources/Catalog/TSCatalogClient.py

This interface allows to automatically add files to transformations ('pushing mode') avoiding the use of the InputDataQuery Agent.

More details about the 2 modes (TSCatalog interface and InputDataQuery Agent) are given here:
https://dirac.diracgrid.org/en/latest/AdministratorGuide/Systems/Transformation/index.html#using-a-catalog-query

In CTAO we have experienced scalability issues when using the TSCatalog interface so that we switched using the InputDataQuery Agent.

My opinion is that we could get rid of it in the transition to diracx, but I would like to gather your opinion about that. Eventually we could implement this kind of functionality directly in diracx (without porting the cuurent code from DIRAC) if it is felt useful.

[fstagni]

IIUC here, the question is: do anyone use TSCatalog? CTAO was the main proponents of its usage, and if you are not using it I doubt others do. But the doubt should be answered (I will raise this).

There is a connected issue in DIRAC: DIRACGrid/DIRAC#7752
And even an open PR: DIRACGrid/DIRAC#8082
The connection here is that the DataFiles table is effectively only used by the TSCatalog interface and can be dropped. If the whole TSCatalog can be dropped (in DIRAC already, no need to wait for the porting to DiracX) then things are simplified.

[arrabito]

The connection here is that the DataFiles table is effectively only used by the TSCatalog interface and can be dropped.

Just one clarification. Reading:
DIRACGrid/DIRAC#7752
and looking at the code, it doesn't seem to me that the DataFile table is effectively used only by the TSCatalog interface.
But anyway, I'm fine with the proposal in DIRACGrid/DIRAC#7752 to drop this table and add an indexed column to the TransformationFiles table.

[andresailer]

We are only using the InputDataQuery agent in iLCDirac

[hmiyake]

Belle2 doesn't use metadata query but directly calls TransformationClient.addFilesToTransformation().