Update xDTD to use Plover or Shepherd/Retriever to get the latest node/edge info

[chunyuma]

Currently, the xDTD database, there are four tables: prediction_table, path_table, node_info, edge_info.

• The prediction_table contains three columns: drug (CURIE), disease (CURIE), prediction score.
• The path_table contains drug (CURIE), disease (CURIE), and their corresponding paths
• The node_info is basically KG2 node TSV file
• The edge_info is basically KG2 edge TSV file

xDTD mainly uses node_info and edge_info to get attributes. However, they might not be the latest version. Thus, we plan to switch to use Plover or Shepherd/Retriever to get the latest node/edge info.

[saramsey]

Just an FYI, quite a bit of node information is also available in the Babel sqlite database from the "stitch" project, which is indexed and highly efficient to query:
https://github.com/Translator-CATRAX/stitch-proj/releases/tag/babel-20250901-p1
It also has a nice python module local_babel.py with various query methods so you don't need to write SQL or think about the database's schema. The underlying sqlite database is, however, 182 GiB in size, so using it requires a build system with some disk space, as well as time allocated to downloading it.

Then there is the option of calling SRI NodeNorm's API (and NameResolver's API in cases where a name-to-CURIE lookup is needed). It is pretty efficient and supports batch querying. And along those lines, @bazarkua is nearing completion of a rewrite of the ARAX node_synonymizer.py module which will internally call NodeNorm and Name Resolver, but presents a more ARAX-friendly python module interface (abstracting away the web API calls inside the node_synonymizer.py module). Perhaps that could be useful for getting some node info.

For getting KG2 edge info efficiently, the KG2/PloverDB service is a good bet. You might want to take a look at the new KG2.10.3/PloverDB instance, on ploverdev.rtx.ai (the endpoint URL is: http://ploverdev.rtx.ai:9991).

I haven't mentioned Neo4j yet. That could be a conversation, if Neo4j hosting of KG2.10.3c is desired. But Neo4j is really slow, in my experience. Fine for debugging work where you are rolling bespoke Cypher queries on the fly. Not so great for an efficient ingest mechanism, in my opinion.

[chunyuma]

Thanks so much for sharing the above info @saramsey. I appreciate it.