Storing and using large artifacts

[dkoslicki]

It's been discussed a fair bit internally in the ARAX team, and a bit during Hackathons and the like with the Shepherd folk, but we have not tracked the issue of how to use large artifacts in the Shepherd framework. For example,

1. Our MVP1 approach uses a large database to quickly return pre-computed results
2. Our NGD values are semi-precomputed via stores of (term, PMIDs) tuples
3. Pathfinder would be enormously faster if it had access to a slimmed down, TSV-like representation of tier 0/1 graphs available in memory at runtime
4. etc. etc.

Perhaps this is worth discussing soon, and/or at the upcoming June hackathon.

Tagging @maximusunc , @tokebe , @chunyuma, and @mohsenht as that is a (subset) of stakeholders in this question

[maximusunc]

Sounds good, @dkoslicki! Maybe we can start information gathering. Can you share a list of each artifact you are currently using with some info on each?

Things I think would be useful to know about each artifact:

• What kind of artifact is it? sqlite, redis, text file?
• How big is it?
• How often is it hit/used per query? (i/o)
• Where is it being used? Do multiple parts of the workflow use it?

I think once we know all this, we can start thinking about hosting and scalability and how they can fit into Shepherd.

[dkoslicki]

@mohsenht Can you provide the artifact info for Pathfinder (noting the external SQLite server currently being hit)

@chunyuma Can you provide this info for xDTD (including things needed like the curie_to_PMID stuff)

@venkataseshtej if you are able to, can you provide this info for the current status (before your planned refactor) of xCRG? I know the transcription factor file(s) and saved model (RF or xGBoost?) are required, I just don't know their size.

But in summary @maximusunc , it's likely a gamut:

• What kind of artifact is it? sqlite, redis, text file?
  • sqlite, saved model files, text files
• How big is it?
  • from 10's of megs to multiple gigs
• How often is it hit/used per query? (i/o)
  • some hit at run time for every query (eg. MVP1/xDTD question is a lookup from a big pre-computed table; Pathfinder hits its SQLite DB multiple times during each expansion of its multi-hop expansion).
• Where is it being used? Do multiple parts of the workflow use it?
  • some hit at run time for every query, some only at re-build/re-train time

But I'll let the folks above chime in about the exact details

[mohsenht]

Following up on David’s note, Pathfinder currently relies on two specific remote MySQL databases:

Artifact Name: curie_ngd_v1_0_kg2_10_2
Artifact Type: MySQL (Remote)

Artifact Names: kg2c_v1_0_kg2_10_2
Artifact Type: MySQL (Remote)