What is the philosophy of handling instance level entities merging?

[kpto]

BioCypher is able to handle class level (ontological classes) merging but I don't quite see any note regarding instance level merging.

If there are multiple datasets describing the same entity in different aspects and one would like to include both informations into their KG, how could this be done using BioCypher?

In actual test, BioCypher will simply reject the seen entities that have the same ID, despite that they come from different sources carrying different labels and sets of properties.

I checked the existing products of BioCypher and it seems that they either avoided this problem or handled this strangely in my opinion. For instance, CrossBar includes both ChemBL and DrugBank. It prefers DrugBank for a substance that could be found in both databases and a ChemBL record is used to connect that DrugBank entity to the targeted gene. If you search for the substance using its ChemBL ID, you get a result that is not easy to work with. An actual example would be drug JNJ-38877605 (ChemBL ID: CHEMBL2133806, DrugBank ID: DB13113) targets gene MET via ChemBL assay CHEMBL3801041. Searching for CHEMBL2133806 will get you a graph but it cannot tell you which node is that substance.

This doesn't seem to be a problem for CrossBar as it seems to me that it is more like an index that focuses on relationships then an actual data source. But for Open Targets KG, I suppose it would also functions as a data source and I'd like to make the graph extensible. One may want to combine my KG with other data that extends what is not covered by Open Targets but it seems that there is no established philosophy on this matter?

[slobentanzer]

Hi @kpto,
this is a good question and your estimation is largely correct. The merging in "pure biocypher" has been so far deferred to processing at the adapter level (using a secondary adapter to merge primary adapters, if complexity demands it). It is a very fragile problem because different project requirements can result in different merging strategies, and we didn't have priority so far to tackle this at a general level.

However, this is an important problem and on the radar, so if you think it should take priority, I am happy to dedicate some work towards it. It would certainly be good to enter an exchange with @jdreo and the other Pasteur engineers, as the fusion algorithm they developed in the ontoweaver project tackles this exact problem as well.

Establishing a philosophy for this is very challenging; maybe we should start more pragmatically with an acute issue to solve.

Side note: the legacy preferred_id field in the schema configuration was created to indicate this; I want to rename it to namespace (biocypher/biocypher#389), as that seems more aligned with the actual semantics of the field. Essentially, this conveys the information of which particular ID namespace is the one that should be harmonised towards in case of conflicts. However, what we realised fairly quickly is that most annotations do not even have the luxury of a very comprehensive namespace (the few exceptions being very established entity repositories like UniProt, but even those are not "all-encompassing"). Most interactions do not have a namespace at all; in most real-world cases, we still need to switch between the most common namespaces frequently (taking UniProt as an example, we still have tons of resources with ambiguous mappings from genes, i.e., HGNC, and competing protein identifiers such as SwissProt). Molecules are another good example of well-annotated but still very variable identifier systems depending on background, exact domain, use case, etc.

[slobentanzer]

Here is the pipeline milestone: https://github.com/biocypher/biocypher/milestone/1 (it is the first one I created, actually)

Which also reminds me: I think it would be more effective in aligning the development if you took a critical look at the project management and documentation/milestones that already exist. That way, you would not have to discover all known problems on your own again. ;)

[kpto]

Thank you very much for your detailed explanation. It is very important for me to know that there is no established philosophy so that I don't violate any. Once I started thinking about the design of the knowledge graph, I just naturally thought of the extensibility of it and thought that BioCypher could have partially solved this already but couldn't find anything.

I agree that that instance level merging is complicated and not a feature that we urgently need but here is my attempt to dissect the problem nonetheless. One way to merge nodes I came up is by namespacing properties. Here a namespace corresponds to a data source so there will be namespaces like ot(open targets), chembl, drugbank and etc. I believe that an entity in a KG should exists as its own rather than as something external in a different form. The external information attaches to it. A knowledge graph has its own ID system and any external ID system are just properties of that entities. For example, I would like to have a node like the below:

id: {random_string}
sources: ["ot", "chembl", "drugbank"]
external_ids: ["ot:CHEMBL112", "chembl:CHEMBL112", "drugbank:DB00316"]
ot:drug_name: ACETAMINOPHEN
ot:prop_b: ...
chembl:name: ACETAMINOPHEN
chembl:prop_b: ...
drugbank:generic_name: Acetaminophen
ot:version: ...
chembl:version: ...
drugbank:version: ...
ot:license: ...
chembl:license: ...
drugbank:license: ...

With this design the KG would be able to accept any more data or reverse a merge if the merge was problematic. An adapter would be responsible in finding the semantic meaning of entities presented in the targeted data that is meaningful for the merging. In other words, an adapter that processes drug bank data should map the drug DB00316 from the drug bank to a node that contains an external ID CHEMBL112. The idea is, a node in a KG itself has no meaning, the adapter brings external meanings to it and it's its responsibility to ensure that the action is meaningful and correct. There is also one bonus advantage: it is easy to detect integration error, for instance, multiple nodes containing the same external ID.

But the obvious problem is, it is unacceptably inefficient to create an edge by nodes' properties (semantic meaning) instead of two IDs.

So I think the bottom line is, due to the technological limitation, referencing IDs is the only way to create an edge efficiently and therefore an ID must directly or indirectly encodes a semantic meaning. One way to do it is to nominate an existing resource for that but unfortunately that there is no good resource that can do this job well. Just putting my thoughts here.

Also a side note, at first I thought the ID normalisation in the existing Open Targets adapter is to make the KG ready for merging other resources. As BioCypher does not support that and my first KG will be scoped within OT data only, I am gonna remove the normalisation as the normalisation lengthens the node and edge definitions quite a lot. Despite not consistent in formatting, the IDs used among the Open Targets datasets match each other and they themselves can form a graph without extra processing. I have obtained the information I need so I will mark this answered.

[kpto]

Again just putting my thoughts here. The idea above is perhaps not that infeasible, all it need is a lookup table mapping external IDs to node unique IDs. For instance during creating a KG, an adapter can keep track of all external IDs of nodes for the edge processing stage and for the online mode, an adapter could ask BioCypher to return a table of existing nodes' external IDs for building that lookup table.

Since a KG is supposed to be a distilled knowledge, I think the size of a lookup table is manageable and getting values is a simple query so it should be quite quick. The problem is that it complicates the framework design and also looking at the aforementioned milestone, it seems that BioCypher is following a direction that has strong dependency of any external ID system so this idea may not be appropriate.

[slobentanzer]

@kpto thanks, good thoughts. However, and I answer this way because this seems a quite philosophical discussion, the original reason for the framework was a pragmatic approach. Abstracting away layers of complexity is only useful as long as the audience is still able to adapt to the workflow, which could be tricky if we don't solve this mapping "magically"; that is, our framework just does this without the user's knowledge, and it is mainly correct. I feel this would be a stretch given the current state of the art, which is extremely manual and domain expertise-driven.

BioCypher pipelines were originally thought to be pragmatic reflections of single purposes, not holistic all-in-one solutions (there are plenty of those already, all of which do not fulfil the all-in-one promise at all). What you describe is in principle the Resource Description Framework, which has been build by many consortia over many years, and which is very complex to onboard into or do practical things with (IMHO of course). Their concept of URIs and crosslinks between ontologies is quite aligned with what you wrote. In contrast, with this project, we try to be a counterbalance to those very elaborate and "technically correct" systems, which implies a certain amount of uncertainty, or heuristics.

For our open targets graph, and also in relation to the AI-readiness (CC @fcarli), this means that the user should not have to know about the technical intricacies behind, for instance, the identifier overlap and harmonisation between drugbank, chembl, and clinicaltrials resources. They want to answer a biologically driven question, and they don't have education in ontologies or IT technology. It is slightly different for a "developer" user that wants to build a custom KG, as they can be expected to have some technical knowledge (and this is where my suggestion of a console report above comes in). What we do not want in any case is to replicate closely a system which already exists (which is the ontology, RDF, OBO, etc space). That would be quite redundant. It is useful to look into what they do in order to understand the barriers they have in those workflows, because those are the ones that we can try to avoid with our system (sacrificing something else in the process, such as "generality" and "100% technically correct crosslinks to all other ontologies").

Happy to discuss further.

[slobentanzer]

BTW, before you go into implementation of instance level merging, you definitely should talk to the Pasteur folks, as this is a primary feature of their ontoweaver approach (they call it fusion). They are on Zulip, and particularly @jdreo is a great fan of these discussions. ;)