reduce PloverDB memory footprint

[hodgesf]

This issue will be dedicated to improving KG2 in PloverDB including evidence for specific edge/node attributes being a major contributor to memory consumption, cost estimates for remediation (or, in the case where we choose not to fix, differential cost estimates for hosting).

[saramsey]

In KG2.10.3c, for edge properties, the largest culprits are publications_info and publications, as shown here, (each of these numbers is per 15M edges):

{'agent_type': '1593.05 MB',
 'id': '1959.35 MB',
 'kg2_ids': '3827.29 MB',
 'knowledge_level': '1670.95 MB',
 'object': '1476.55 MB',
 'object_aspect_qualifier': '98.92 MB',
 'object_direction_qualifier': '102.49 MB',
 'predicate': '1630.14 MB',
 'primary_knowledge_source': '1811.0 MB',
 'publications': '2390.23 MB',
 'publications_info': '18859.86 MB',
 'qualified_predicate': '96.65 MB',
 'subject': '1499.98 MB'}

For node properties, we have (in total storage size per property, over all nodes):

{'all_categories': '1199.52 MB',
 'all_names': '1528.88 MB',
 'category': '1145.86 MB',
 'description': '526.49 MB',
 'in_taxon': '20.21 MB',
 'iri': '761.29 MB',
 'name': '833.55 MB',
 'publications': '482.09 MB',
 'synonym': '1653.38 MB'}

And after removing publications_info and publications for edges, we have (in total size, over all edges):

{'agent_type': '3969.1 MB',
 'id': '4884.94 MB',
 'kg2_ids': '9608.78 MB',
 'knowledge_level': '4072.45 MB',
 'object': '3658.66 MB',
 'object_aspect_qualifier': '302.32 MB',
 'object_direction_qualifier': '307.5 MB',
 'predicate': '4057.2 MB',
 'primary_knowledge_source': '4505.22 MB',
 'qualified_predicate': '290.07 MB',
 'subject': '3713.76 MB'}

[saramsey]

See the branch issue-66-steve-python3.12 which is a version of PloverDB that can build the KG2.10.3c index on my MBP, so long as these two lines are in the config.json file:

    "ignore_edge_properties": ["domain_range_exclusion", "publications_info", "publications"],
    "ignore_node_properties": ["publications", "description"],

WARNING: the issue-66-steve-python3.12 branch has had extensive code refactoring and is still experimental. We should not merge that branch to main, before we can complete extensive testing and confirm that it works for querying/hosting KG2.10.3c, multiomics KPs, Docker deployment, etc.

[saramsey]

At this point, unless coding work has already commenced within the branch, may be best to wait until PR 85 is merged and then, at that time, a new branch can be created for the issue-75 work, off of main.

[saramsey]

Let's pause this until we know what the memory requirements of hosting the Tier 0 DogPark graph are.