Run build_ngd_database.py using new node_synonymizer.py

[hodgesf]

This issue will be to track the action item assigned to me of building NGD database using the new node_synonymizer.

I will initially attempt doing this by following the instructions found here.

[hodgesf]

These are the steps I've taken so far and the planned next steps based on the results:

1. SSH'd into EC2 instance (ngdbuild2.rtx.ai)
2. Ran sudo apt update && sudo apt install python3.12 python3.12-venv python3.12-dev
3. Created a virtual environment: python3.12 -m venv venv && source venv/bin/activate
4. Installed dependencies: pip install -r dev-requirements.txt
5. Ran the partial build (a conceptname_to_pmids.db already existed from a prior build): python build_ngd_database.py

The build completed but produced a database with 0 rows. Two independent failures occurred:

Issue 1: Name Resolver API timeout → 0% concept name recognition

The script sent 7,941,174 concept names to NodeSynonymizer.get_canonical_curies(names=...). The new API-based synonymizer calls _call_name_resolver_api(), which sends all names in a single POST request to https://name-lookup.ci.transltr.io/bulk-lookup with a 30-second timeout. This predictably timed out:

Warning: Name Resolver API call failed: HTTPSConnectionPool(host='name-lookup.ci.transltr.io', port=443): Read timed out. (read timeout=30)

On failure, the method silently sets all results to None and returns, so the script continued with 0% recognition.

Root cause: _call_normalizer_api() already batches requests in chunks of 1,000, but _call_name_resolver_api() has no batching & it sends the entire list in one request. This works fine for small lookups but cannot handle the millions of names that build_ngd_database.py sends.

Fix needed: Add batching to _call_name_resolver_api() in node_synonymizer.py, similar to how _call_normalizer_api() already does it. A reasonable batch size (~1,000 names per request) with a longer timeout would make this workable, though it will still take significant time (~8,000 API calls for ~8M names).

Additionally, the new name-matching logic rejects results where the returned label doesn't case-insensitively match the input name. This stricter matching may result in a lower recognition rate compared to the old SQLite-based synonymizer's fuzzy/simplified-name matching, even after the batching fix.

Issue 2: KG2pre Neo4j unreachable

Both edge and node queries failed with a connection timeout to 44.229.232.81:7687 (the KG2pre Neo4j instance configured via RTXConfiguration):

neobolt.exceptions.ServiceUnavailable: Timed out trying to establish connection to ('44.229.232.81', 7687)

Possible causes:

• The EC2 instance's security group doesn't allow outbound traffic to port 7687 on that host
• The KG2pre Neo4j instance's security group doesn't allow inbound from this EC2

Next steps

1. Code fix: Add batching to _call_name_resolver_api() in node_synonymizer.py (batch size ~1,000, increase timeout to 60–120s per batch)
2. Networking: Verify the KG2pre Neo4j instance at 44.229.232.81:7687 is running and accessible from this EC2 instance
3. Re-run the partial build after both fixes are in place

Full output log

((venv)) ubuntu@ip-172-31-14-139:~/RTX/code/ARAX/ARAXQuery/Overlay/ngd$ python build_ngd_database.py
config_secrets.json                                                                   100%  572    10.0KB/s   00:00    
2026-04-02 03:03:13,759 INFO: Importing linkml:types as /home/ubuntu/venv/lib/python3.12/site-packages/linkml_runtime/linkml_model/model/schema/types from source https://raw.githubusercontent.com/biolink/biolink-model/v4.2.2/biolink-model.yaml; base_dir=None
2026-04-02 03:03:13,830 INFO: Starting to build curie_to_pmids.sqlite..
2026-04-02 03:03:13,830 INFO:   Loading pickle DB containing pubmed scrapings (conceptname_to_pmids.db)..
2026-04-02 03:05:13,902 INFO:   Sending NodeSynonymizer.get_canonical_curies() a list of 7941174 concept names..
Warning: Name Resolver API call failed: HTTPSConnectionPool(host='name-lookup.ci.transltr.io', port=443): Read timed out. (read timeout=30)
2026-04-02 03:06:04,992 INFO:   Got results back from NodeSynonymizer. (Returned dict contains 7941174 keys.)
2026-04-02 03:06:07,311 INFO:   NodeSynonymizer recognized 0% of concept names scraped from pubmed.
2026-04-02 03:06:17,709 INFO:   Unrecognized concept names were written to unrecognized_pubmed_concept_names.txt.
2026-04-02 03:06:17,709 INFO:   Mapping canonical curies to PMIDs..
2026-04-02 03:06:17,709 INFO:   Mapped 0 canonical curies to PMIDs based on pubmed scrapings.
2026-04-02 03:06:17,709 INFO:   Getting PMIDs from edges in KG2 neo4j..
2026-04-02 03:06:28,223 ERROR: Encountered an error interacting with KG2 neo4j. Traceback (most recent call last):
  File "/home/ubuntu/venv/lib/python3.12/site-packages/neobolt/direct.py", line 831, in _connect
    s.connect(resolved_address)
TimeoutError: timed out
...
neobolt.exceptions.ServiceUnavailable: Timed out trying to establish connection to ('44.229.232.81', 7687)

2026-04-02 03:06:28,223 INFO:   Processing results..
2026-04-02 03:06:28,225 INFO:   Sending a batch of 0 curies to NodeSynonymizer.get_canonical_curies()
2026-04-02 03:06:28,225 INFO:   Got results back from synonymizer
2026-04-02 03:06:28,225 INFO:   Getting PMIDs from nodes in KG2 neo4j..
2026-04-02 03:06:33,234 ERROR: Encountered an error interacting with KG2 neo4j. Traceback (most recent call last):
  File "/home/ubuntu/venv/lib/python3.12/site-packages/neobolt/direct.py", line 831, in _connect
    s.connect(resolved_address)
TimeoutError: timed out
...
neobolt.exceptions.ServiceUnavailable: Timed out trying to establish connection to ('44.229.232.81', 7687)

2026-04-02 03:06:33,235 INFO:   Processing results..
2026-04-02 03:06:33,235 INFO:   Sending a batch of 0 curies to NodeSynonymizer.get_canonical_curies()
2026-04-02 03:06:33,235 INFO:   Got results back from synonymizer
2026-04-02 03:06:33,236 INFO:   In the end, found PMID lists for 0 (canonical) curies
2026-04-02 03:06:33,236 INFO:   Loading data into sqlite database..
2026-04-02 03:06:33,277 INFO:   Gathering row data..
2026-04-02 03:06:33,277 INFO:   Inserting row data into database..
2026-04-02 03:06:33,277 INFO:   Done saving data in sqlite; database contains 0 rows.
2026-04-02 03:06:33,277 INFO: Done! Building curie_to_pmids.sqlite took 3 minutes.

[hodgesf]

cc @bazarkua & @saramsey:

Based on the results above, it looks like build_ngd_database.py is not currently compatible with the API-based node_synonymizer for large-scale usage (due to lack of batching). Can you confirm whether batching is expected to be handled within build_ngd_database.py, or within the synonymizer itself?

Unless I hear otherwise, I’ll proceed assuming batching should be handled in build_ngd_database.py and continue with the fixes outlined above.

[saramsey]

Good analysis, @hodgesf. I have a couple of questions about this. You are only running Step 2, right?

[hodgesf]

Updates based on slack conversation from Dr. Koslicki's and Dr. Ramsey's comments:

• Build NGD database using Tier-0 DogPark graph (instead of kg2.10.2)
• need to update the PubMed ingest (last ingest available on the instance is 2024)
• update build_ngd_database.py to send batched requests to node_synonymizer.py

[saramsey]

Yes, and specifically, we are aiming to read the KGX files of the Tier-0 DogPark graph (i.e., we will not be using Neo4j to host it, at least, not in the context of work on this issue).

[hodgesf]

RE: Update PubMed articles.

The current PubMed articles located on ngdbuild2.rtx.ai are from 2024. In order to update them, I need to get the baseline/ data from PubMed's FTP servers. There is not currently a way to update the 2024 files to be up to date in 2026. The updatefiles/ directory is for daily incremental changes, so if we refreshed the NGD database on a weekly or even monthly basis, we could apply those incrementally. Because we don't do that, it doesn't make sense to modify the code right now. Instead, we should just run a full build.

[hodgesf]

This may be wrong, but I am operating under the assumption that the tier-0 graph nodes and edges files are already canonicalized.

[hodgesf]

Changes I've made to build_ngd_database.py

Switched from KG2 neo4j to Tier-0 Dogpark JSONL files

Replaced _add_pmids_from_kg2_edges() and _add_pmids_from_kg2_nodes() with _add_pmids_from_tier0_edges() and _add_pmids_from_tier0_nodes(). These read edges.jsonl and nodes.jsonl line-by-line instead of querying KG2 neo4j. Edge publications are added to both subject and object curies. No canonicalization step is needed since Tier-0 curies are expected to be pre-canonicalized.

Batched and parallelized NodeSynonymizer calls

The previous code sent all ~8M PubMed concept names to get_canonical_curies() in a single call, which timed out against the new API-based NodeSynonymizer. Now splits into batches of 5,000 and resolves them in parallel using a ThreadPoolExecutor with 4 workers. Failed batches are logged and skipped without killing the entire run.

Switched to streaming XML parsing

Replaced etree.parse() (loads entire PubMed XML file into memory) with etree.iterparse() that streams one PubmedArticle at a time. Each article element is cleared after processing to prevent memory buildup. This also eliminated the need for the _destroy_etree() memory leak workaround.

Fixed unrecognized concept tracking

Concept names from failed synonymizer batches are now included in the unrecognized concepts set, so the coverage metrics and unrecognized_pubmed_concept_names.txt output are accurate.

Removed dead code

• _run_cypher_query() (neo4j)
• _get_canonicalized_curies_dict() (no longer needed without neo4j)
• _destroy_etree() (no longer needed with iterparse)
• Imports: neo4j, traceback, gzip, RTXConfiguration

Other cleanup

• Fixed .index() inside loop (O(n) per iteration) with enumerate()
• Simplified _divide_list_into_chunks() to a one-liner list comprehension
• Added docstrings to all methods (and improved current docstrings)
• PEP8 compliance (line lengths, blank lines, indentation)

[hodgesf]

The build script now reads nodes.jsonl and edges.jsonl from the Tier-0 Dogpark graph instead of querying KG2 neo4j.

Nodes have no publications field, so all graph-sourced PMID data comes from edges. Example node:

{
  "id": "CHEBI:100",
  "category": ["biolink:SmallMolecule", "biolink:MolecularEntity", ...],
  "name": "(-)-medicarpin",
  "equivalent_identifiers": ["CAS:32383-76-9", "CHEBI:100", "CHEMBL.COMPOUND:CHEMBL238845", ...],
  "information_content": 95.4,
  "description": "The (-)-enantiomer of medicarpin."
}

Edges have a publications field containing a list of PMIDs. Example edge:

{
  "id": "urn:uuid:e1797dc5-2efc-44f0-a518-2d6a64eeac8e",
  "subject": "NCBIGene:18993",
  "predicate": "biolink:expressed_in",
  "object": "UBERON:0001016",
  "publications": ["PMID:14681479"],
  "knowledge_level": "knowledge_assertion",
  "agent_type": "manual_agent",
  ...
}

When an edge has publications, the script adds those PMIDs to both the subject and object curies. So in the example above, both NCBIGene:18993 and UBERON:0001016 would get PMID:14681479 added to their PMID lists.

Since nodes have no publication data in this graph, _add_pmids_from_tier0_nodes() will process 0 nodes. The graph-sourced PMID data comes entirely from edges. The majority of the PMID data comes from the PubMed XML scrape (MeSH headings, keywords, chemical names mapped to curies vianode_synonimizer.py).

[bazarkua]

This may be wrong, but I am operating under the assumption that the tier-0 graph nodes and edges files are already canonicalized.

Yes that's correct translator_kg (tier 0) is already canonicalized.

[hodgesf]

Wanted to flag what I'm seeing: the Name Resolver doesn't seem to hold up under sustained batch load. Sending nodes one at a time isn't feasible (NGD sends ~8.6M concept names through the synonymizer) so I'm benchmarking different batch_size + worker combinations to find what the API can actually handle.

What I've found so far running against the provided endpoint:

• At batch=100: ~17% of names come back as null (silent API failures)
• At batch=1000: ~50% null returns, with frequent 504 Gateway Timeouts
• Failures are silent: the synonymizer logs nothing because warnings only fire when len(names) <= 10.

I've ruled out client-side issues: same API call as your test suite, used unique name slices to avoid cache effects, tried with/without retries and exponential backoff. The 504s come from the Name Resolver gateway giving up on its upstream. I'll share final config recommendations once the worker sweep finishes.

Open question: is there a known per-batch-size limit on the Name Resolver, or guidance on max sustained QPS?

[bazarkua]

Thank you for your testing @hodgesf, I think I found the root cause. It's actually on the Python client side, did a curl vs requests vs httpx comparison to see it.
urllib3 corrupts its connection pool state after ~7 sustained POSTs to /bulk-lookup on the CI endpoint. Fixes commited to the issue-2585 branch

I couldn't find the specific batch size limit, but through testing I found batch size of 50 names to be the perfect spot (I added it as default batch size for /bulk-lookup with my recent fix to Node Synonymizer)
it should be working for ngd build now, please let me know if you notice any issues