Fix parse_doi: DOI lost when a non-doi ELocationID (e.g. pii) follows it - #177
Open
ErisonChen wants to merge 1 commit into
Open
Fix parse_doi: DOI lost when a non-doi ELocationID (e.g. pii) follows it#177ErisonChen wants to merge 1 commit into
ErisonChen wants to merge 1 commit into
Conversation
`parse_doi` looped over every <ELocationID> and unconditionally
reassigned `doi` each iteration, with non-"doi" types resolving to "".
When a record lists the "doi" ELocationID before another type (e.g.
"pii"), the trailing entry overwrote the DOI with an empty string.
Example (PMID 24746000):
<ELocationID EIdType="doi" ...>10.1016/S2213-2600(14)70019-0</ELocationID>
<ELocationID EIdType="pii" ...>S2213-2600(14)70019-0</ELocationID>
parsed to doi="" instead of the real DOI. This ordering is common in
PubMed, so many records lost their DOI.
Additionally, the `ArticleIdList` fallback lived in the `else` branch of
`if len(elocation_ids) > 0`, so it was unreachable whenever any
ELocationID existed -- even if none of them carried a DOI.
Fix:
- iterate ELocationID and stop at the first EIdType="doi" entry, so a
later non-doi entry can no longer clobber it
- always fall back to PubmedData/ArticleIdList when no DOI was found in
the ELocationID list (covers records that only have e.g. a pii entry)
Add a regression test using PMID 24746000 (doi-before-pii ordering).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
parse_doiinmedline_parser.pyloops over every<ELocationID>and unconditionally reassignsdoion each iteration. AnyELocationIDwhoseEIdTypeis not"doi"resolves to"". So when a record lists thedoientry before another type (commonlypii), the trailing entry overwrites the already-parsed DOI with an empty string.Real example — PMID 24746000:
parse_medline_xmlreturnsdoi == ""instead of10.1016/S2213-2600(14)70019-0. Thisdoi-then-piiordering is very common in PubMed, so a large number of records silently lose their DOI.There is also a second, related bug: the
PubmedData/ArticleIdListfallback is in theelsebranch ofif len(elocation_ids) > 0, making it unreachable whenever anyELocationIDis present — even if none of them is a DOI (e.g. a record that only has apiiELocationID but carries the DOI inArticleIdList).Current code
Fix
ELocationIDand stop at the firstEIdType="doi"entry, so a later non-doi entry can no longer clobber it.PubmedData/ArticleIdListwhen no DOI was found in theELocationIDlist (covers records that only have e.g. apiientry).Tests
Added a regression test (
test_doi_with_trailing_pii_elocation_id) using PMID24746000, whose<Article>listsdoibeforepii. It fails on the current code and passes with the fix.Verified the new logic against four cases — doi-before-pii, pii-only with DOI in ArticleIdList, pii-before-doi, and no-DOI-at-all — all behave correctly, and the no-DOI case still returns
"".