ingest: Update ingestion library and horizon to adapt to protocol 23 representation of evictions #5619
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Checklist
PR Structure
otherwise).
services/friendbot
, orall
ordoc
if the changes are broad or impact manypackages.
Thoroughness
.md
files, etc... affected by this change). Take a look in the
docs
folder for a given service,like this one.
Release planning
CHANGELOG.md
within the component folder structure. For example, if I changed horizon, then I updated (services/horizon/CHANGELOG.md. I add a new line item describing the change and reference to this PR. If I don't update a CHANGELOG, I acknowledge this PR's change may not be mentioned in future release notes.semver, or if it's mainly a patch change. The PR is targeted at the next
release branch if it's not a patch change.
What
CAP-62 and CAP-66 updates the LedgerCloseMeta xdr to no longer populate the
evictedPersistentLedgerEntries
field. Instead, both persistent and temporary ledger keys will be included in theevictedTemporaryLedgerKeys
field which will be renamed toevictedKeys
in protocol 23.The ingestion library previously presented evictions of persistent ledger entries as a
Change
instance with theReason
set toLedgerEntryChangeReasonEviction
. This allowed downstream consumers of aChange
stream to handle evictions as if they were equivalent to ledger entry deletions. However, in protocol 23 we can no longer maintain this strategy because the full contents of the evicted ledger entry will not be present in the LedgerCloseMeta xdr. We will only have the ledger key for the evicted ledger entry.This means that ledger entry evictions will need to be handled outside of
Change
stream processing by downstream services. The second commit of this PR fixes the asset stats processor in Horizon to adapt to this new reality where evictions no longer appear in theChange
stream.The asset stats processor can ignore evicted contract balance ledger entries because contract balance ledger entries are deleted from the horizon db immediately upon expiration. However, the asset stats processor still needs to ingest the stellar asset contract instance storage so that we can link stellar classic assets with their corresponding contract id. So when the stellar asset contract instance storage is evicted we also need to handle that event so we can remove the link between stellar classic asset and the stellar asset contract id.
Also fixes #5578
Known limitations
[N/A]