Token Transfer Processor - Derive classic events from operation and operation results #5609

karthikiyer56 · 2025-02-19T05:21:23Z

This PR deals with deriving TokenTransferEvents from operation and operation results as described in #5580
This PR is strictly dealing with classic and classic operations, i.e the PR implements the ask from #5592

amount/main.go

… operations

ingest/processors/token_transfer/token_transfer_processor.go

…action_envelope and the test fixture. It is not used anyway

…rios

… work, regardless of order

karthikiyer56 · 2025-02-22T04:50:12Z

ingest/ledger_transaction.go

@@ -592,31 +596,6 @@ func (t *LedgerTransaction) AccountMuxed() (string, bool) {

 }

-func (t *LedgerTransaction) FeeAccount() (string, bool) {


The name of the function seems to suggest that this function will give the underlying fee account - be it a regular transaction or a fee bump transaction.
IMO, this is very misleading.

I found that it behaved contrary to what the name suggested when my code was not working as expected, in that if it the tx is not a fee bump tx, it returns "".

xxxAccount being the function name, i'd expect the return type to be xdr.MuxedAccount, to be consistent with other functions here.
Besides, I dont see it being used anywhere other than in a sample unit test.
I am removing this function and the function below so as to not cause further confusion

I agree with your assessment but we should sync with @chowbao to make sure etl isn't impacted by this change

ingest/ledger_transaction_test.go

ingest/processors/token_transfer/token_transfer_event_test.go

karthikiyer56 · 2025-02-22T05:01:15Z

ingest/processors/token_transfer/token_transfer_processor_test.go

+
+import (
+	"github.com/stellar/go/ingest"
+	addressProto "github.com/stellar/go/ingest/address"


This file is the real deal and needs to be reviewed thoroughly.
Adding a comment to ensure that this doesnt slip up - since the load diff might not render the file.

karthikiyer56 · 2025-02-22T05:03:48Z

ingest/processors/token_transfer/token_transfer_processor_test.go

+)
+
+var (
+	someTxAccount = xdr.MustMuxedAddress("GBF3XFXGBGNQDN3HOSZ7NVRF6TJ2JOD5U6ELIWJOOEI6T5WKMQT2YSXQ")


There are several variables here that I am using throughout the test, and so they have been clubbed here at the top.
They, IMHO, improve the code readability when reviewing the tests, and I am not inclined to move them to individual tests, but rather have them at the top-level here

…ad of a muxed account. This helps simplify the logic at call site for the operations when called with LPhash or CBid

…entries from operation changes

…y the most complicated functions to implement

…ase to validate some logic

karthikiyer56 · 2025-03-01T08:51:44Z

services/horizon/internal/integration/change_test.go

@@ -18,6 +22,240 @@ import (
 	"github.com/stellar/go/xdr"
 )

+func TestLiquidityPoolHappyPath2(t *testing.T) {


the changes in this integration test are purely for PoC.
will be removed later

…ated this claimable balance

… work after all

ingest/processors/token_transfer/token_transfer_processor.go

tamirms · 2025-03-03T18:12:21Z

ingest/processors/token_transfer/token_transfer_processor.go

+	return generatedClaimableBalanceIds, nil
+}
+
+func getClaimableBalanceEntriesFromOperationChanges(tx ingest.LedgerTransaction, opIndex uint32) ([]xdr.ClaimableBalanceEntry, error) {


this function is used to find claimable balances that were either created or removed. however, the caller expects the returned list to never have a mix between the two types. I think it would be better to add a xdr.LedgerEntryChangeType parameter which can be used to filter for the exact change types desired by the caller (e.g. only created claimable balances or only removed claimable balances)

Seeing that this is an internal function, I see little chance for misuse/mis-representation of what this function is intended to do.
at any rate, I have changed the function to accept a changeType xdr.LedgerEntryChangeType and added a comment at the top.
callers of the function will either pass - LedgerEntryChangeTypeLedgerEntryRemoved or LedgerEntryChangeTypeLedgerEntryCreated. It will fail when anything else is passed

ingest/processors/token_transfer/token_transfer_processor.go

tamirms · 2025-03-03T23:20:59Z

ingest/processors/token_transfer/token_transfer_processor.go

+	if len(entries) == 0 {
+		return nil, ErrNoLiquidityPoolEntryFound
+	}


nit: I think it's better style to return the empty list without an error so the caller can decide how the empty list should be handled. it's not obvious from the name of the function that at least 1 liquidity pool should be present and anything otherwise would be an error

reworked to do error/length check at call-site instead of inside the function

tamirms · 2025-03-03T23:22:46Z

ingest/processors/token_transfer/token_transfer_processor.go

+	if len(entries) == 0 {
+		return nil, ErrNoClaimableBalanceEntryFound
+	}


nit, same comment as ErrNoLiquidityPoolEntryFound:

I think it's better style to return the empty list without an error so the caller can decide how the empty list should be handled.

reworked to do error/length check at call-site instead of inside the function

…dress code review comments

…n and its results

karthikiyer56 · 2025-03-04T07:16:03Z

ingest/processors/token_transfer/token_transfer_processor.go

+	return generateEventsForRevokedTrustlines(tx, opIndex)
+}
+
+func generateEventsForRevokedTrustlines(tx ingest.LedgerTransaction, opIndex uint32) ([]*TokenTransferEvent, error) {


this is the dooziest logic of all the code in this file.
I have written unit tests that test this ad-nauseum
This is the only part of the code where I have felt the need to be over-liberal with comments to indicate what the code is doing, since it is very non-trivial how the Cbs are generated from LPs when trustline revocation of an asset happens for a trustor account

I reviewed the logic and it looks correct to me. I think the code would be easier to understand if you refactored it to avoid iterating over impactedLiquidityPools twice, but that's just my personal opinion. Here's my attempt to simplify the code:

ab27384

feel free to take or reject whichever parts you see fit

karthikiyer56 · 2025-03-04T07:24:41Z

support/converters/main.go

+)
+
+/*
+Helper function to convert LiquidityPoolId to generate ClaimableBalanceIds in a deterministic way as per


I needed a place to park this generic logic that generates CBids from LPids.
I use this in the code and also in the unit tests.

It didnt make sense to subsume it under token_transfer_processor, so parking it here under a new directory -converters under support
Happy to move this to another place, so long as this is a public function.

I would put it either in the xdr package or along with the xdrill code in the ingest package. Also, I suggest using a more concise name for the function, maybe, ClaimableBalanceIdFromRevocation or BalanceIdFromRevocation?

tamirms · 2025-03-04T10:34:46Z

ingest/processors/token_transfer/token_transfer_processor.go

+	if !(changeType == xdr.LedgerEntryChangeTypeLedgerEntryRemoved || changeType == xdr.LedgerEntryChangeTypeLedgerEntryCreated) {
+		return nil, fmt.Errorf("changeType: %v, not allowed", changeType)
+	}


I don't think this error is necessary. The function will obtain all cb changes occurring from an operation filtered by a change type. The fact that the function happens to be called only for removed or created changes is something determined by the caller

oh nevermind, I just realized that with updates it's not clear which claimable balance (either pre or post) to include. you can ignore this comment

you could simplify the if statement condition to:

Suggested change

if !(changeType == xdr.LedgerEntryChangeTypeLedgerEntryRemoved || changeType == xdr.LedgerEntryChangeTypeLedgerEntryCreated) {

return nil, fmt.Errorf("changeType: %v, not allowed", changeType)

}

if changeType == xdr.LedgerEntryChangeTypeLedgerEntryUpdated {

return nil, fmt.Errorf("LEDGER_ENTRY_UPDATED is not a valid filter")

}

tamirms · 2025-03-04T10:42:51Z

ingest/processors/token_transfer/token_transfer_processor.go

+		if change.Type != xdr.LedgerEntryTypeClaimableBalance {
+			continue
+		}
+		// Check if claimable balance entry is deleted
+		//?? maybe it is not necessary to check change.Pre != nil && change.Post since that will be true for deleted entries?
+		if changeType == xdr.LedgerEntryChangeTypeLedgerEntryRemoved && change.Pre != nil && change.Post == nil {
+			cb = change.Pre.Data.MustClaimableBalance()
+			entries = append(entries, cb)
+		} else if changeType == xdr.LedgerEntryChangeTypeLedgerEntryCreated && change.Post != nil && change.Pre == nil { // check if claimable balance entry is created
+			cb = change.Post.Data.MustClaimableBalance()
+			entries = append(entries, cb)
+		}


Suggested change

if change.Type != xdr.LedgerEntryTypeClaimableBalance {

continue

}

// Check if claimable balance entry is deleted

//?? maybe it is not necessary to check change.Pre != nil && change.Post since that will be true for deleted entries?

if changeType == xdr.LedgerEntryChangeTypeLedgerEntryRemoved && change.Pre != nil && change.Post == nil {

cb = change.Pre.Data.MustClaimableBalance()

entries = append(entries, cb)

} else if changeType == xdr.LedgerEntryChangeTypeLedgerEntryCreated && change.Post != nil && change.Pre == nil { // check if claimable balance entry is created

cb = change.Post.Data.MustClaimableBalance()

entries = append(entries, cb)

}

if change.Type != xdr.LedgerEntryTypeClaimableBalance || change.LedgerEntryChangeType() != changeType {

continue

}

if change.Pre != nil {

cb = change.Pre.Data.MustClaimableBalance()

entries = append(entries, cb)

} else if change.Post != nil {

cb = change.Post.Data.MustClaimableBalance()

entries = append(entries, cb)

}

tamirms · 2025-03-04T11:08:15Z

ingest/processors/token_transfer/token_transfer_processor.go

+	amtA, amtB := delta.amountChangeForAssetA, delta.amountChangeForAssetB
+	if amtA <= 0 {
+		return nil,
+			fmt.Errorf("deposited amount (%v) for asset: %v, cannot be negative in LiquidityPool: %v", amtA, assetA.String(), lpIdToStrkey(lpId))


error string indicates negative values are not allowed but the if statement is actually including 0 values as well in the illegal state

tamirms · 2025-03-04T11:25:46Z

ingest/processors/token_transfer/token_transfer_processor.go

+var (
+	xlmProtoAsset                   = assetProto.NewNativeAsset()
+	ErrNoLiquidityPoolEntryFound    = errors.New("no liquidity pool entry found in operation changes")
+	ErrNoClaimableBalanceEntryFound = errors.New("no claimable balance entry found in operation changes")


this is no longer used

tamirms · 2025-03-04T13:42:19Z

ingest/processors/token_transfer/token_transfer_processor.go

+
+func lpIdToStrkey(lpId xdr.PoolId) string {
+	return xdr.Hash(lpId).HexString()
+}


this file is quite long, it would be easier to read the code if the code in this file was split into multiple files. maybe this could be refactored into three files:

one for path payment and offer related operations

one for revocation and liquidity pool operations

one for everything else

karthikiyer56 added 3 commits February 18, 2025 21:21

Initial boiler plate commit for token transfer processor

291c0b6

Add support for feeEvent

b27d886

Add support for createAccount and accountMerge

f76543a

karthikiyer56 linked an issue Feb 19, 2025 that may be closed by this pull request

Generate TokenTransferEvents from operation and operation results for classic operations #5592

Open

karthikiyer56 requested a review from tamirms February 20, 2025 05:00

karthikiyer56 force-pushed the karthik/ttp-classic-operations branch from 159c84a to 36946d4 Compare February 20, 2025 05:08

karthikiyer56 added 2 commits February 19, 2025 21:10

Add support for simple payment

b97f95c

Add unit tests for createAccount, mergeAccount, simple Payment

571679c

karthikiyer56 force-pushed the karthik/ttp-classic-operations branch from 36946d4 to 571679c Compare February 20, 2025 23:41

karthikiyer56 requested a review from Shaptic February 20, 2025 23:41

karthikiyer56 commented Feb 20, 2025

View reviewed changes

amount/main.go Show resolved Hide resolved

karthikiyer56 added 2 commits February 20, 2025 17:29

refactor unit tests

eb92532

Add logic for ManageBuyOffer, ManageSellOffer, CreatePassiveSellOffer…

5aebc69

… operations

tamirms reviewed Feb 21, 2025

View reviewed changes

ingest/processors/token_transfer/token_transfer_processor.go Outdated Show resolved Hide resolved

tamirms reviewed Feb 21, 2025

View reviewed changes

ingest/processors/token_transfer/token_transfer_processor.go Show resolved Hide resolved

karthikiyer56 added 5 commits February 21, 2025 10:34

Add comments to explain sellerId in manageBuy/Sell operations

00cead5

Remove the incorreectly named FeeAccount receiver function from trans…

a858f64

…action_envelope and the test fixture. It is not used anyway

Update the way we generate the fee event

30472ff

rework code and add fixtures for all manageOffer operations and scena…

8e7f358

…rios

Make the assertion on the equality of proto expected vs actual events…

3e731c5

… work, regardless of order

karthikiyer56 commented Feb 22, 2025

View reviewed changes

ingest/ledger_transaction_test.go Show resolved Hide resolved

karthikiyer56 commented Feb 22, 2025

View reviewed changes

ingest/processors/token_transfer/token_transfer_event_test.go Show resolved Hide resolved

karthikiyer56 commented Feb 22, 2025

View reviewed changes

karthikiyer56 added 5 commits February 21, 2025 21:10

rework some test code

e52beb3

rework some anonymous functions in test cases

088ca7e

Add unit test for feeEvent

d6969b2

Unit test cleanup

816e550

Make mintOrBurnOrTransferEvent function take an address wrapper inste…

ab52fd5

…ad of a muxed account. This helps simplify the logic at call site for the operations when called with LPhash or CBid

karthikiyer56 added 4 commits February 28, 2025 15:25

rework code for finding liquidity pool details and claimable balance …

ce666c4

…entries from operation changes

logic for allowTrust and setTrustlineFlags operations. These areeasil…

ecab46d

…y the most complicated functions to implement

Ignore the changes in this file for now. this is just a sample test c…

5313f59

…ase to validate some logic

undo changes to xdr/claimable_balance.go

aa13ae9

karthikiyer56 commented Mar 1, 2025

View reviewed changes

karthikiyer56 added 4 commits March 1, 2025 11:16

Fix bug in the way comparision was happening to find out which LP cre…

0f87a2b

…ated this claimable balance

More working changes to integration test. Made the setTrustline flags…

551eea5

… work after all

ensure that the LP delta is always positive

f3126a8

add one more test for burn event from trustline revocation

77304e9

tamirms reviewed Mar 3, 2025

View reviewed changes

ingest/processors/token_transfer/token_transfer_processor.go Show resolved Hide resolved

tamirms reviewed Mar 3, 2025

View reviewed changes

ingest/processors/token_transfer/token_transfer_processor.go Show resolved Hide resolved

karthikiyer56 added 3 commits March 3, 2025 10:34

sample test cases for revoke trustline case and some code cleanup

6a6e9af

working test for account

a5a2583

pull out the integration test into its own file

f80473a

tamirms reviewed Mar 3, 2025

View reviewed changes

karthikiyer56 added 2 commits March 3, 2025 22:53

Add extensive unit tests for revoke trustline case. Rework code to ad…

7f5c233

…dress code review comments

Rename function to indicate that the events are derived from operatio…

a4a9e24

…n and its results

karthikiyer56 changed the title ~~Initial boiler plate commit for token transfer processor~~ Token Transfer Processor - Derive classic events from operation and operation results Mar 4, 2025

karthikiyer56 requested review from a team March 4, 2025 07:01

karthikiyer56 commented Mar 4, 2025

View reviewed changes

correct the comments

d444f9d

karthikiyer56 commented Mar 4, 2025

View reviewed changes

tamirms reviewed Mar 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Token Transfer Processor - Derive classic events from operation and operation results #5609

Token Transfer Processor - Derive classic events from operation and operation results #5609

karthikiyer56 commented Feb 19, 2025 •

edited

Loading

karthikiyer56 Feb 22, 2025 •

edited

Loading

tamirms Mar 4, 2025

karthikiyer56 Feb 22, 2025

karthikiyer56 Feb 22, 2025

karthikiyer56 Mar 1, 2025 •

edited

Loading

tamirms Mar 3, 2025

karthikiyer56 Mar 4, 2025

tamirms Mar 3, 2025

karthikiyer56 Mar 4, 2025

tamirms Mar 3, 2025

karthikiyer56 Mar 4, 2025

karthikiyer56 Mar 4, 2025

tamirms Mar 4, 2025 •

edited

Loading

karthikiyer56 Mar 4, 2025

tamirms Mar 4, 2025

tamirms Mar 4, 2025

tamirms Mar 4, 2025

tamirms Mar 4, 2025

tamirms Mar 4, 2025

tamirms Mar 4, 2025

tamirms Mar 4, 2025

tamirms Mar 4, 2025

		@@ -592,31 +596,6 @@ func (t *LedgerTransaction) AccountMuxed() (string, bool) {

		}

		func (t *LedgerTransaction) FeeAccount() (string, bool) {

Token Transfer Processor - Derive classic events from operation and operation results #5609

Are you sure you want to change the base?

Token Transfer Processor - Derive classic events from operation and operation results #5609

Conversation

karthikiyer56 commented Feb 19, 2025 • edited Loading

karthikiyer56 Feb 22, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karthikiyer56 Mar 1, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tamirms Mar 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karthikiyer56 commented Feb 19, 2025 •

edited

Loading

karthikiyer56 Feb 22, 2025 •

edited

Loading

karthikiyer56 Mar 1, 2025 •

edited

Loading

tamirms Mar 4, 2025 •

edited

Loading