Pagination of long transaction list responses

[colatkinson]

Currently, if the server generates an extremely long response, it simply returns an error message, such as

{ message: 'response too large (at least 1021611 bytes)', code: -32600 }

If the client receives this message, it will just hang indefinitely while attempting to sync.

Proposed Solution

Implement pagination in relevant methods, using block heights as a range. This has a couple advantages:

1. Allow handling of very active addresses
2. During sync, allow clients to only fetch transactions after where they left off, instead of fetching the entirety of their history. Same goes for getting tx inputs. This could conceivably lead to issues with a reorg, but the client could just fetch transactions starting from last height - n where n is e.g. 6

Affected RPC Methods

• blockchain.address.get_history
• blockchain.address.get_mempool
• blockchain.address.listunspent
• blockchain.scripthash.get_history
• blockchain.scripthash.listunspent

It seems to me like the simplest solution would be to bump the protocol version number, and add optional pagination parameters to each of these methods. The version bump would allow clients to detect support for this, and thus be able to opt to get only the new txs during sync.

[racquemis]

Just here to say i support this idea. I'm currently running into problems because of addresses with thousands of transactions in it. If we had the ability to fetch get_history in pages it would also be easier on mobile wallets during import.

[joesixpack]

This needs to be implemented ASAP. Blockchain isn't getting any smaller.

[AdamSEY]

I highly support this feature.

[luggs-co]

I also support the implementation of this feature,

[kyuupichan]

I've had some thoughts about doing history sensibly. Once I've fleshed them out I'll revisit this. I would want to require the new API for large histories and simply reject any such requests to the old API.

[kyuupichan]

@erasmospunk @ecdsa @SomberNight @colatkinson @racquemis @joesixpack @AdamSEY @luggs-co

Pleaes leave any comments you have on my proposal to solve this, which I have started to implement:

http://electrumx.readthedocs.io/en/latest/protocol-changes.html#version-1-5

[SomberNight]

As @bauerj mentioned on slack, using the last confirmed tx hash as status has the downside of it being very hard for clients to detect lying by omission. Currently, if a server does not tell a client about a tx, the client would at least realise when it connects to another server, as the address statuses (which is a hash of the whole history) would mismatch. However, if there is a confirmed tx "after" the missing tx, then the new style statuses would no longer mismatch.

I understand the main motivation for changing how status is defined is that it has a linear time complexity (in terms of size of the whole history for the scripthash), and that using the last confirmed tx hash only requires constant time.

An alternative construction could be status := H(status || H(tx || height)), where for each confirmed tx for a given scripthash, in blockchain order, you iteratively hash the current status concatted with some hash that describes this tx. (to bootstrap, you could set status to all zeroes for example)

This too would have constant time complexity to update from previous status to new status when a new txn comes. However you would need an extra constant size storage to save the status itself for a scripthash.
There are further space-time tradeoffs one can make, such as,

• to save space, set a threshold k of minimum txns associated with a scripthash before you start saving the status for the scripthash (below that, you always recompute)
• reorgs are expensive with a naive implementation. you might need to recompute the whole status. to avoid this, you could delay saving the status for a given scripthash by m blocks -- but then you also need to store some extra info to know for what tx/height the stored status is for

[kyuupichan]

Even those ideas require a server to read the whole history of any client connecting to it at least once, and as you say maintain a cache. I think the whole approach is suboptimal and unnecessary. Its only redeeming feature is its easier for client devs to work with.

I want the client to bear more of the burden of it being SPV. It is going to have to become a bit smarter, but that is putting the burden in the right place - it is the SPV, and so it is responsible for convincing itself it has got the truth. I'd like to see more ideas on improving this client-side without burdening servers with cumbersome protocols.

[SomberNight]

Even those ideas require a server to read the whole history of any client connecting to it at least once, and as you say maintain a cache.

No, you can persist it into the DB. You don't need to load the whole history when the client connects. During initial sync, you process the transaction anyway; at that time you can compute the status.
If you set e.g. k=5, the extra storage requirements should be negligible (I would guess 1 GB max for current chain, maybe a lot less).

I think the whole approach is suboptimal

I don't think there is an optimal solution as there are multiple non-independent things to optimise for.

[kyuupichan]

I think you misunderstood me

• I don't want to persist caches to the DB. Caches are a great source of cache invalidation bugs, plus they take up space. Space will be at a premium for blockchain indices; they will be huge.
• to get a cache in the first place, persisted or not, you need to read the whole history.

I don't want that to be the default behaviour the client requires of any server it connects to just to find out what happened since last week. It should have a confidence level to some height and only be requesting data from that height. It will increase its height of confidence based on results it receives from various sources to various queries, based on a statistical analysis of data it collects. Clients will be motivated to not make expensive requests, because they will become ... expensive. The user will have an option to force-refresh one or more addresses if they "appear wrong".

There is a lot of scope client-side for being smart and figuring out strategies in having confidence in served data. Lying isn't easy or free. I have several ideas but I'm sure others can come up with better ones. At present no one is even trying because the status quo is offloading the costs to server operators and it's easy to argue for no change. Also today running a server is fairly inexpensive.

I'm planning for data served becoming big, not staying small. It will inevitably become commercial and very likely be costed if used beyond a certain introductory level, because it won't be cheap to maintain indices of everything and serve them for free.