Antalya: Cache the list objects operation on object storage using a TTL + prefix matching cache implementation #743

arthurpassos · 2025-04-17T19:47:37Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Cache for listobjects calls

Documentation entry for user-facing changes

src/Storages/Cache/ObjectStorageListObjectsCache.cpp

arthurpassos · 2025-04-18T12:01:37Z

arthur :) SELECT date, count()
FROM s3('s3://aws-public-blockchain/v1.0/btc/transactions/*/*.parquet', NOSIGN)
WHERE date between '2025-01-01' and '2025-01-31'
GROUP BY date ORDER BY date
SETTINGS use_hive_partitioning=1, use_object_storage_list_objects_cache=0;

SELECT
    date,
    count()
FROM s3('s3://aws-public-blockchain/v1.0/btc/transactions/*/*.parquet', NOSIGN)
WHERE (date >= '2025-01-01') AND (date <= '2025-01-31')
GROUP BY date
ORDER BY date ASC
SETTINGS use_hive_partitioning = 1, use_object_storage_list_objects_cache = 0

Query id: 29d096ab-0297-43a3-8844-b83b6a7856fb

    ┌─date───────┬─count()─┐
 1. │ 2025-01-01 │  292213 │
 2. │ 2025-01-02 │  402440 │
 3. │ 2025-01-03 │  409341 │
 4. │ 2025-01-04 │  432302 │
 5. │ 2025-01-05 │  433954 │
 6. │ 2025-01-06 │  366260 │
 7. │ 2025-01-07 │  352121 │
 8. │ 2025-01-08 │  399976 │
 9. │ 2025-01-09 │  534013 │
10. │ 2025-01-10 │  408769 │
11. │ 2025-01-11 │  361190 │
12. │ 2025-01-12 │  380525 │
13. │ 2025-01-13 │  408248 │
14. │ 2025-01-14 │  352684 │
15. │ 2025-01-15 │  354014 │
16. │ 2025-01-16 │  375439 │
17. │ 2025-01-17 │  425661 │
18. │ 2025-01-18 │  360666 │
19. │ 2025-01-19 │  388509 │
20. │ 2025-01-20 │  350291 │
21. │ 2025-01-21 │  324412 │
22. │ 2025-01-22 │  432369 │
23. │ 2025-01-23 │  326010 │
24. │ 2025-01-24 │  369243 │
25. │ 2025-01-25 │  338988 │
26. │ 2025-01-26 │  309651 │
27. │ 2025-01-27 │  332102 │
28. │ 2025-01-28 │  305953 │
29. │ 2025-01-29 │  355332 │
30. │ 2025-01-30 │  335134 │
31. │ 2025-01-31 │  328684 │
    └────────────┴─────────┘

31 rows in set. Elapsed: 4.080 sec. Processed 11.55 million rows, 0.00 B (2.83 million rows/s., 0.00 B/s.)
Peak memory usage: 3.17 MiB.

arthur :) SELECT date, count()
FROM s3('s3://aws-public-blockchain/v1.0/btc/transactions/*/*.parquet', NOSIGN)
WHERE date between '2025-01-01' and '2025-01-31'
GROUP BY date ORDER BY date
SETTINGS use_hive_partitioning=1, use_object_storage_list_objects_cache=1;

SELECT
    date,
    count()
FROM s3('s3://aws-public-blockchain/v1.0/btc/transactions/*/*.parquet', NOSIGN)
WHERE (date >= '2025-01-01') AND (date <= '2025-01-31')
GROUP BY date
ORDER BY date ASC
SETTINGS use_hive_partitioning = 1, use_object_storage_list_objects_cache = 1

Query id: 4afafbfb-eb93-4b96-8c4f-4a94f723805c

    ┌─date───────┬─count()─┐
 1. │ 2025-01-01 │  292213 │
 2. │ 2025-01-02 │  402440 │
 3. │ 2025-01-03 │  409341 │
 4. │ 2025-01-04 │  432302 │
 5. │ 2025-01-05 │  433954 │
 6. │ 2025-01-06 │  366260 │
 7. │ 2025-01-07 │  352121 │
 8. │ 2025-01-08 │  399976 │
 9. │ 2025-01-09 │  534013 │
10. │ 2025-01-10 │  408769 │
11. │ 2025-01-11 │  361190 │
12. │ 2025-01-12 │  380525 │
13. │ 2025-01-13 │  408248 │
14. │ 2025-01-14 │  352684 │
15. │ 2025-01-15 │  354014 │
16. │ 2025-01-16 │  375439 │
17. │ 2025-01-17 │  425661 │
18. │ 2025-01-18 │  360666 │
19. │ 2025-01-19 │  388509 │
20. │ 2025-01-20 │  350291 │
21. │ 2025-01-21 │  324412 │
22. │ 2025-01-22 │  432369 │
23. │ 2025-01-23 │  326010 │
24. │ 2025-01-24 │  369243 │
25. │ 2025-01-25 │  338988 │
26. │ 2025-01-26 │  309651 │
27. │ 2025-01-27 │  332102 │
28. │ 2025-01-28 │  305953 │
29. │ 2025-01-29 │  355332 │
30. │ 2025-01-30 │  335134 │
31. │ 2025-01-31 │  328684 │
    └────────────┴─────────┘

31 rows in set. Elapsed: 0.040 sec. Processed 11.55 million rows, 0.00 B (287.50 million rows/s., 0.00 B/s.)
Peak memory usage: 844.09 KiB.

arthur :)

arthurpassos · 2025-04-18T12:03:31Z

laptop@arthur:~/work/altinity/list_objects_cache$ ./cmake-build-release/programs/clickhouse benchmark -i 10 --cumulative -q "SELECT date, count()
FROM s3('s3://aws-public-blockchain/v1.0/btc/transactions/*/*.parquet', NOSIGN)
WHERE date between '2025-01-01' and '2025-01-31'
GROUP BY date ORDER BY date
SETTINGS use_hive_partitioning=1, use_object_storage_list_objects_cache=0;"

Queries executed: 10.

localhost:9000, queries: 10, QPS: 0.389, RPS: 4487684.448, MiB/s: 0.000, result RPS: 12.049, result MiB/s: 0.000.

0%		2.363 sec.	
10%		2.379 sec.	
20%		2.382 sec.	
30%		2.391 sec.	
40%		2.402 sec.	
50%		2.410 sec.	
60%		2.410 sec.	
70%		2.451 sec.	
80%		2.458 sec.	
90%		3.159 sec.	
95%		3.212 sec.	
99%		3.212 sec.	
99.9%		3.212 sec.	
99.99%		3.212 sec.

laptop@arthur:~/work/altinity/list_objects_cache$ ./cmake-build-release/programs/clickhouse benchmark -i 10 --cumulative -q "SELECT date, count()
FROM s3('s3://aws-public-blockchain/v1.0/btc/transactions/*/*.parquet', NOSIGN)
WHERE date between '2025-01-01' and '2025-01-31'
GROUP BY date ORDER BY date
SETTINGS use_hive_partitioning=1, use_object_storage_list_objects_cache=1;"
Loaded 1 queries.

Queries executed: 10.

localhost:9000, queries: 10, QPS: 33.280, RPS: 384262109.406, MiB/s: 0.000, result RPS: 1031.666, result MiB/s: 0.028.

0%		0.015 sec.	
10%		0.015 sec.	
20%		0.015 sec.	
30%		0.016 sec.	
40%		0.017 sec.	
50%		0.017 sec.	
60%		0.017 sec.	
70%		0.018 sec.	
80%		0.018 sec.	
90%		0.018 sec.	
95%		0.018 sec.	
99%		0.018 sec.	
99.9%		0.018 sec.	
99.99%		0.018 sec.

:)

arthurpassos · 2025-04-18T13:44:32Z

src/Storages/Cache/ObjectStorageListObjectsCache.cpp

+    {
+        if (const auto it = cache.find(key); it != cache.end())
+        {
+            if (IsStaleFunction()(it->first))


This case is interesting. In case we find an exact match, but it has expired. Should we try to find a prefix match or simply update the entry?

Well, there can be a more up-to-date prefix entry, so why not try to reuse it

The only reason is that this entry would cease to exist. It would never be cached again. And it would become a linear search forever.

Actually, not forever, if the more up-to-date prefix entry gets evicted and this query is performed again, it would re-appear.

But I think you are right.

zvonand · 2025-04-18T14:00:38Z

src/Storages/ObjectStorage/StorageObjectStorageSource.cpp

+        {
+            throw Exception(
+                ErrorCodes::BAD_ARGUMENTS,
+                "Using glob iterator with path without globs is not allowed (used path: {})",


Shall not it be LOGICAL_ERROR ?
This looks like a branch of code where we cannot get normally (user does not select which iterator to use manually)

I agree, it should probably be LOGICAL_ERROR. But:

This is mostly a copy and paste from the existing GlobIterator.

I might refactor this to avoid duplication. For now, this is just a draft implementation.

Even if I refactor this, I would opt for keeping parity with existing code & upstream. This will make the review and merges with upsteam easier

…ear prefix matchin search

Enmk · 2025-04-18T17:47:49Z

src/Core/Settings.cpp

@@ -6108,6 +6108,9 @@ Limit for hosts used for request in object storage cluster table functions - azu
 Possible values:
 - Positive integer.
 - 0 — All hosts in cluster.
+)", EXPERIMENTAL)                       \
+    DECLARE(Bool, use_object_storage_list_objects_cache, true, R"(


please add it to the src/Core/SettingsChangesHistory.cpp

Enmk · 2025-04-18T17:49:11Z

src/Storages/Cache/ObjectStorageListObjectsCache.cpp

+    cache.setMaxCount(count);
+}
+
+void ObjectStorageListObjectsCache::setTTL(std::size_t ttl_)


is it in seconds/miliseconds/minutes/hours?

In seconds, will modify the argument name

Enmk · 2025-04-18T21:25:21Z

src/Interpreters/InterpreterSystemQuery.cpp

@@ -435,6 +436,16 @@ BlockIO InterpreterSystemQuery::execute()
            break;
 #else
            throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, "The server was compiled without the support for Parquet");
+#endif
+        }
+        case Type::DROP_OBJECT_STORAGE_LIST_OBJECTS_CACHE:


Is caching works only on Parquet files or generally on any S3 ListObject requests?

Ah, copy and paste issues. Should be any :D

Enmk · 2025-04-18T21:30:58Z

src/Storages/ObjectStorage/StorageObjectStorageSource.cpp

+            throw Exception(
+                ErrorCodes::BAD_ARGUMENTS,
+                "Using glob iterator with path without globs is not allowed (used path: {})",


This is a minor nitpick, but maybe throw early, i.e. effectively do something like this?:

if (!configuration->isPathWithGlobs()) { throw Exception(...); } // rest of the function as it was, but not inside indented block.

Yeah, agreed. Like I said, this is both: 1) a copy & paste; 2) wip.

I'll look into it with more attention once the core features and testing have been implemented.

arthurpassos · 2025-04-19T14:35:26Z

Kind of problematic to test this using stateless or integeration tests. A single glob query can perform multiple list object calls, which affect profileevents counters. Not only that, but some of these list calls do not iterate through the entire list, hence not inserting into the cache.

Relying on hard-coded numbers based on current behavior is kind of bad. Best I can do with stateless test would be some sort of event_counter > 0.

Or test the cache alone using unit tests.

zvonand · 2025-04-19T14:40:46Z

Or test the cache alone using unit tests.

As long as we are storing the cache in the system anyway, maybe we could make it available as some kind of system table, e.g. system.s3_prefix_cache? This also could be greatly helpful for debugging

arthurpassos · 2025-04-19T17:18:10Z

As long as we are storing the cache in the system anyway,

I didn't understand what you mean by this

maybe we could make it available as some kind of system table, e.g. system.s3_prefix_cache

Yeah, we need to design one that also covers #586, just not sure I'll include it in this PR

zvonand · 2025-04-19T17:29:03Z

I meant that "we have a cache, why not make an interface to view it"

src/Storages/ObjectStorage/StorageObjectStorageSource.cpp

…factor

src/Storages/ObjectStorage/StorageObjectStorageSource.cpp

arthurpassos · 2025-04-21T19:59:52Z

Reverted my last two commits due to the performance degradation #743 (comment)

src/Storages/ObjectStorage/StorageObjectStorageSource.cpp

…tions

ianton-ru · 2025-04-22T13:31:05Z

src/Storages/Cache/ObjectStorageListObjectsCache.cpp

+
+        std::vector<Key> to_remove;
+
+        for (auto it = cache.begin(); it != cache.end(); ++it)


I don't like this cycle.
In my opinion better something like (in pseudocode)

while (!key.prefix.empty()) { if (auto res = cache.getWithKey(key)) { // should not be more than one passed key, if key '/foo/bar' exists, key '/foo' can't be in cache if (IsStaleFunction(res)) { BasePolicy::remove(res); return std::nullopt; } else return res; } key.prefix.pop_back(); }

Can you explain why it is better?

And why do you assume the below?

{ // should not be more than one passed key, if key '/foo/bar' exists, key '/foo' can't be in cache

Well, assuming this version you suggested works, the time complexity goes to O(key_path_size). Which should probably be better than O(N).

But it won't find "the best match", tho.

Just implemented it, can you please have a look?

Btw, thanks for the suggestion, it's a great one.

src/Storages/Cache/ObjectStorageListObjectsCache.cpp

ianton-ru

Wrote some comments.

…f O(key_size) rather than O(cache_length)

ianton-ru

LGTM

arthurpassos added 2 commits April 17, 2025 16:46

draft immpl

3dc33f3

remove garbage

7e182c7

arthurpassos changed the title ~~draft immpl~~ Cache the list objects operation on object storage using a TTL + prefix matching cache implementation Apr 17, 2025

arthurpassos added 3 commits April 17, 2025 17:01

use starts_with

ef985c9

fix a few obvious bugs

43c1383

some other fixes

989cfe0

arthurpassos force-pushed the list_objects_cache branch from 18592c1 to 989cfe0 Compare April 17, 2025 21:02

arthurpassos added 2 commits April 17, 2025 20:16

another fix

157450d

Merge branch 'antalya' into list_objects_cache

d4af4ae

arthurpassos commented Apr 18, 2025

View reviewed changes

src/Storages/Cache/ObjectStorageListObjectsCache.cpp Outdated Show resolved Hide resolved

arthurpassos commented Apr 18, 2025

View reviewed changes

src/Storages/Cache/ObjectStorageListObjectsCache.cpp Outdated Show resolved Hide resolved

arthurpassos commented Apr 18, 2025

View reviewed changes

src/Storages/Cache/ObjectStorageListObjectsCache.cpp Outdated Show resolved Hide resolved

arthurpassos mentioned this pull request Apr 18, 2025

Cache for ListObjects #744

Open

arthurpassos added 2 commits April 18, 2025 10:37

remove unused method

727b64d

some other fixes

68cbad7

arthurpassos commented Apr 18, 2025

View reviewed changes

some other fixes

00f58b3

zvonand reviewed Apr 18, 2025

View reviewed changes

arthurpassos added 4 commits April 18, 2025 11:17

use steady_clock instead of system_clock

67ccaf0

small fixes

2b37e0c

Dont return null in case of exact match with stale entry, perform lin…

0d6e343

…ear prefix matchin search

add some metrics

0f605c4

svb-alt added the antalya-25.2.2 Planned for 25.2.2 release label Apr 18, 2025

Enmk reviewed Apr 18, 2025

View reviewed changes

implement cache clear command

3ed2349

Enmk reviewed Apr 18, 2025

View reviewed changes

remove ifdef from parquet

f345b33

Enmk reviewed Apr 18, 2025

View reviewed changes

arthurpassos added 6 commits April 20, 2025 17:39

add stateless tests

9242843

add unit tests

4e19b09

Merge branch 'antalya' into list_objects_cache

7a6eaec

rename ttl argument and member variable

8c4ea48

new settings history

74b980c

make the setting false by default so other tests are not affected

4f55a75

arthurpassos commented Apr 21, 2025

View reviewed changes

src/Storages/ObjectStorage/StorageObjectStorageSource.cpp Outdated Show resolved Hide resolved

Enmk changed the title ~~Cache the list objects operation on object storage using a TTL + prefix matching cache implementation~~ Antalya: Cache the list objects operation on object storage using a TTL + prefix matching cache implementation Apr 21, 2025

arthurpassos added 3 commits April 21, 2025 16:10

add ref file

303ee27

remove cachedglobiterator in favor of expensive copy. I think I'll re…

e6b379e

…factor

simplify things a bit

d7b50f4

arthurpassos commented Apr 21, 2025

View reviewed changes

src/Storages/ObjectStorage/StorageObjectStorageSource.cpp Outdated Show resolved Hide resolved

arthurpassos commented Apr 22, 2025

View reviewed changes

src/Storages/ObjectStorage/StorageObjectStorageSource.cpp Outdated Show resolved Hide resolved

arthurpassos force-pushed the list_objects_cache branch from 0b57378 to d7b50f4 Compare April 22, 2025 12:28

arthurpassos added 3 commits April 22, 2025 09:46

add some more tests

6cfa510

make cache return a copy instead of a pointer, we don't want modifica…

be8c6a1

…tions

update ut

b60cb95

ianton-ru reviewed Apr 22, 2025

View reviewed changes

improve prefix matching by implementing search with time complexity o…

d91bf00

…f O(key_size) rather than O(cache_length)

ianton-ru approved these changes Apr 24, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Antalya: Cache the list objects operation on object storage using a TTL + prefix matching cache implementation #743

Antalya: Cache the list objects operation on object storage using a TTL + prefix matching cache implementation #743

arthurpassos commented Apr 17, 2025

arthurpassos commented Apr 18, 2025

arthurpassos commented Apr 18, 2025

arthurpassos Apr 18, 2025

zvonand Apr 18, 2025

arthurpassos Apr 18, 2025

zvonand Apr 18, 2025

arthurpassos Apr 18, 2025

Enmk Apr 18, 2025

Enmk Apr 18, 2025

arthurpassos Apr 18, 2025

Enmk Apr 18, 2025 •

edited

Loading

arthurpassos Apr 18, 2025

arthurpassos Apr 18, 2025

Enmk Apr 18, 2025

arthurpassos Apr 18, 2025

arthurpassos commented Apr 19, 2025

zvonand commented Apr 19, 2025

arthurpassos commented Apr 19, 2025

zvonand commented Apr 19, 2025

arthurpassos commented Apr 21, 2025

ianton-ru Apr 22, 2025

arthurpassos Apr 22, 2025

arthurpassos Apr 22, 2025

arthurpassos Apr 22, 2025

ianton-ru left a comment

ianton-ru left a comment


		std::vector<Key> to_remove;

		for (auto it = cache.begin(); it != cache.end(); ++it)

Antalya: Cache the list objects operation on object storage using a TTL + prefix matching cache implementation #743

Are you sure you want to change the base?

Antalya: Cache the list objects operation on object storage using a TTL + prefix matching cache implementation #743

Conversation

arthurpassos commented Apr 17, 2025

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

arthurpassos commented Apr 18, 2025

arthurpassos commented Apr 18, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Enmk Apr 18, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arthurpassos commented Apr 19, 2025

zvonand commented Apr 19, 2025

arthurpassos commented Apr 19, 2025

zvonand commented Apr 19, 2025

arthurpassos commented Apr 21, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ianton-ru left a comment

Choose a reason for hiding this comment

ianton-ru left a comment

Choose a reason for hiding this comment

Enmk Apr 18, 2025 •

edited

Loading