Remove redundant locations when constructing access policies #2149
Conversation
polaris-core/src/main/java/org/apache/polaris/core/storage/StorageUtil.java
Outdated
Show resolved
Hide resolved
polaris-core/src/main/java/org/apache/polaris/core/storage/StorageUtil.java
Outdated
Show resolved
Hide resolved
|
|
||
| /** Removes "redundant" locations, so {/a/b/, /a/b/c, /a/b/d} will be reduced to just {/a/b/} */ | ||
| private static @Nonnull Set<String> removeRedundantLocations(Set<String> locationStrings) { | ||
| HashSet<String> result = new HashSet<>(locationStrings); |
There was a problem hiding this comment.
Since this is a new collection, it can be produced by
locationStrings.stream()
.filter(Objects::nonNull)
.map(StorageLocation::of)
.collect(Collectors.toCollection(HashSet::new));There was a problem hiding this comment.
That would remove duplicate locations, but not redundant locations like we want to. We'd still need to loop over the collection.
There was a problem hiding this comment.
Yes, but you'd save the exponential instantiation of SotrageLocation objects
There was a problem hiding this comment.
Actually you could safe the inner loop with a sorted collection, if the locations end with a /.
polaris-core/src/test/java/org/apache/polaris/core/storage/StorageUtilTest.java
Show resolved
Hide resolved
service/common/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java
Outdated
Show resolved
Hide resolved
| @@ -2612,16 +2572,6 @@ protected FileIO loadFileIO(String ioImpl, Map<String, String> properties) { | |||
| callContext, ioImpl, properties, identifier, locations, storageActions, resolvedPath); | |||
| } | |||
|
|
|||
| private void blockedUserSpecifiedWriteLocation(Map<String, String> properties) { | |||
There was a problem hiding this comment.
loadFileIO is also unused and should be removed as well.
There was a problem hiding this comment.
Agreed that it can be removed, but unlike blockedUserSpecifiedWriteLocation it's not related to this PR so I think we should do that separately.
Iceberg tables can technically store data across any number of paths, but Polaris currently uses 3 different locations for credential vending:
write.data.path, if setwrite.metadata.path, if setThis was intended to capture scenarios where e.g. (2) is not a child path of (1), so that the vended credentials can still be valid for reading the entire table. However, there are systems that seem to always set (2) and (3), such as:
s3:/my-bucket/base/icebergs3:/my-bucket/base/iceberg/datas3:/my-bucket/base/iceberg/metadataIn such cases the extra paths (e.g. extra resources in the AWS Policy) are redundant. In one such case, these redundant paths caused the policy to exceed the maximum allowable 2048 characters.
This PR removes redundant paths -- those that are the child of another path -- from the list of accessible locations tracked for a given table and does some slight refactoring to consolidate the logic for extracting these paths from a TableMetadata.