Extend FileIOFactory to Improve Customization #724

XJDKC · 2025-01-13T23:00:32Z

Context

Currently, we have three places where a FileIO is constructed:

PolarisBaseCatalog::loadFileIO: Used as the default FileIO across the entire catalog during testing.
PolarisBaseCatalog::refreshIOWithCredential: Refreshes storage credentials and constructs a new FileIO.
TaskFileIOSupplier::apply: Fetches subscoped credentials and uses FileIOFactory to create a FileIO for cleanup tasks.

We’ve identified an issue where not all properties required by FileIO are generated by the FileIOFactory. For instance, subscoped credentials are retrieved outside the FileIOFactory and then passed to it. Additionally, the current FileIOFactory interface has limited access to contextual information.

To enhance the ability to customize FileIOFactory, we plan to extend it by passing additional information. This improvement will enable developers to implement custom logic such as auditing, proxy, and more.

Some code pointers:

snazy · 2025-01-14T10:38:38Z

Thanks for your contribution @XJDKC!
I think this change blurs the line between "properties" and "internal properties" quite a bit. I'm also, in general, concerned about generic property bag per se containing all kinds of information - things that can be exposed (upstream/downstream) and things that must not - nobody can ensure that no "bad things" (think: logging secrets) happen to this information.

service/common/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java

dimas-b

Thanks for the update @XJDKC ! Overall, the new code looks good to me :) A few minor comments below.

service/common/src/main/java/org/apache/polaris/service/catalog/io/FileIOFactory.java

service/common/src/main/java/org/apache/polaris/service/catalog/io/DefaultFileIOFactory.java

service/common/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java

service/common/src/main/java/org/apache/polaris/service/catalog/io/FileIOFactory.java

service/common/src/main/java/org/apache/polaris/service/catalog/io/DefaultFileIOFactory.java

...service/src/test/java/org/apache/polaris/service/quarkus/catalog/BasePolarisCatalogTest.java

service/common/src/main/java/org/apache/polaris/service/catalog/io/FileIOFactory.java

...service/src/test/java/org/apache/polaris/service/quarkus/catalog/BasePolarisCatalogTest.java

service/common/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java

service/common/src/main/java/org/apache/polaris/service/catalog/io/FileIOFactory.java

service/common/src/main/java/org/apache/polaris/service/task/TaskFileIOSupplier.java

gradle/libs.versions.toml

dennishuo

LGTM

sfc-gh-dhuo · 2025-01-24T23:16:22Z

@snazy @dimas-b Looks like all the major comments have been resolved; any final/remaining suggestions before merging this?

XJDKC · 2025-01-24T23:46:21Z

Hey @snazy @dimas-b May I get another review for this PR, thanks!

dimas-b

Sorry about the delay. The latest changes LGTM with one remaining concern about WasbTranslatingFileIOFactory (which I'd like to clarity) and a few nit comments (which are optional).

service/common/src/main/java/org/apache/polaris/service/catalog/io/FileIOFactory.java

service/common/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java

...common/src/main/java/org/apache/polaris/service/catalog/io/WasbTranslatingFileIOFactory.java

service/common/src/main/java/org/apache/polaris/service/catalog/io/DefaultFileIOFactory.java

service/common/src/main/java/org/apache/polaris/service/catalog/io/FileIOUtil.java

service/common/src/test/java/org/apache/polaris/service/catalog/io/FileIOFactoryTest.java

...s/service/src/test/java/org/apache/polaris/service/quarkus/catalog/io/TestFileIOFactory.java

snazy · 2025-01-27T18:20:24Z

Recent changes LGTM - deferring to @dimas-b

dimas-b

I believe @XJDKC mentioned making a few minor follow-up changes... otherwise LGTM.

…ies-to-fileio

service/common/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java

XJDKC · 2025-01-28T22:32:23Z

Hi folks, I think this PR is ready to be merged into main branch.
I have addressed all the comments, there are still some followup tasks, but we can do them later:
e.g. remove test only loadFileIO interface from PolarisBaseCatalog.

Also I have updated the PR desc to clarify the purposes of this PR.
Can I get a final stamp for this PR? Thanks!
cc: @dimas-b @snazy @dennishuo @eric-maynard @collado-mike

service/common/src/main/java/org/apache/polaris/service/catalog/io/FileIOUtil.java

…vel for FileIOUtil

XJDKC requested review from RussellSpitzer, adutra, ashvina, collado-mike, dimas-b, ebyhr, eric-maynard, flyrain, jackye1995, jbonofre, snazy, takidau and vvcephei as code owners January 13, 2025 23:00

XJDKC force-pushed the pass-internal-properties-to-fileio branch from 057bd17 to 836d3f1 Compare January 13, 2025 23:01

XJDKC changed the title ~~Pass InternalProperties to FileIO~~ Pass Internal Properties to FileIO Jan 13, 2025

XJDKC mentioned this pull request Jan 13, 2025

Add additional storage config for PolarisCredentialProperty #701

Closed

XJDKC force-pushed the pass-internal-properties-to-fileio branch from 836d3f1 to 0a10632 Compare January 14, 2025 09:04

dimas-b reviewed Jan 14, 2025

View reviewed changes

service/common/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java Outdated Show resolved Hide resolved

XJDKC force-pushed the pass-internal-properties-to-fileio branch from 88bc1af to 6ff8f2b Compare January 14, 2025 18:10

dimas-b mentioned this pull request Jan 17, 2025

Replace "property bags" in polaris-management-service.yml with specific attributes #555

Open

XJDKC changed the title ~~Pass Internal Properties to FileIO~~ Extend FileIOFactory to Improve Customization Jan 22, 2025

dimas-b reviewed Jan 22, 2025

View reviewed changes

snazy reviewed Jan 22, 2025

View reviewed changes

sfc-gh-rxing and others added 2 commits January 22, 2025 02:50

Initial Commit

ea62b41

Fix test errors

decc5b8

XJDKC force-pushed the pass-internal-properties-to-fileio branch from 00a9ae4 to decc5b8 Compare January 22, 2025 17:09

snazy reviewed Jan 22, 2025

View reviewed changes

XJDKC added 2 commits January 22, 2025 10:46

Resolved some comments

67d7730

Resolved some comments

a5c4557

XJDKC added 5 commits January 22, 2025 15:41

Removed unused package

898079e

Make FileIOFactory as an ApplicationScoped bean

9e5cfe0

Revert some changes

2cb4552

Use FileIOFactory to get the creds for cleanup task

67ab108

Use new interface in tests

c89e831

dennishuo approved these changes Jan 24, 2025

View reviewed changes

dimas-b reviewed Jan 25, 2025

View reviewed changes

dimas-b reviewed Jan 27, 2025

View reviewed changes

dimas-b approved these changes Jan 27, 2025

View reviewed changes

Merge remote-tracking branch 'apache/main' into pass-internal-propert…

d8c1804

…ies-to-fileio

XJDKC requested a review from MonkeyCanCode as a code owner January 27, 2025 22:38

XJDKC added 2 commits January 27, 2025 14:58

Move some util classes to polaris-service-common as test fixtures

519e0b6

Delete the interface in FileIOFactory that is only used for testing

f3a5ef0

XJDKC force-pushed the pass-internal-properties-to-fileio branch from 9fd3482 to f3a5ef0 Compare January 27, 2025 22:59

dimas-b approved these changes Jan 28, 2025

View reviewed changes

service/common/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java Outdated Show resolved Hide resolved

XJDKC added 3 commits January 28, 2025 13:57

Refactor TestServices and use it in FileIOFactoryTest

3bb504f

Small change

ce73197

Rename TestFileIOFactory to MeasuredFileIOFactory

5170f6c

dimas-b approved these changes Jan 29, 2025

View reviewed changes

eric-maynard reviewed Jan 29, 2025

View reviewed changes

service/common/src/main/java/org/apache/polaris/service/catalog/io/FileIOUtil.java Show resolved Hide resolved

Use FileIOFactory to load catalog default FileIO in tests, fix log le…

5a2e046

…vel for FileIOUtil

eric-maynard enabled auto-merge (squash) January 29, 2025 21:27

eric-maynard approved these changes Jan 29, 2025

View reviewed changes

eric-maynard merged commit 23385a0 into apache:main Jan 29, 2025
5 checks passed

Extend FileIOFactory to Improve Customization #724

Extend FileIOFactory to Improve Customization #724

Conversation

XJDKC commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Uh oh!

snazy commented Jan 14, 2025

Uh oh!

Uh oh!

dimas-b left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dennishuo left a comment

Choose a reason for hiding this comment

Uh oh!

sfc-gh-dhuo commented Jan 24, 2025

Uh oh!

XJDKC commented Jan 24, 2025

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

snazy commented Jan 27, 2025

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

XJDKC commented Jan 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

XJDKC commented Jan 13, 2025 •

edited

Loading

dimas-b left a comment •

edited

Loading

XJDKC commented Jan 28, 2025 •

edited

Loading