Extend FileIOFactory to Improve Customization

[XJDKC]

Context

Currently, we have three places where a FileIO is constructed:

1. PolarisBaseCatalog::loadFileIO: Used as the default FileIO across the entire catalog during testing.
2. PolarisBaseCatalog::refreshIOWithCredential: Refreshes storage credentials and constructs a new FileIO.
3. TaskFileIOSupplier::apply: Fetches subscoped credentials and uses FileIOFactory to create a FileIO for cleanup tasks.

We’ve identified an issue where not all properties required by FileIO are generated by the FileIOFactory. For instance, subscoped credentials are retrieved outside the FileIOFactory and then passed to it. Additionally, the current FileIOFactory interface has limited access to contextual information.

To enhance the ability to customize FileIOFactory, we plan to extend it by passing additional information. This improvement will enable developers to implement custom logic such as auditing, proxy, and more.

Some code pointers:

• PolarisBaseEntity.java#L61-L62
• TaskFileIOSupplier.java#L56-L88

[snazy]

Thanks for your contribution @XJDKC!
I think this change blurs the line between "properties" and "internal properties" quite a bit. I'm also, in general, concerned about generic property bag per se containing all kinds of information - things that can be exposed (upstream/downstream) and things that must not - nobody can ensure that no "bad things" (think: logging secrets) happen to this information.

[dimas-b]

Thanks for the update @XJDKC ! Overall, the new code looks good to me :) A few minor comments below.

[dennishuo]

LGTM

[sfc-gh-dhuo]

@snazy @dimas-b Looks like all the major comments have been resolved; any final/remaining suggestions before merging this?

[XJDKC]

Hey @snazy @dimas-b May I get another review for this PR, thanks!

[dimas-b]

Sorry about the delay. The latest changes LGTM with one remaining concern about WasbTranslatingFileIOFactory (which I'd like to clarity) and a few nit comments (which are optional).

[snazy]

Recent changes LGTM - deferring to @dimas-b

[dimas-b]

I believe @XJDKC mentioned making a few minor follow-up changes... otherwise LGTM.

[XJDKC]

Hi folks, I think this PR is ready to be merged into main branch.
I have addressed all the comments, there are still some followup tasks, but we can do them later:
e.g. remove test only loadFileIO interface from PolarisBaseCatalog.

Also I have updated the PR desc to clarify the purposes of this PR.
Can I get a final stamp for this PR? Thanks!
cc: @dimas-b @snazy @dennishuo @eric-maynard @collado-mike