Policy Store: Add PolicyEntity and PolicyTypes #1133

HonahX · 2025-03-07T03:26:29Z

This PR introduces basic classes for supporting Policy Store in Polaris, as discussed in #1059.

PolicyEntity – Represents a policy object in Polaris.
PolicyType – Defines a category of policies, which can be either predefined or custom (user-defined). Currently only predefined classes supported

This will be the first step to unblock all other Policy PRs and discussions

cc: @flyrain

flyrain

LGTM. Left minor comments! Thanks @HonahX for the PR.

flyrain · 2025-03-07T19:46:25Z

polaris-core/src/main/java/org/apache/polaris/core/policy/PolicyEntity.java

+
+  @JsonIgnore
+  public int getPolicyTypeCode() {
+    String policyTypeCode = getPropertiesAsMap().get(POLICY_TYPE_CODE_KEY);


We will need policy type to be a top-level field instead of hidden in property map in the future, but it's fine now.

flyrain · 2025-03-07T21:57:52Z

polaris-core/src/main/java/org/apache/polaris/core/policy/PredefinedPolicyType.java

+import jakarta.annotation.Nullable;
+
+/* Represents all predefined policy types in Polaris */
+public enum PredefinedPolicyType implements PolicyType {


Minor: a name like SystemPolicyTypes ? Just wanna brainstorming a bit.

I think "Predefined Policy" is widely used throughout our design doc, so I still lean slightly toward keeping it. However, I’ll update it to the plural form.

polaris-core/src/main/java/org/apache/polaris/core/policy/PolicyEntity.java

dimas-b · 2025-03-07T23:19:30Z

As discussed in the sync meeting, if possible, please hold this PR until a NoSQL peristence impl. is available (hopefully soon).

HonahX · 2025-03-07T23:27:13Z

Hi @dimas-b, do you mean hold #1104 which includes the new Persistence Table, interface and impl? For that, I marked #1104 as draft to allow some time for ongoing refactoring and new impl.

For this one, the PolicyEntity will be persisted just like other existing entities. So I think it should not be related to any changes in persistence refactoring or new persistence impl and we can proceed this in parallel with ongoing persistence layer work.

Will NoSQL persistence impl introduce any changes that affect the current persistence model for Entitites?

dimas-b · 2025-03-07T23:30:21Z

Current NoSQL POC using typeCode (#1112), so adding new types will cause some interference, hence my request to hold this PR as well.

flyrain · 2025-03-08T00:14:17Z

I think it’s unreasonable to hold multiple PRs due to one persistence implementation. The NoSQL implementation isn’t the only ongoing persistence change—@singhpk234 and I are actively working on the Postgres implementation as well. Given that multiple efforts are running in parallel, we shouldn’t allow one refactor to block overall development.

cc @eric-maynard @dennishuo

jbonofre · 2025-03-08T05:42:29Z

@flyrain I agee. However, I'm a bit surprised @singhpk234 and you are working on PostgreSQL backend while the "new" API are not yet finalize. @dennishuo and @dimas-b are still working on it. I just want to avoid dual effort.
I propose to open draft PRs to give visibility and anticipate questions/comments to gain time.

flyrain · 2025-03-09T06:10:04Z

Thanks @jbonofre! Just like the MongoDb effort, Postgres support is an ongoing effort. I have made this PR(#1128) pass all tests,

snazy

I think the test coverage is not good enough, because I could imagine errors in the Iceberg catalog implementations as soon as there's some policy in there. Also the interaction of these new types via the entity manager is untested.

Generally, I think it's better to see the whole Policy thing working in a feature branch, which can be a PR in draft state, instead of pushing small bits of rather unused code.

HonahX · 2025-03-11T23:33:55Z

Hi @snazy, thanks for the feedback!

I think the test coverage is not good enough, because I could imagine errors in the Iceberg catalog implementations as soon as there's some policy in there.

Could you clarify this concern? The Policy implementation is independent of the Iceberg catalog. It has its own service interface/implementation, handler, and a PolicyCatalog within service-common, all separate from Iceberg. (I realize part of the design doc is outdated—I'll update that section.)

Also the interaction of these new types via the entity manager is untested.

I can add more tests in ResolverTest and PolarisTestMetastoreManager to improve coverage.

Generally, I think it's better to see the whole Policy thing working in a feature branch, which can be a PR in draft state, instead of pushing small bits of rather unused code.

I do have a draft PR #1104 that includes all necessary Polaris-core changes for the policy store, including basic classes and the PolicyMappingRecord persistence interfaces/implementations. I can publish a larger draft PR that combines core changes with the service implementation for reference. That said, I’d still recommend reviewing and merging smaller pieces incrementally—this makes it easier for everyone to review and avoids one massive PR that’s harder to manage.

Let me know what you think!

polaris-core/src/main/java/org/apache/polaris/core/policy/PolicyEntity.java

singhpk234 · 2025-03-13T22:38:02Z

polaris-core/src/main/java/org/apache/polaris/core/policy/PredefinedPolicyTypes.java

+
+/* Represents all predefined policy types in Polaris */
+public enum PredefinedPolicyTypes implements PolicyType {
+  DATA_COMPACTION(0, "system.data-compaction", true),


[doubt] I might be late to this, but why system prefix ?

The prefix is to make it consistent with the path of the content schema of these policies, introduced in #969.

singhpk234 · 2025-03-13T22:46:50Z

polaris-core/src/main/java/org/apache/polaris/core/policy/PolicyType.java

+ * either predefined or custom (user-defined).
+ *
+ * <p>A policy type can be either inheritable or non-inheritable. Inheritable policies are passed
+ * down to lower-level entities (e.g., from a namespace to a table).


[doubt] may be a naive question, what stops these policies to inherited from namespace -> view ?
As we don't wanna run stuff for views >

As we don't wanna run stuff for views >

Good question, I think that's a valid concern. Currently, our engine determines how policies are enforced, and the TMS service is focused solely on tables. That said, if we later find it necessary to restrict inheritance, we can extend the PolicyType to provide the functionality to limit the valid target types.

cc @flyrain

Currently, our engine determines how policies are enforced, and the TMS service is focused solely on tables

Thats true for enforcing part, but polaris here is acting as a policy store, which should make sure if such inheritence if they are un-intentional are blocked. So a call to get a TMS policy on view should return nothing.

That being said it fine if we want address this later, considering presently the caller just wants TMS on tables and never on a view.

singhpk234

LGTM as well, thanks @HonahX !

The comment is resolved by the comment and #1133 (comment) and new unit test cases added in the new commits.

github-project-automation bot added this to Basic Kanban Board Mar 7, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Mar 7, 2025

HonahX mentioned this pull request Mar 7, 2025

Policy Store: PolicyMappingRecord with Persistence Impl #1104

Merged

HonahX marked this pull request as ready for review March 7, 2025 18:15

HonahX requested review from adutra, ashvina, dennishuo, dimas-b, eric-maynard, jackye1995, jbonofre, vvcephei, collado-mike, snazy, RussellSpitzer, takidau, MonkeyCanCode, flyrain and ebyhr as code owners March 7, 2025 18:15

flyrain reviewed Mar 7, 2025

View reviewed changes

flyrain approved these changes Mar 7, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Mar 7, 2025

eric-maynard approved these changes Mar 10, 2025

View reviewed changes

snazy previously requested changes Mar 11, 2025

View reviewed changes

github-project-automation bot moved this from Ready to merge to PRs In Progress in Basic Kanban Board Mar 11, 2025

singhpk234 reviewed Mar 13, 2025

View reviewed changes

HonahX added 6 commits March 13, 2025 18:45

PolicyEntity and PolicyType

967398c

add more tests

5a5c73e

use plural form

4cfb7f1

fix test

f2f64dd

fix test

45dd562

Throw error instead of -1

875caf2

singhpk234 approved these changes Mar 14, 2025

View reviewed changes

change return type to int

444fd1b

HonahX force-pushed the honahx_add_policy_base_classes branch from 61c0059 to 444fd1b Compare March 18, 2025 00:02

flyrain approved these changes Mar 18, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Mar 18, 2025

flyrain merged commit b1f7c68 into apache:main Mar 18, 2025
5 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Mar 18, 2025

rakesh-das08 pushed a commit to rakesh-das08/polaris that referenced this pull request Mar 22, 2025

Policy Store: Add PolicyEntity and PolicyTypes (apache#1133)

ddd55af

Policy Store: Add PolicyEntity and PolicyTypes #1133

Policy Store: Add PolicyEntity and PolicyTypes #1133

Uh oh!

Conversation

HonahX commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

flyrain Mar 7, 2025

Choose a reason for hiding this comment

Uh oh!

flyrain Mar 7, 2025

Choose a reason for hiding this comment

Uh oh!

HonahX Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dimas-b commented Mar 7, 2025

Uh oh!

HonahX commented Mar 7, 2025

Uh oh!

dimas-b commented Mar 7, 2025

Uh oh!

flyrain commented Mar 8, 2025

Uh oh!

jbonofre commented Mar 8, 2025

Uh oh!

flyrain commented Mar 9, 2025

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

HonahX commented Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

singhpk234 Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

HonahX Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singhpk234 Mar 13, 2025

Choose a reason for hiding this comment

Uh oh!

HonahX Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

singhpk234 Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

HonahX commented Mar 7, 2025 •

edited

Loading

HonahX commented Mar 11, 2025 •

edited

Loading

HonahX Mar 13, 2025 •

edited

Loading