Apache Polaris Roadmap Proposal

[flyrain]

Over the past months, we've collaborated with a wide range of stakeholders—companies, developers, and users—who are invested in the evolution of Apache Polaris. This roadmap consolidates those insights into a shared vision, ensuring that our efforts address the most impactful and widely supported improvements. We appreciate the valuable feedback and collaboration that have shaped this direction.

Note that the features can move in and out of the milestones based on prioritization, and available resources.

The roadmap Items can be broadly classified into several categories such as

1. Core Polaris functions
2. Catalog Federation and Integrations
3. Data Security, Data Governance and Compliance
4. Observability and Reliability
5. AI/ML

Feature Proposal List

Category	Feature	0.9	1.0	1.1	1.2	1.3	1.4	1.5+
Core Polaris	Iceberg REST Spec support (including view and multi-table transactions)	GA	GA	GA	GA	GA	GA	GA
	Support for Delta Format as Generic Tables		Preview	X	X	X	X	X
	Policy Store		GA	GA	GA	GA	GA	GA
	Table Maintenance Framework			X	X	X	X	X
	JDBC Persistence layer		GA	GA	GA	GA	GA	GA
	No SQL Persistence layer			X	X	X	X	X
	Catalog Browser experience(UI)							X
Catalog Federation & Integrations	Catalog Federation		Preview	X	X	X	X	X
	Catalog Migrator		X	X	X	X	X	X
	Identity Federation and SSO		GA	GA	GA	GA	GA	GA
Data Security, Governance and Compliance	Role-based Access Control(RBAC)	GA	GA	GA	GA	GA	GA	GA
	Credential Vending (including support for S3, ADLS and GCP)	GA	GA	GA	GA	GA	GA	GA
	S3 Request Signing			X	X	X	X	X
	Table Maintenance Policies		GA	GA	GA	GA	GA	GA
	Row level and column level policies				X	X	X	X
	Encryption Support					X	X	X
	Audit and events interface			X	X	X	X	X
	Data Lineage					X	X	X
	Data tagging and classifications							X
	Attribute-Based Access Control(ABAC)							X
Observability and Telemetry	Data lake operational metrics							X
	Data health monitoring and alerts							X
AI/ML	Volumes/Directory Tables							X

Legend:
X: Tentatively Planned
GA: Generally Available
Preview: Experimental Feature

Core Polaris Functions

Generic Tables and Delta Format Support

Polaris enables support for non-Iceberg table formats through the concept of Generic Tables. These tables behave similarly to regular tables but include an additional format attribute that defines the table format type. This approach not only opens up flexibility to support additional formats but also sets the stage for enhancing Polaris’ capabilities.

For example, while Polaris currently supports the Iceberg REST catalog for managing and querying large datasets, incorporating support for additional formats—such as Delta—would further extend its capabilities. Delta format support would allow for enhanced governance, compliance, data management, disaster recovery, and migrations within the same Catalog. This enhancement involves generating Iceberg metadata to read Delta tables and enabling both Delta read and write operations from engines like Apache Spark.

More details here

1. Generic Table Support
2. Generate Iceberg metadata to read Delta tables

Milestone: 1.1

Policy Store

Apache Polaris' support for a Policy Store allows it to serve as a centralized repository for all policies related to data assets, ensuring consistent governance and compliance across the organization. This includes policies for table maintenance, access control, data security, and overall data governance, enabling administrators to easily enforce, track, and audit these policies. By consolidating policy management in Polaris, organizations can streamline their data management processes while maintaining compliance and security standards.

More details here Policy Management in Apache Polaris
Milestone 1.0

Table Maintenance Framework

Table Maintenance Framework brings capabilities to store Table maintenance policies, properties, statistics, and events necessary for performing Table maintenance and Optimizations. This does not include actual Table maintenance operations that need to run a compute infrastructure. More details here Table Maintenance in Polaris
Milestone: 1.1

SQL and NoSQL Persistence

Enable SQL (ex. Postgres) and NoSQL (ex. DynamoDB, Cassandra, etc) persistence storage backends for Polaris. More details here Apache Polaris (incubating) - SQL/NoSQL persistence backend support
Milestone: 1.0

S3-compatible storage support

Support the s3-compatible storage, such as MinIO, Ceph, Dell ECS. More details are here, #389.
Milestone: 1.0

Catalog Browser experience (UI)

User Experience and Interface for Apache Polaris. Enable users to browse catalogs, databases and tables. Provides basic operations on governance, policy management, and other governance functions. #572 is the corresponding issue.
Milestone: 1.5+

Catalog Federation and Integrations

Catalog Federation

Enable federation of reads and writes to any remote catalog thus making Apache Polaris a Catalog of Catalogs. This primarily includes catalogs that support IRC and Hive protocols. Some details here Polaris Roadmap and Catalog Federation Diagrams
Milestone: 1.0

Catalog Migrator

Users may want to move Iceberg Tables from several Catalog solutions into Apache Polaris. Catalog migrator enables migration of tables registered in catalogs such as Glue, Hive, or other Iceberg Rest Catalogs into Apache Polaris.
Milestone: 1.0

Identity federation, SCIM, SSO and OAuth support

Supporting SCIM and SAML is essential for efficient user provisioning, seamless access management, and enhanced security, ensuring that users can securely access and manage data resources while complying with organizational policies. This also enable easy identity federation and OAuth federation to third party identity providers. More details here Adding Federated User and Role Support in Polaris
Milestone: 1.0

Data Security, Data Governance and Compliance

Governance Policies for Tables

Polaris will provide the ability to define access policies and other governance policies (such Retention) by Tables. More details here Policy Management in Apache Polaris
Milestone: 1.2

Column level and Row level Policies

Provides the capability to define and enforce column level and row level access and other governance policies. More details here project
Milestone: 1.2

Audit and Events Interface

Enable audit logs and history for Catalog, Database, Table, Property and Policy changes through events interface. Initial spec details here Polaris Event Listeners
Milestone 1.2

Data Lineage

Data Lineage functionality allows users to trace the flow of data across different systems and tables providing visibility into its origin, origins and usage. This feature enhances data governance auditability and troubleshooting by visually representing data’s lifecycle from source to destination. This includes Table and Column lineages.
Milestone 1.5+

Data Tagging and Classification

Enable categorizing and labeling data assets based on predefined criteria, such as data type, sensitivity, or usage. This helps organizations efficiently organize, search, and secure their data by assigning meaningful tags and classifications, enabling better governance and compliance management. Through this feature, users can quickly locate relevant data and ensure appropriate access controls are in place
Milestone 1.5+

Encryption Support

Enable support for encrypted Iceberg tables by managing Key Management Service (KMS) integrations. Facilitate the vending of encryption keys, ensuring seamless key retrieval and rotation for standard KMS solutions
Milestone 1.3

Observability, Telemetry and Reliability

Data Lake Operational Metrics

Enable operational metrics on Catalog, Databases and Tables to enable operational manageability of the Data Lake. This includes

• Data-level metrics: Number of files, Number of Partitions, Partition sizes, Total table size and more.
• Access Metrics: Number of R/W access on table, Query load per file, R/W Latency
• Data Health Metrics: Data Skew, Data Freshness, and more
• Storage-level Metrics: Storage Utilization, Number of Small Files, Storage Growth, Hot Partitions, and more

Milestone 1.5+

Data Health Monitoring and Alerts

Enable capabilities to monitor and alert on health of the data including

• Table Size and Growth Monitoring
• Un-compacted partitions and files
• Data Skew monitoring and alerts

Milestone 1.5+

AI/ML

Volumes/Directory Tables

A table-like entity like volumes can be used for organizing and managing unstructured data. Volumes provide a way to group related data files logically, similar to directories or containers. More details here Unstructured Data Support in Polaris
Milestone 1.5+

[mmgaggle]

Thank you for putting all of this information together. Is there any sort of consensus on a v1.0 release target, future release cadence, etc?

This is really exciting stuff.

[flyrain]

The current milestone and versions in the roadmap are based on the three-month release cycle proposed in #584. While the roadmap outlines the feature priorities and direction, the timelines could be subject to change as needed.

[jbonofre]

We know that the 1.0.0 release will take some time as it will include binary distributions (compared to 0.9.0). I already start to work on the legal aspects.
I think we can reasonably expect "release pace acceleration" after 1.0.0.

[snazy]

Thanks for summing up the features. It's great to give users visibility on the planned topics.

Most of these features haven't been discussed yet. We have to discuss first, then get a consensus and then implement and finish it.
Having said that, attaching version numbers and/or dates to any specific feature is perceived as a fixed plan and perceived as a "promise", irrespective whether there's some fine print that more or less says "that's just a best effort".

I'd rather see the only list of features and some ordering within that list.

Especially when we have mostly automated releases, the release cadence can (and should IMHO) be quite high - definitely much higher than "every three months". This will render "target version numbers" or related milestones invalid.

This also raises the question of how many major/minor versions shall be maintained or whether there's just one version being maintained.

To when a feature is finally merged into "main" and released: I'm a big fan of "it's ready when it's ready" and not tie anything to any particular date. Deadline-Driven-Development leads to compromises, to tech-debt, to unnecessary issues and eventually more work than necessary.

[sfc-gh-vlakkundi]

Thanks for sharing your thoughtful feedback!

I want to clarify that the milestones in the roadmap are not set in stone. They’re intended to reflect the relative priorities of the features rather than fixed commitments.

Features will be ready when they’re ready, and nothing will move forward without proper discussion and community consensus. Our goal with this roadmap is to provide visibility into the potential direction of the project, which can help consumers make informed decisions about adoption and integration.

Also, by laying out this view, we hope to inspire more contributions from the community. Seeing the roadmap might encourage contributors to engage and even help pull some features into earlier milestones through collaborative effort.

We appreciate your input. it’s essential in ensuring that Apache Polaris evolves in a way that stays true to open-source values and community-driven development.

[jbonofre]

I added S3 Request Signing to the roadmap table, with best effort to target 1.0.0 release.

[flyrain]

Hi @jbonofre , is there anyone working on s3 request signing?

[jbonofre]

Not yet as far as I know

[dimas-b]

@flyrain : if you're planning to start working on request signing, let's sync up on the dev ML about the design / approach before coding starts. I believe there are some prior ideas to be shared and discussed.

[flyrain]

I don't have a plan for that. Discussions are welcome always.

[jbonofre]

I updated the document to use Feature Proposal List instead of just Feature List.
As discussed during the last community meeting, I would propose to add "status" for each roadmap item ("proposed", "in discussion", "approved", "scheduled").

[sfc-gh-cwiese]

This enhancement involves generating Iceberg metadata to read Delta tables and enabling both Delta read and write operations from engines like Apache Spark.

@flyrain The design document you linked to explicitly lists writes to Delta tables as a non-goal, which seems to contradict your summary.

[flyrain]

There are two separated efforts. There is a conversion part for read-only use cases, and there is a direct support without conversion part which supports both read and write. Here is the design doc for the second part, https://docs.google.com/document/d/1_R9jBIwoH3CV9G7gSoRJPQVEcOsROp4IEiGoeYeQE8A/edit?tab=t.0#heading=h.1qpl30dg4lui. Will update the page soon.

[melin]

support paimon?
https://paimon.apache.org/

[adam-christian-software]

@melin - Feel free to file an issue and we can discuss it.

[flyrain]

@melin, Polaris now supports Delta Table via Generic table(https://polaris.apache.org/in-dev/unreleased/generic-table/), and we are working on Hudi support(#1862). We could support Paimon in a similar way, and I'm also open for alternative solutions.

[dimas-b]

To be precise, I believe Delta is supported via the Polaris Spark Client (a Spark plugin) that uses the Polaris Generic Table API for finding table locations.

[snazy]

Paimon has a rich REST API.
Generic tables is rather "just" a named pointer to a table location. I don't think Paimon fits into well there.

If there's demand, then we should offer the Paimon REST API in Polairs.