Add KEP-924: Introduce PVC-based artifact storage for Kubeflow Pipelines

[hbelmiro]

Resolves: #924

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign juliusvonkohout for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[juliusvonkohout]

That is now the execption and we go directly to seaweedfs with per namespace credentials instead https://github.com/kubeflow/manifests/releases/tag/v1.11.0-rc.1 So we have empty zero overhead namespaces by default.

[hbelmiro]

@juliusvonkohout sorry, I'm not following.
If users need complete PVC isolation, we need one artifacts server in each namespace because the launcher wouldn't be able to access the PVC directly in a different namespace.
That said, central mode is the default in this KEP (single artifact server + PVC for all namespaces). Namespace-local mode is optional for environments with strict isolation requirements.

[juliusvonkohout]

Here are some thoughts, but I still need to read the full document.

Tthat is very far away from V1 artifact passing via PVC and would violate zero overhead namespaces. Just imagine a cluster with 1000 namespaces for scalability. Then you add 1000 permanent pods, so massive overhead. In V1 we just accepted that the artifacts will not be in the UI instead. PVC for all namespace also sounds scary. Why should we offer such a security nightmare in the first place? That would break the namespace isolation contract.

Maybe I am missing something and I very much admire that you spend the time for the KEP, but there might be some fundamental problems as mentioned above regarding security and scalability that we did not have in the kfp V1 Implementation.

But as I said some of my statements could be wrong and that is just my initial assessment without checking it thoroughly.

[hbelmiro]

Got it. Those are really valid concerns for using it in production. But the idea of using PVCs is to make it easier for people who want to try KFP without having to worry about deploying and configuring an object storage. That means development (and mostly local) environments.
I didn't want to limit the usage only to development as it can be used on simple production environments.
We need to add clear documentation about this telling that the usage in production is not encouraged although not forbidden. Your concern is a good example that we can use to discourage using it in production.

[mprahl]

@hbelmiro could you reopen this KEP in the KFP repo since this is KFP specific and not Kubeflow wide?

[hbelmiro]

Done!

• Add KEP-12513: Introduce PVC-based artifact storage for Kubeflow Pipe… pipelines#12515