Implement ApplicationScope in Crossplane

[resouer]

What problem are you facing?

We didn't find concrete use case for Scope concept on K8s before so we decide to skip it temporarily in Crossplane. Though we now found Scope is actually a strong need for traffic mgmt in app. For example the Istio BookInfo sample should be able to be modeled easily with Crossplane's application model.

The mapping would be:

• Component - Deployment
• Trait - Service/Ingress per Deployment
• Scope - Service/Ingress/APIGateway across Deployments
• ApplicationConfiguration - a YAML file to grouping Component/Trait/Scope together

Mirror issue: oam-dev/rudr#558 (comment)

How could Crossplane help solve your problem?

Implement Scope feature in Crossplane's application model.

/cc @omkensey @ryanzhang-oss

[resouer]

@artursouza volunteered offline to tackle this issue, though I have no privilege to assign issue to him ...

@negz or @hasheddan help?

[hasheddan]

@resouer i think he needs to be added to Crossplane org. @jbw976 could you add @artursouza?

[jbw976]

I have sent an invitation to @artursouza to join the Crossplane org as a member. Once he accepts then issues can be assigned to him. I've never understood that GitHub limitation 🤔

[hasheddan]

Thanks so much @jbw976 :)

[resouer]

@hasheddan Hi Daniel, seems now the issue can be assigned to Artur. Could you help with that?

[hasheddan]

@resouer done :)

[draveness]

Hi @artursouza, do we have any ongoing PR or implementations on this issue? :)

[wonderflow]

I'm not sure if there's any existing proposal about how to implement scope? @artursouza

In my mind, scope is create by stand alone CR but component joining a scope should be controlled by oam-k8s-runtime.

For example, a workflow may be something like below:

1. Create a scope CR first

apiVersion: core.oam.dev/v1alpha2
kind: NetworkScope
metadata:
  name: my-vpc-network
  labels:
    region: us-west
    environment: production
spec:
  networkId:  my-vpc
  subnetIds:
  - my-subnet1
  - my-subnet2
  internetGatewayType: nat

2. create AppConfig who's going to use this scope

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: custom-single-app
spec:
  components:
    - componentName: backend
      scopes:
        - scopeRef:
            apiVersion: core.oam.dev/v1alpha2
            kind: NetworkScope
            name: my-vpc-network

3. AppConfig Controller(oam-k8s-runtime) will know nothing about the schema of this NetworkScope, but he can change the annotation. We use core.oam.dev/workloads including all workload instance created by component who's joined in this scope. For easy to use, we use json to encode.

apiVersion: core.oam.dev/v1alpha2
kind: NetworkScope
metadata:
  name: my-vpc-network
  labels:
    region: us-west
    environment: production
  annotation:
     core.oam.dev/workloads: '[{"apiversion":"data.oam.dev/v1","kind":"Cassandra","name":"backend"},{"apiversion":"apps/v1","kind":"Deployment","name":"frontend"}]'
spec:
  networkId:  my-vpc
  subnetIds:
  - my-subnet1
  - my-subnet2
  internetGatewayType: nat

4. finally the controller behind the NetworkScope can control these workload instances.

[artursouza]

Hi,

Sorry for the delay on this. Here is what I am thinking for ApplicationScope:

1. A CR(s) to deploy a resource first. In case of a network in Azure, it is done in 3 CRs, like in the example below:

---
apiVersion: network.azure.crossplane.io/v1alpha3
kind: VirtualNetwork
metadata:
  name: myvpc-example
spec:
  resourceGroupNameRef:
    name: aks-example
  location: West US 2
  properties:
    addressSpace:
      addressPrefixes:
        - 10.2.0.0/16
  reclaimPolicy: Delete
  providerRef:
    name: example
---
apiVersion: network.azure.crossplane.io/v1alpha3
kind: Subnet
metadata:
  name: mysubnet1-example
spec:
  resourceGroupNameRef:
    name: aks-example
  virtualNetworkNameRef:
    name: myvpc-example
  properties:
    addressPrefix: 10.2.0.0/24
    serviceEndpoints:
      - service: Microsoft.Sql
  reclaimPolicy: Delete
  providerRef:
    name: example

2. A CR to declare the Scope, referencing the resource above:

apiVersion: core.oam.dev/v1alpha2
kind: NetworkScope
metadata:
  name: myvpc-network
spec:
  networkId:  myvpc-example
  subnetIds:
  - mysubnet1-example
  - my-subnet2
  internetGatewayType: nat

3. Create AppConfig to use scope:

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: custom-single-app
spec:
  components:
    - componentName: backend
      scopes:
        - scopeRef:
            apiVersion: core.oam.dev/v1alpha2
            kind: NetworkScope
            name: myvpc-network

4. AppConfig Controller(oam-k8s-runtime) must be aware of the structure of each Scope Kind. This way, it know how to provision the workloads underneath. The network controller cannot change the deployed workload. The network where a workload is deployed is defined at the creation time of the workload, so the appconfig controller must understand "NetworkScope".

[wonderflow]

If appconfig controller MUST understand a specific kind of scope, it will be very hard to extend without changing the core of appconfig controller. Now we have very flexible extendability of OAM v1alpha2, we'd better not break it.

[hongchaodeng]

I agree with the point that appconfig controller should not know the spec of any scope types. Only the scope CR controller knows how to handle specific scope types, like NetworkScope. The workloads to the scope could be passed down as mentioned in https://github.com/crossplane/crossplane/issues/1480#issuecomment-625732816.

[artursouza]

I've created the work item for HealthScope.

For Ingress, since it is a Trait and not Scope, I did not create a work item for this here.

I suggest we continue the design discussion for Scopes in a Google Docs. I can share the link once we have a draft.

[resouer]

@artursouza Just clarify, Scope will be used to model cross components traffic service like Virtual Service in Istio, it could also be Ingress/APIGateway in some context. This is a real world use case and let's put it in design doc.

Essentially, Ingress is a trait or scope depends on it is used for single component or cross component.

[resouer]

Also, we will need to differentiate "bring your own Scope" and "develope your Scope" in the this design just like Trait - Workload proposal.

[wonderflow]

Yes, exactly, we'd better have a proposal about scope like Trait - Workload proposal before implementation.

[artursouza]

We have a document to discuss the proposal:
https://docs.google.com/document/d/1uMxRZKba0QnBJ0BpnjhyedbiEVgjWgliZ1ljWrbO0es/edit?usp=sharing

[resouer]

@artursouza Great! Could you please open comment access btw?

[negz]

Per #55 we're centralising all things OAM into one repo, and oam-kubernetes-runtime seems like the most likely place. I'm going to move this issue there.

[ryanzhang-oss]

@artursouza can we close this issue now?

[artursouza]

We did implement HealthScope. If this issue is about other Application Scopes too, then it should remain open.

[wonderflow]

Let's close this issue as health scope already be implemented as a good example about how to implement Application Scopes. Feel free to create a new one if new scopes are needed.