Integration of expectations

[csviri]

See references and description of expectations here:
https://ahmet.im/blog/controller-pitfalls/#expectations-pattern

This does not seems to be supported in controller runtime either:
kubernetes-sigs/controller-runtime#644

We should investigate how we could support this in different layers of the framework.

[csviri]

What I could imaging is to have an API that looks something like this:

UpdateControl reconcile(Primary primary, Context<Primary> context) {

// omit reconcile details

  return UpdateControl.expect((context,primary)-> {
     var resources = context.getSecondaryResources(MyResource.class)
     return checkIfResourcesInexpectedState(resources) 
   },Duration.ofSeconds(5))

}

In other words, that the reconcile could return an expectation with a timeout, and the reconciliation would not be triggered until that expectation predicate is true or it timeouts.

While this sounds nice, and might provide a benefit for the users, execution, we won't have too much space to optimize it, so in terms of execution, it would have to be done anyway in the executor service thread that we use now for reconciliation. So this can be directly transformed to something like this:

UpdateControl reconcile(Primary primary, Context<Primary> context) {
  var resources = context.getSecondaryResources(MyResource.class)
   if (!checkIfResourcesInexpectedState(resources)) {
      return UpdateControl.noUpdate()
   } 
}

Except the timeout for which user might have to track hat in a ConcurrentHashMap or such.

It would be a completely different story if this worked as a cross-cutting filter, where all the event source caches are available.

It would worth to make a prototype, how that would look like, but for that we would need a new abstraction, probably a subset of Context to access secondary resources from the cache and directly event sources in case.

[csviri]

cc @xstefank @metacosm @shawkins, maybe something to discuss in the next community meeting.